From epiphani at gmail.com Mon Nov 2 23:51:40 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Mon, 2 Nov 2009 17:51:40 -0500 Subject: [rsyslog] Queuing bug (4.5.5) Message-ID: Greetings all, I appear to have run into an issue with 4.5.5 where queues are not being flushed in a timely manner. In this specific case, I have data from last wednesday that is being written to disk in small chunks today since last wednesday. Unfortunately I cannot be too specific in details, but here is what I can see: Using omfile+gzip, there appears to have been a decent burst in traffic sometime last wednesday. The rsyslog instance has grown to 1.8GB of memory and stayed there for a while. I have both main message and action queues defined using fixedarray, and I see no on-disk queues (appears to be entirely in memory). I've got templates writing out to filenames using an hourly timestamp (filenames like: [token]-[host]-YYYYMMDD-HH.txt.gz) In some of those files, all of them less than 5k in size, there are between 5 and 15 lines of content, all of them from last wednesday, and within a few seconds of each other. It's almost like there was a significant queue built up, the hour rolled over, and only the first block of lines were pulled from the queue. Then the hour rolled over again, and another block of lines were pulled from the queue. Then the next hour, then another 5-15 lines. Is it possible that one of the queues still has a good chunk of data built up, and is flushing it out very slowly? It hasn't been consistantly at the top of the hour either, and not every hour. But the log lines themselves are sequentially written out, and usually the lines are within a few seconds of each other. For example: syslog-myhost-20091102-18.txt.gz: 3 lines, 2 with TS Oct 21 18:46:34 and one 18:46:35 syslog-myhost-20091102-19.txt.gz: 17 lines, 3 Oct 21 18:46:35, 14 lines Oct 21 18:46:36 syslog-myhost-20091102-20.txt.gz: 12 lines, 8 Oct 21 18:46:36, 4 lines Oct 21 18:46:37 Thoughts? -Aaron From rgerhards at hq.adiscon.com Tue Nov 3 07:46:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 07:46:29 +0100 Subject: [rsyslog] Queuing bug (4.5.5) References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> mhhh... This is obviously not intended behavior. There are some rate-limiting features that can deliberately slow down a queue, but I guess you have not configured these. So there either is a bug that activates them at some point during processing (e.g. an invalid memory access could do that) or there is some other bug that causes the dequeue to happen very slowly. In any case, it is hard to guess. Given the volume of the messages, a debug log probably does not help. We could gain a little insight in at least the queue sizes that rsyslog sees via imdiag plus the (very rudamentary) v5 debug front-end (it doesn't require the engine to be v5!). I would need to spend at least a little work on the front-end to enable remote access, but that's not really a problem. One other thing is that I am still holding a v4-beta with additional fixes as I didn't want to put these in the middle of some other debugging. However, these fixes address potential memory problems, so the most appropriate course of action would be to give that version at least a try. It needs to be released in the next days in any case. I have uploaded that (pre-4.5.6) version so that you can give it a try if you like: http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz I think it would good if you could try it at least once, because I think any other troubleshooting will boil down to looking at the fixes this version contains. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Monday, November 02, 2009 11:52 PM > To: rsyslog-users > Subject: [rsyslog] Queuing bug (4.5.5) > > Greetings all, > > I appear to have run into an issue with 4.5.5 where queues are not > being flushed in a timely manner. In this specific case, I have data > from last wednesday that is being written to disk in small chunks > today since last wednesday. Unfortunately I cannot be too specific in > details, but here is what I can see: > > Using omfile+gzip, there appears to have been a decent burst in > traffic sometime last wednesday. The rsyslog instance has grown to > 1.8GB of memory and stayed there for a while. I have both main > message and action queues defined using fixedarray, and I see no > on-disk queues (appears to be entirely in memory). > > I've got templates writing out to filenames using an hourly timestamp > (filenames like: [token]-[host]-YYYYMMDD-HH.txt.gz) In some of those > files, all of them less than 5k in size, there are between 5 and 15 > lines of content, all of them from last wednesday, and within a few > seconds of each other. It's almost like there was a significant queue > built up, the hour rolled over, and only the first block of lines were > pulled from the queue. Then the hour rolled over again, and another > block of lines were pulled from the queue. Then the next hour, then > another 5-15 lines. > > Is it possible that one of the queues still has a good chunk of data > built up, and is flushing it out very slowly? It hasn't been > consistantly at the top of the hour either, and not every hour. But > the log lines themselves are sequentially written out, and usually the > lines are within a few seconds of each other. > > For example: > > syslog-myhost-20091102-18.txt.gz: 3 lines, 2 with TS Oct 21 18:46:34 > and one 18:46:35 > syslog-myhost-20091102-19.txt.gz: 17 lines, 3 Oct 21 18:46:35, 14 > lines Oct 21 18:46:36 > syslog-myhost-20091102-20.txt.gz: 12 lines, 8 Oct 21 18:46:36, 4 > lines Oct 21 18:46:37 > > Thoughts? > -Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 3 07:59:23 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 07:59:23 +0100 Subject: [rsyslog] Queue sizing References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103316@GRFEXC.intern.adiscon.com> Sorry for the late reply, but I was really busy with my article last week. Let me provide some rough numbers. For usual syslog traffic (< 100 bytes message size), 768 bytes is a good size to be expected for use by each message. If you have larger traffic, add the full average size to that picture (e.g. average size is 200 bytes, the typical size of a message object is around 968 bytes, NOT 868 - this has to do with some internal optimization). Then, simply multiply the configured size of the queue with that number. For example, you have a 2,000,000 message max configured, so 2,000,000 * 768 which gives you ~ 1.4 GB message store if the queue is full. The same applies to the action queues, here we have 200,000 entries per queue, that would sum up to ~140 MB per queue. Multiply that with the number of action queues. If, for example, you have 20 action queues, that would sum up to a total memory requirement of 20*.14+1.4 = 4.2 GB of total queue memory. Of course, this is an extreme number, and typically no system will have all action queues totally filled up. For omfile, with background writing (always enabled in ZIP mode), we use double-buffering with the configured buffers sizes (256KB being the default). So add 0.5 MB per open dynafile in this case. Multiply that by the maximum number of dynafiles you permit to be kept open. Add these numbers for each dynafile action. That gives you a (very theoretical) upper bound of memory use for the output system. Limiting the memory use is so simple that it probably is easy to overlook: just reduce the maximum numbers - that's why they exist ;) Depending on the input source, rsyslog then tries to slow down too-fast senders if it can't keep up. This works pretty well with TCP and not at all with UDP. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Wednesday, October 28, 2009 4:14 PM > To: rsyslog-users > Subject: [rsyslog] Queue sizing > > Greetings, > > I'm trying to get a good handle on queue sizing and memory usage. In > a higher-traffic environment, I have a dedicated rsyslog machine with > a large amount of ram. I want to try and find the right balance > between large queues for burst traffic, but I also want to make sure > that the queue sizes are small enough that we don't run out of memory > and crash. I'm using 4.5.5 currently. > > I'm currently using disk-backed memory queues, configured in > fixedarray, like so: > > $MainMsgQueueSize 2000000 > $MainMsgQueueMaxFileSize 50g > $MainMsgQueueSaveOnShutdown on > $MainMsgQueueHighWaterMark 15000000 > $MainMsgQueueLowWaterMark 10000000 > $MainMsgQueueType FixedArray > $MainMsgQueueWorkerThreads 2 > > $ActionQueueSize 200000 > $ActionQueueMaxFileSize 50g > $ActionQueueSaveOnShutdown on > $ActionQueueHighWaterMark 1500000 > $ActionQueueLowWaterMark 1000000 > $ActionQueueType FixedArray > $ActionQueueWorkerThreads 1 > > > I usually have upwards of a dozen action queue definitions, and my > omfile outputs are based on hostname of the incoming connection. Is > there some way I can calculate maximum memory use by the queuing > system, or force the system to a set memory maximum where I -know- > that it will simply start pushing back on the incoming connection > instead of adding more data to the system? > > Thoughts? > -Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 3 09:59:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 09:59:44 +0100 Subject: [rsyslog] queue configuration References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103318@GRFEXC.intern.adiscon.com> David and all, I have created a new output plugin, omruleset, which permits to nest rulesets. What it does is copy a message over from one ruleset to another and make it process in parallel. Some more details in the doc: http://www.rsyslog.com/doc-omruleset.html It permits to do what you asked for below. It also permits to do many more things and create very advanced configurations. But one must be VERY careful to receive the intended results. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 23, 2009 10:23 PM > To: rsyslog-users > Subject: [rsyslog] queue configuration > > I know that you can create an action queue for a specific output. > > is there any way to create an action queue for multiple outputs? > > for example, in my configuration I have > > :fromhost, !isequal, "127.0.0.1" > /var/log/messages;TraditionalFormat > :fromhost, isequal, "127.0.0.1" > @192.168.1.1;TraditionalForwardFormat > *.* @192.168.1.115 > *.* @192.168.1.241 > *.* @192.168.1.242 > *.* @192.168.1.6 > *.* @192.168.1.7 > *.* @192.168.1.122 > :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 > & @192.168.1.1;fixsnareForwardFormat2 > & ~ > :syslogtag, startswith, "MSWinEventLog#011" > /var/log/messages;fixsnareFormat > & @192.168.1.1;fixsnareForwardFormat > & ~ > *.* /var/log/messages;TraditionalFormat > *.* @192.168.1.1 > > it would be nice to move the outbound relays off to a different queue, > and > to put the list of rules that have order dependancies (because they > output > in different formats to fix up formatting problem) to a seperate queue > to > spread the work across multiple processes and not slow down the basic > writing to file. > > but as far as I can tell I would have to create a queue for each > individual item, and I can't create a queue for the part of the config > where I need to discard. > > am I missing something (including a better way to do everything ;-) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Nov 3 11:58:02 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 3 Nov 2009 11:58:02 +0100 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200910201639.43143.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> Message-ID: <200911031158.02666.marc.schiffbauer@mightycare.de> Hello Rainer, is there some news about this issue? TIA -Marc Am Dienstag, 20. Oktober 2009 18:38:16 schrieb Rainer Gerhards: > thanks! > > Interesting, I see that only one of the messages that should be in the .qi > are actually present. I wonder if this is related to the fact that ompgsql > emits an error message itself while being down (the others don't do this > AFIK). Will try to dig down to this (but I have to do so as time permits). > > Rainer > From mikel at irontec.com Tue Nov 3 12:20:18 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 03 Nov 2009 12:20:18 +0100 Subject: [rsyslog] working directoy dude Message-ID: <4AF011F2.7040501@irontec.com> Hi! I have a simple question: When you configure reliable TCP mesage forwarding, the "$WorkDirectory /rsyslog/work" have to be created? So, mkdir -p /rsyslog/work ? Thanks From rgerhards at hq.adiscon.com Tue Nov 3 12:28:19 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 12:28:19 +0100 Subject: [rsyslog] working directoy dude References: <4AF011F2.7040501@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710331C@GRFEXC.intern.adiscon.com> yes :) > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, November 03, 2009 12:20 PM > To: rsyslog-users > Subject: [rsyslog] working directoy dude > > Hi! > > I have a simple question: > > When you configure reliable TCP mesage forwarding, the "$WorkDirectory > /rsyslog/work" have to be created? > > So, mkdir -p /rsyslog/work ? > > Thanks > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mikel at irontec.com Tue Nov 3 12:36:02 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 03 Nov 2009 12:36:02 +0100 Subject: [rsyslog] working directoy dude In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710331C@GRFEXC.intern.adiscon.com> References: <4AF011F2.7040501@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA710331C@GRFEXC.intern.adiscon.com> Message-ID: <4AF015A2.6090303@irontec.com> Thanks Rainer!! Rainer Gerhards wrote: > yes :) > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Tuesday, November 03, 2009 12:20 PM >> To: rsyslog-users >> Subject: [rsyslog] working directoy dude >> >> Hi! >> >> I have a simple question: >> >> When you configure reliable TCP mesage forwarding, the "$WorkDirectory >> /rsyslog/work" have to be created? >> >> So, mkdir -p /rsyslog/work ? >> >> Thanks >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From epiphani at gmail.com Tue Nov 3 15:59:57 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 3 Nov 2009 09:59:57 -0500 Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> Message-ID: This is still taking place this hour - though I obviously can't restart onto a newer version without losing this case. Is there anything I can do in the current configuration to try and debug this situation? (We're up to 18:46:51 now!) -Aaron On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards wrote: > mhhh... This is obviously not intended behavior. There are some rate-limiting > features that can deliberately slow down a queue, but I guess you have not > configured these. So there either is a bug that activates them at some point > during processing (e.g. an invalid memory access could do that) or there is > some other bug that causes the dequeue to happen very slowly. In any case, it > is hard to guess. > > Given the volume of the messages, a debug log probably does not help. We > could gain a little insight in at least the queue sizes that rsyslog sees via > imdiag plus the (very rudamentary) v5 debug front-end (it doesn't require the > engine to be v5!). I would need to spend at least a little work on the > front-end to enable remote access, but that's not really a problem. > > One other thing is that I am still holding a v4-beta with additional fixes as > I didn't want to put these in the middle of some other debugging. However, > these fixes address potential memory problems, so the most appropriate course > of action would be to give that version at least a try. It needs to be > released in the next days in any case. > > I have uploaded that (pre-4.5.6) version so that you can give it a try if you > like: > > http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz > > I think it would good if you could try it at least once, because I think any > other troubleshooting will boil down to looking at the fixes this version > contains. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> Sent: Monday, November 02, 2009 11:52 PM >> To: rsyslog-users >> Subject: [rsyslog] Queuing bug (4.5.5) >> >> Greetings all, >> >> I appear to have run into an issue with 4.5.5 where queues are not >> being flushed in a timely manner. ?In this specific case, I have data >> from last wednesday that is being written to disk in small chunks >> today since last wednesday. ?Unfortunately I cannot be too specific in >> details, but here is what I can see: >> >> Using omfile+gzip, there appears to have been a decent burst in >> traffic sometime last wednesday. ?The rsyslog instance has grown to >> 1.8GB of memory and stayed there for a while. ?I have both main >> message and action queues defined using fixedarray, and I see no >> on-disk queues (appears to be entirely in memory). >> >> I've got templates writing out to filenames using an hourly timestamp >> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of those >> files, all of them less than 5k in size, there are between 5 and 15 >> lines of content, all of them from last wednesday, and within a few >> seconds of each other. ?It's almost like there was a significant queue >> built up, the hour rolled over, and only the first block of lines were >> pulled from the queue. ?Then the hour rolled over again, and another >> block of lines were pulled from the queue. ?Then the next hour, then >> another 5-15 lines. >> >> Is it possible that one of the queues still has a good chunk of data >> built up, and is flushing it out very slowly? ?It hasn't been >> consistantly at the top of the hour either, and not every hour. ?But >> the log lines themselves are sequentially written out, and usually the >> lines are within a few seconds of each other. >> >> For example: >> >> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 18:46:34 >> and one 18:46:35 >> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, 14 >> lines Oct 21 18:46:36 >> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, 4 >> lines Oct 21 18:46:37 >> >> Thoughts? >> -Aaron >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Tue Nov 3 16:06:15 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 3 Nov 2009 07:06:15 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 3 Nov 2009, Aaron Wiebe wrote: > This is still taking place this hour - though I obviously can't > restart onto a newer version without losing this case. Is there > anything I can do in the current configuration to try and debug this > situation? if you can strace each thread for a few seconds it may give you an idea what's happening. in the meantime you need to stop the system from getting further behind, can you redirect or reconfigure the senders so that they are not sending new logs to this box so that it can dig itself out (stopping the input may be enough by itself, rsyslog has historicly done a LOT of locking around the main queue, and if you have a full queue with more data trying to be delivered it can really slow things down. I wouldn't expect it to be this much, but it could be part of it) David Lang > (We're up to 18:46:51 now!) > > -Aaron > > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards > wrote: >> mhhh... This is obviously not intended behavior. There are some rate-limiting >> features that can deliberately slow down a queue, but I guess you have not >> configured these. So there either is a bug that activates them at some point >> during processing (e.g. an invalid memory access could do that) or there is >> some other bug that causes the dequeue to happen very slowly. In any case, it >> is hard to guess. >> >> Given the volume of the messages, a debug log probably does not help. We >> could gain a little insight in at least the queue sizes that rsyslog sees via >> imdiag plus the (very rudamentary) v5 debug front-end (it doesn't require the >> engine to be v5!). I would need to spend at least a little work on the >> front-end to enable remote access, but that's not really a problem. >> >> One other thing is that I am still holding a v4-beta with additional fixes as >> I didn't want to put these in the middle of some other debugging. However, >> these fixes address potential memory problems, so the most appropriate course >> of action would be to give that version at least a try. It needs to be >> released in the next days in any case. >> >> I have uploaded that (pre-4.5.6) version so that you can give it a try if you >> like: >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz >> >> I think it would good if you could try it at least once, because I think any >> other troubleshooting will boil down to looking at the fixes this version >> contains. >> >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>> Sent: Monday, November 02, 2009 11:52 PM >>> To: rsyslog-users >>> Subject: [rsyslog] Queuing bug (4.5.5) >>> >>> Greetings all, >>> >>> I appear to have run into an issue with 4.5.5 where queues are not >>> being flushed in a timely manner. ?In this specific case, I have data >>> from last wednesday that is being written to disk in small chunks >>> today since last wednesday. ?Unfortunately I cannot be too specific in >>> details, but here is what I can see: >>> >>> Using omfile+gzip, there appears to have been a decent burst in >>> traffic sometime last wednesday. ?The rsyslog instance has grown to >>> 1.8GB of memory and stayed there for a while. ?I have both main >>> message and action queues defined using fixedarray, and I see no >>> on-disk queues (appears to be entirely in memory). >>> >>> I've got templates writing out to filenames using an hourly timestamp >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of those >>> files, all of them less than 5k in size, there are between 5 and 15 >>> lines of content, all of them from last wednesday, and within a few >>> seconds of each other. ?It's almost like there was a significant queue >>> built up, the hour rolled over, and only the first block of lines were >>> pulled from the queue. ?Then the hour rolled over again, and another >>> block of lines were pulled from the queue. ?Then the next hour, then >>> another 5-15 lines. >>> >>> Is it possible that one of the queues still has a good chunk of data >>> built up, and is flushing it out very slowly? ?It hasn't been >>> consistantly at the top of the hour either, and not every hour. ?But >>> the log lines themselves are sequentially written out, and usually the >>> lines are within a few seconds of each other. >>> >>> For example: >>> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 18:46:34 >>> and one 18:46:35 >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, 14 >>> lines Oct 21 18:46:36 >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, 4 >>> lines Oct 21 18:46:37 >>> >>> Thoughts? >>> -Aaron >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 3 16:25:30 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 16:25:30 +0100 Subject: [rsyslog] Queuing bug (4.5.5) References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, November 03, 2009 4:06 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) > > On Tue, 3 Nov 2009, Aaron Wiebe wrote: > > > This is still taking place this hour - though I obviously can't > > restart onto a newer version without losing this case. Is there > > anything I can do in the current configuration to try and debug this > > situation? > > if you can strace each thread for a few seconds it may give you an idea > what's happening. This is a good suggestion. It would also potentially be enlightening to attach gdb to the process at various points in time and get a backtrace from all running threads. Other than that, there is little you can do on a running version. If it were compiled for debug, and the environment setup so that we could capture stdout/stderr, sending SIGUSR2 would yield to much the same information as the gdb backtrace BUT when runtime instrumentation is enabled, the build is operating at a third to a quarter of its normal speed. > in the meantime you need to stop the system from getting further > behind, > can you redirect or reconfigure the senders so that they are not > sending > new logs to this box so that it can dig itself out (stopping the input > may > be enough by itself, rsyslog has historicly done a LOT of locking > around > the main queue, and if you have a full queue with more data trying to > be > delivered it can really slow things down. I wouldn't expect it to be > this > much, but it could be part of it) > This may clean up things, but I really doubt it, given the magnitude of delays. Also, Aaron runs 4.4.5, which already has lots of the locking removed/restructure (not to compare with the recent v5-devel, but much more effcient than early v4 and v3). Rainer > David Lang > > > (We're up to 18:46:51 now!) > > > > -Aaron > > > > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards > > wrote: > >> mhhh... This is obviously not intended behavior. There are some > rate-limiting > >> features that can deliberately slow down a queue, but I guess you > have not > >> configured these. So there either is a bug that activates them at > some point > >> during processing (e.g. an invalid memory access could do that) or > there is > >> some other bug that causes the dequeue to happen very slowly. In any > case, it > >> is hard to guess. > >> > >> Given the volume of the messages, a debug log probably does not > help. We > >> could gain a little insight in at least the queue sizes that rsyslog > sees via > >> imdiag plus the (very rudamentary) v5 debug front-end (it doesn't > require the > >> engine to be v5!). I would need to spend at least a little work on > the > >> front-end to enable remote access, but that's not really a problem. > >> > >> One other thing is that I am still holding a v4-beta with additional > fixes as > >> I didn't want to put these in the middle of some other debugging. > However, > >> these fixes address potential memory problems, so the most > appropriate course > >> of action would be to give that version at least a try. It needs to > be > >> released in the next days in any case. > >> > >> I have uploaded that (pre-4.5.6) version so that you can give it a > try if you > >> like: > >> > >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz > >> > >> I think it would good if you could try it at least once, because I > think any > >> other troubleshooting will boil down to looking at the fixes this > version > >> contains. > >> > >> Rainer > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > >>> Sent: Monday, November 02, 2009 11:52 PM > >>> To: rsyslog-users > >>> Subject: [rsyslog] Queuing bug (4.5.5) > >>> > >>> Greetings all, > >>> > >>> I appear to have run into an issue with 4.5.5 where queues are not > >>> being flushed in a timely manner. ?In this specific case, I have > data > >>> from last wednesday that is being written to disk in small chunks > >>> today since last wednesday. ?Unfortunately I cannot be too specific > in > >>> details, but here is what I can see: > >>> > >>> Using omfile+gzip, there appears to have been a decent burst in > >>> traffic sometime last wednesday. ?The rsyslog instance has grown to > >>> 1.8GB of memory and stayed there for a while. ?I have both main > >>> message and action queues defined using fixedarray, and I see no > >>> on-disk queues (appears to be entirely in memory). > >>> > >>> I've got templates writing out to filenames using an hourly > timestamp > >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of > those > >>> files, all of them less than 5k in size, there are between 5 and 15 > >>> lines of content, all of them from last wednesday, and within a few > >>> seconds of each other. ?It's almost like there was a significant > queue > >>> built up, the hour rolled over, and only the first block of lines > were > >>> pulled from the queue. ?Then the hour rolled over again, and > another > >>> block of lines were pulled from the queue. ?Then the next hour, > then > >>> another 5-15 lines. > >>> > >>> Is it possible that one of the queues still has a good chunk of > data > >>> built up, and is flushing it out very slowly? ?It hasn't been > >>> consistantly at the top of the hour either, and not every hour. > ?But > >>> the log lines themselves are sequentially written out, and usually > the > >>> lines are within a few seconds of each other. > >>> > >>> For example: > >>> > >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 > 18:46:34 > >>> and one 18:46:35 > >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, 14 > >>> lines Oct 21 18:46:36 > >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, 4 > >>> lines Oct 21 18:46:37 > >>> > >>> Thoughts? > >>> -Aaron > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > From epiphani at gmail.com Tue Nov 3 18:27:07 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 3 Nov 2009 12:27:07 -0500 Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> Message-ID: Ok - I captured an strace. This appears to be the action thread. This specific set of calls took minutes. I'll re-run this with -t and/or -T. Note that this syslog instance is not actually receiving any data right now. -Aaron 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL 26365 <... futex resumed> ) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL 26365 <... futex resumed> ) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL 26365 <... futex resumed> ) = 0 26365 clock_gettime(CLOCK_REALTIME, 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 26365 <... futex resumed> ) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL 26365 <... futex resumed> ) = 0 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL 26365 <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable) 26365 clock_gettime(CLOCK_REALTIME, 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 26365 <... futex resumed> ) = 0 26365 clock_gettime(CLOCK_REALTIME, 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Tuesday, November 03, 2009 4:06 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >> >> > This is still taking place this hour - though I obviously can't >> > restart onto a newer version without losing this case. ?Is there >> > anything I can do in the current configuration to try and debug this >> > situation? >> >> if you can strace each thread for a few seconds it may give you an idea >> what's happening. > > This is a good suggestion. > > It would also potentially be enlightening to attach gdb to the process at > various points in time and get a backtrace from all running threads. > > Other than that, there is little you can do on a running version. If it were > compiled for debug, and the environment setup so that we could capture > stdout/stderr, sending SIGUSR2 would yield to much the same information as > the gdb backtrace BUT when runtime instrumentation is enabled, the build is > operating at a third to a quarter of its normal speed. > >> in the meantime you need to stop the system from getting further >> behind, >> can you redirect or reconfigure the senders so that they are not >> sending >> new logs to this box so that it can dig itself out (stopping the input >> may >> be enough by itself, rsyslog has historicly done a LOT of locking >> around >> the main queue, and if you have a full queue with more data trying to >> be >> delivered it can really slow things down. I wouldn't expect it to be >> this >> much, but it could be part of it) >> > > This may clean up things, but I really doubt it, given the magnitude of > delays. Also, Aaron runs 4.4.5, which already has lots of the locking > removed/restructure (not to compare with the recent v5-devel, but much more > effcient than early v4 and v3). > > Rainer >> David Lang >> >> > (We're up to 18:46:51 now!) >> > >> > -Aaron >> > >> > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >> > wrote: >> >> mhhh... This is obviously not intended behavior. There are some >> rate-limiting >> >> features that can deliberately slow down a queue, but I guess you >> have not >> >> configured these. So there either is a bug that activates them at >> some point >> >> during processing (e.g. an invalid memory access could do that) or >> there is >> >> some other bug that causes the dequeue to happen very slowly. In any >> case, it >> >> is hard to guess. >> >> >> >> Given the volume of the messages, a debug log probably does not >> help. We >> >> could gain a little insight in at least the queue sizes that rsyslog >> sees via >> >> imdiag plus the (very rudamentary) v5 debug front-end (it doesn't >> require the >> >> engine to be v5!). I would need to spend at least a little work on >> the >> >> front-end to enable remote access, but that's not really a problem. >> >> >> >> One other thing is that I am still holding a v4-beta with additional >> fixes as >> >> I didn't want to put these in the middle of some other debugging. >> However, >> >> these fixes address potential memory problems, so the most >> appropriate course >> >> of action would be to give that version at least a try. It needs to >> be >> >> released in the next days in any case. >> >> >> >> I have uploaded that (pre-4.5.6) version so that you can give it a >> try if you >> >> like: >> >> >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz >> >> >> >> I think it would good if you could try it at least once, because I >> think any >> >> other troubleshooting will boil down to looking at the fixes this >> version >> >> contains. >> >> >> >> Rainer >> >> >> >>> -----Original Message----- >> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> >>> Sent: Monday, November 02, 2009 11:52 PM >> >>> To: rsyslog-users >> >>> Subject: [rsyslog] Queuing bug (4.5.5) >> >>> >> >>> Greetings all, >> >>> >> >>> I appear to have run into an issue with 4.5.5 where queues are not >> >>> being flushed in a timely manner. ?In this specific case, I have >> data >> >>> from last wednesday that is being written to disk in small chunks >> >>> today since last wednesday. ?Unfortunately I cannot be too specific >> in >> >>> details, but here is what I can see: >> >>> >> >>> Using omfile+gzip, there appears to have been a decent burst in >> >>> traffic sometime last wednesday. ?The rsyslog instance has grown to >> >>> 1.8GB of memory and stayed there for a while. ?I have both main >> >>> message and action queues defined using fixedarray, and I see no >> >>> on-disk queues (appears to be entirely in memory). >> >>> >> >>> I've got templates writing out to filenames using an hourly >> timestamp >> >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of >> those >> >>> files, all of them less than 5k in size, there are between 5 and 15 >> >>> lines of content, all of them from last wednesday, and within a few >> >>> seconds of each other. ?It's almost like there was a significant >> queue >> >>> built up, the hour rolled over, and only the first block of lines >> were >> >>> pulled from the queue. ?Then the hour rolled over again, and >> another >> >>> block of lines were pulled from the queue. ?Then the next hour, >> then >> >>> another 5-15 lines. >> >>> >> >>> Is it possible that one of the queues still has a good chunk of >> data >> >>> built up, and is flushing it out very slowly? ?It hasn't been >> >>> consistantly at the top of the hour either, and not every hour. >> ?But >> >>> the log lines themselves are sequentially written out, and usually >> the >> >>> lines are within a few seconds of each other. >> >>> >> >>> For example: >> >>> >> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >> 18:46:34 >> >>> and one 18:46:35 >> >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, 14 >> >>> lines Oct 21 18:46:36 >> >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, 4 >> >>> lines Oct 21 18:46:37 >> >>> >> >>> Thoughts? >> >>> -Aaron >> >>> _______________________________________________ >> >>> rsyslog mailing list >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>> http://www.rsyslog.com >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com >> >> >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 3 18:30:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 18:30:17 +0100 Subject: [rsyslog] Queuing bug (4.5.5) References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> first shot at the data: this looks like a full queue where the enqueue operation is waiting on a drain that does not happen (fast enough). Of course, that doesn't explain why this happens... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Tuesday, November 03, 2009 6:27 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) > > Ok - I captured an strace. This appears to be the action thread. > This specific set of calls took minutes. I'll re-run this with -t > and/or -T. > > Note that this syslog instance is not actually receiving any data right > now. > > -Aaron > > 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- > 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL > 26365 <... futex resumed> ) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} > 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) > 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL > 26365 <... futex resumed> ) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} > 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) > 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL > 26365 <... futex resumed> ) = 0 > 26365 clock_gettime(CLOCK_REALTIME, > 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 > 26365 <... futex resumed> ) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} > 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) > 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL > 26365 <... futex resumed> ) = 0 > 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL > 26365 <... futex resumed> ) = -1 EAGAIN (Resource > temporarily unavailable) > 26365 clock_gettime(CLOCK_REALTIME, > 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 > 26365 <... futex resumed> ) = 0 > 26365 clock_gettime(CLOCK_REALTIME, > 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} > 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) > 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL > > > On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards > wrote: > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >> Sent: Tuesday, November 03, 2009 4:06 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Queuing bug (4.5.5) > >> > >> On Tue, 3 Nov 2009, Aaron Wiebe wrote: > >> > >> > This is still taking place this hour - though I obviously can't > >> > restart onto a newer version without losing this case. ?Is there > >> > anything I can do in the current configuration to try and debug > this > >> > situation? > >> > >> if you can strace each thread for a few seconds it may give you an > idea > >> what's happening. > > > > This is a good suggestion. > > > > It would also potentially be enlightening to attach gdb to the > process at > > various points in time and get a backtrace from all running threads. > > > > Other than that, there is little you can do on a running version. If > it were > > compiled for debug, and the environment setup so that we could > capture > > stdout/stderr, sending SIGUSR2 would yield to much the same > information as > > the gdb backtrace BUT when runtime instrumentation is enabled, the > build is > > operating at a third to a quarter of its normal speed. > > > >> in the meantime you need to stop the system from getting further > >> behind, > >> can you redirect or reconfigure the senders so that they are not > >> sending > >> new logs to this box so that it can dig itself out (stopping the > input > >> may > >> be enough by itself, rsyslog has historicly done a LOT of locking > >> around > >> the main queue, and if you have a full queue with more data trying > to > >> be > >> delivered it can really slow things down. I wouldn't expect it to be > >> this > >> much, but it could be part of it) > >> > > > > This may clean up things, but I really doubt it, given the magnitude > of > > delays. Also, Aaron runs 4.4.5, which already has lots of the locking > > removed/restructure (not to compare with the recent v5-devel, but > much more > > effcient than early v4 and v3). > > > > Rainer > >> David Lang > >> > >> > (We're up to 18:46:51 now!) > >> > > >> > -Aaron > >> > > >> > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards > >> > wrote: > >> >> mhhh... This is obviously not intended behavior. There are some > >> rate-limiting > >> >> features that can deliberately slow down a queue, but I guess you > >> have not > >> >> configured these. So there either is a bug that activates them at > >> some point > >> >> during processing (e.g. an invalid memory access could do that) > or > >> there is > >> >> some other bug that causes the dequeue to happen very slowly. In > any > >> case, it > >> >> is hard to guess. > >> >> > >> >> Given the volume of the messages, a debug log probably does not > >> help. We > >> >> could gain a little insight in at least the queue sizes that > rsyslog > >> sees via > >> >> imdiag plus the (very rudamentary) v5 debug front-end (it doesn't > >> require the > >> >> engine to be v5!). I would need to spend at least a little work > on > >> the > >> >> front-end to enable remote access, but that's not really a > problem. > >> >> > >> >> One other thing is that I am still holding a v4-beta with > additional > >> fixes as > >> >> I didn't want to put these in the middle of some other debugging. > >> However, > >> >> these fixes address potential memory problems, so the most > >> appropriate course > >> >> of action would be to give that version at least a try. It needs > to > >> be > >> >> released in the next days in any case. > >> >> > >> >> I have uploaded that (pre-4.5.6) version so that you can give it > a > >> try if you > >> >> like: > >> >> > >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz > >> >> > >> >> I think it would good if you could try it at least once, because > I > >> think any > >> >> other troubleshooting will boil down to looking at the fixes this > >> version > >> >> contains. > >> >> > >> >> Rainer > >> >> > >> >>> -----Original Message----- > >> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > >> >>> Sent: Monday, November 02, 2009 11:52 PM > >> >>> To: rsyslog-users > >> >>> Subject: [rsyslog] Queuing bug (4.5.5) > >> >>> > >> >>> Greetings all, > >> >>> > >> >>> I appear to have run into an issue with 4.5.5 where queues are > not > >> >>> being flushed in a timely manner. ?In this specific case, I have > >> data > >> >>> from last wednesday that is being written to disk in small > chunks > >> >>> today since last wednesday. ?Unfortunately I cannot be too > specific > >> in > >> >>> details, but here is what I can see: > >> >>> > >> >>> Using omfile+gzip, there appears to have been a decent burst in > >> >>> traffic sometime last wednesday. ?The rsyslog instance has grown > to > >> >>> 1.8GB of memory and stayed there for a while. ?I have both main > >> >>> message and action queues defined using fixedarray, and I see no > >> >>> on-disk queues (appears to be entirely in memory). > >> >>> > >> >>> I've got templates writing out to filenames using an hourly > >> timestamp > >> >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of > >> those > >> >>> files, all of them less than 5k in size, there are between 5 and > 15 > >> >>> lines of content, all of them from last wednesday, and within a > few > >> >>> seconds of each other. ?It's almost like there was a significant > >> queue > >> >>> built up, the hour rolled over, and only the first block of > lines > >> were > >> >>> pulled from the queue. ?Then the hour rolled over again, and > >> another > >> >>> block of lines were pulled from the queue. ?Then the next hour, > >> then > >> >>> another 5-15 lines. > >> >>> > >> >>> Is it possible that one of the queues still has a good chunk of > >> data > >> >>> built up, and is flushing it out very slowly? ?It hasn't been > >> >>> consistantly at the top of the hour either, and not every hour. > >> ?But > >> >>> the log lines themselves are sequentially written out, and > usually > >> the > >> >>> lines are within a few seconds of each other. > >> >>> > >> >>> For example: > >> >>> > >> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 > >> 18:46:34 > >> >>> and one 18:46:35 > >> >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, > 14 > >> >>> lines Oct 21 18:46:36 > >> >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, > 4 > >> >>> lines Oct 21 18:46:37 > >> >>> > >> >>> Thoughts? > >> >>> -Aaron > >> >>> _______________________________________________ > >> >>> rsyslog mailing list > >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >>> http://www.rsyslog.com > >> >> _______________________________________________ > >> >> rsyslog mailing list > >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> http://www.rsyslog.com > >> >> > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com > >> > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From epiphani at gmail.com Tue Nov 3 18:33:40 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 3 Nov 2009 12:33:40 -0500 Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> Message-ID: Here's one with times: 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, 3889000}) = 0 <0.000047> 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, 4147000}) = 0 <0.000013> 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) <2.002462> 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 250) = 250 <0.000305> 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, 7292000}) = 0 <0.000027> 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, 414932000}) = 0 <0.000048> 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, 415191000}) = 0 <0.000013> 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) <2.002220> 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 197) = 197 <0.000279> 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, 417992000}) = 0 <0.000027> 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, 608226000}) = 0 <0.000047> 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, 608486000}) = 0 <0.000014> 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} It looks like the locks are waiting a -very- long time. -Aaron On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards wrote: > first shot at the data: this looks like a full queue where the enqueue > operation is waiting on a drain that does not happen (fast enough). Of > course, that doesn't explain why this happens... > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> Sent: Tuesday, November 03, 2009 6:27 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> Ok - I captured an strace. ?This appears to be the action thread. >> This specific set of calls took minutes. ?I'll re-run this with -t >> and/or -T. >> >> Note that this syslog instance is not actually receiving any data right >> now. >> >> -Aaron >> >> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> timed out) >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> timed out) >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, ? >> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> timed out) >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource >> temporarily unavailable) >> 26365 clock_gettime(CLOCK_REALTIME, ? >> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, ? >> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> timed out) >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL >> >> >> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards >> wrote: >> >> -----Original Message----- >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >> Sent: Tuesday, November 03, 2009 4:06 PM >> >> To: rsyslog-users >> >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> >> >> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >> >> >> >> > This is still taking place this hour - though I obviously can't >> >> > restart onto a newer version without losing this case. ?Is there >> >> > anything I can do in the current configuration to try and debug >> this >> >> > situation? >> >> >> >> if you can strace each thread for a few seconds it may give you an >> idea >> >> what's happening. >> > >> > This is a good suggestion. >> > >> > It would also potentially be enlightening to attach gdb to the >> process at >> > various points in time and get a backtrace from all running threads. >> > >> > Other than that, there is little you can do on a running version. If >> it were >> > compiled for debug, and the environment setup so that we could >> capture >> > stdout/stderr, sending SIGUSR2 would yield to much the same >> information as >> > the gdb backtrace BUT when runtime instrumentation is enabled, the >> build is >> > operating at a third to a quarter of its normal speed. >> > >> >> in the meantime you need to stop the system from getting further >> >> behind, >> >> can you redirect or reconfigure the senders so that they are not >> >> sending >> >> new logs to this box so that it can dig itself out (stopping the >> input >> >> may >> >> be enough by itself, rsyslog has historicly done a LOT of locking >> >> around >> >> the main queue, and if you have a full queue with more data trying >> to >> >> be >> >> delivered it can really slow things down. I wouldn't expect it to be >> >> this >> >> much, but it could be part of it) >> >> >> > >> > This may clean up things, but I really doubt it, given the magnitude >> of >> > delays. Also, Aaron runs 4.4.5, which already has lots of the locking >> > removed/restructure (not to compare with the recent v5-devel, but >> much more >> > effcient than early v4 and v3). >> > >> > Rainer >> >> David Lang >> >> >> >> > (We're up to 18:46:51 now!) >> >> > >> >> > -Aaron >> >> > >> >> > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >> >> > wrote: >> >> >> mhhh... This is obviously not intended behavior. There are some >> >> rate-limiting >> >> >> features that can deliberately slow down a queue, but I guess you >> >> have not >> >> >> configured these. So there either is a bug that activates them at >> >> some point >> >> >> during processing (e.g. an invalid memory access could do that) >> or >> >> there is >> >> >> some other bug that causes the dequeue to happen very slowly. In >> any >> >> case, it >> >> >> is hard to guess. >> >> >> >> >> >> Given the volume of the messages, a debug log probably does not >> >> help. We >> >> >> could gain a little insight in at least the queue sizes that >> rsyslog >> >> sees via >> >> >> imdiag plus the (very rudamentary) v5 debug front-end (it doesn't >> >> require the >> >> >> engine to be v5!). I would need to spend at least a little work >> on >> >> the >> >> >> front-end to enable remote access, but that's not really a >> problem. >> >> >> >> >> >> One other thing is that I am still holding a v4-beta with >> additional >> >> fixes as >> >> >> I didn't want to put these in the middle of some other debugging. >> >> However, >> >> >> these fixes address potential memory problems, so the most >> >> appropriate course >> >> >> of action would be to give that version at least a try. It needs >> to >> >> be >> >> >> released in the next days in any case. >> >> >> >> >> >> I have uploaded that (pre-4.5.6) version so that you can give it >> a >> >> try if you >> >> >> like: >> >> >> >> >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz >> >> >> >> >> >> I think it would good if you could try it at least once, because >> I >> >> think any >> >> >> other troubleshooting will boil down to looking at the fixes this >> >> version >> >> >> contains. >> >> >> >> >> >> Rainer >> >> >> >> >> >>> -----Original Message----- >> >> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> >> >>> Sent: Monday, November 02, 2009 11:52 PM >> >> >>> To: rsyslog-users >> >> >>> Subject: [rsyslog] Queuing bug (4.5.5) >> >> >>> >> >> >>> Greetings all, >> >> >>> >> >> >>> I appear to have run into an issue with 4.5.5 where queues are >> not >> >> >>> being flushed in a timely manner. ?In this specific case, I have >> >> data >> >> >>> from last wednesday that is being written to disk in small >> chunks >> >> >>> today since last wednesday. ?Unfortunately I cannot be too >> specific >> >> in >> >> >>> details, but here is what I can see: >> >> >>> >> >> >>> Using omfile+gzip, there appears to have been a decent burst in >> >> >>> traffic sometime last wednesday. ?The rsyslog instance has grown >> to >> >> >>> 1.8GB of memory and stayed there for a while. ?I have both main >> >> >>> message and action queues defined using fixedarray, and I see no >> >> >>> on-disk queues (appears to be entirely in memory). >> >> >>> >> >> >>> I've got templates writing out to filenames using an hourly >> >> timestamp >> >> >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of >> >> those >> >> >>> files, all of them less than 5k in size, there are between 5 and >> 15 >> >> >>> lines of content, all of them from last wednesday, and within a >> few >> >> >>> seconds of each other. ?It's almost like there was a significant >> >> queue >> >> >>> built up, the hour rolled over, and only the first block of >> lines >> >> were >> >> >>> pulled from the queue. ?Then the hour rolled over again, and >> >> another >> >> >>> block of lines were pulled from the queue. ?Then the next hour, >> >> then >> >> >>> another 5-15 lines. >> >> >>> >> >> >>> Is it possible that one of the queues still has a good chunk of >> >> data >> >> >>> built up, and is flushing it out very slowly? ?It hasn't been >> >> >>> consistantly at the top of the hour either, and not every hour. >> >> ?But >> >> >>> the log lines themselves are sequentially written out, and >> usually >> >> the >> >> >>> lines are within a few seconds of each other. >> >> >>> >> >> >>> For example: >> >> >>> >> >> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >> >> 18:46:34 >> >> >>> and one 18:46:35 >> >> >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, >> 14 >> >> >>> lines Oct 21 18:46:36 >> >> >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, >> 4 >> >> >>> lines Oct 21 18:46:37 >> >> >>> >> >> >>> Thoughts? >> >> >>> -Aaron >> >> >>> _______________________________________________ >> >> >>> rsyslog mailing list >> >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >>> http://www.rsyslog.com >> >> >> _______________________________________________ >> >> >> rsyslog mailing list >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> http://www.rsyslog.com >> >> >> >> >> > _______________________________________________ >> >> > rsyslog mailing list >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> > http://www.rsyslog.com >> >> > >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 3 18:38:40 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 18:38:40 +0100 Subject: [rsyslog] Queuing bug (4.5.5) References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> It really looks like a full queue. The 2-second wait is the default wait interval before discarding a message when a queue is full. So for some reason the action queue seems to think it is full, so that the main queue worker can no longer add any data to it. Out of the strace, I unfortunately do not see why this happens. If possible, it would be good to try the 4.5.6 release, as this may actually be caused by the memory bug that is solved in that release. If it doesn't help, we would need to think about a way to get more in-depth info when the problem happens. I have an idea, but need to check if it can be done. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Tuesday, November 03, 2009 6:34 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) > > Here's one with times: > > 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- > 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL > > 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> > 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, > 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, > 3889000}) = 0 <0.000047> > 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 > 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> > 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, > 4147000}) = 0 <0.000013> > 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} > > 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) <2.002462> > 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 > WT-GARULF6"..., 250) = 250 <0.000305> > 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> > 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, > 7292000}) = 0 <0.000027> > 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL > > 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> > 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, > 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, > 414932000}) = 0 <0.000048> > 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 > 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> > 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, > 415191000}) = 0 <0.000013> > 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} > > 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) <2.002220> > 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 > WT-GARULF6"..., 197) = 197 <0.000279> > 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> > 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, > 417992000}) = 0 <0.000027> > 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL > > 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> > 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, > 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, > 608226000}) = 0 <0.000047> > 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 > 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> > 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, > 608486000}) = 0 <0.000014> > 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} > > > It looks like the locks are waiting a -very- long time. > > -Aaron > > On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards > wrote: > > first shot at the data: this looks like a full queue where the > enqueue > > operation is waiting on a drain that does not happen (fast enough). > Of > > course, that doesn't explain why this happens... > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > >> Sent: Tuesday, November 03, 2009 6:27 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Queuing bug (4.5.5) > >> > >> Ok - I captured an strace. ?This appears to be the action thread. > >> This specific set of calls took minutes. ?I'll re-run this with -t > >> and/or -T. > >> > >> Note that this syslog instance is not actually receiving any data > right > >> now. > >> > >> -Aaron > >> > >> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- > >> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} ...> > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection > >> timed out) > >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} ...> > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection > >> timed out) > >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, ? > >> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} ...> > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection > >> timed out) > >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource > >> temporarily unavailable) > >> 26365 clock_gettime(CLOCK_REALTIME, ? > >> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, ? > >> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} ...> > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection > >> timed out) > >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL > >> > >> > >> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards > >> wrote: > >> >> -----Original Message----- > >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >> >> Sent: Tuesday, November 03, 2009 4:06 PM > >> >> To: rsyslog-users > >> >> Subject: Re: [rsyslog] Queuing bug (4.5.5) > >> >> > >> >> On Tue, 3 Nov 2009, Aaron Wiebe wrote: > >> >> > >> >> > This is still taking place this hour - though I obviously can't > >> >> > restart onto a newer version without losing this case. ?Is > there > >> >> > anything I can do in the current configuration to try and debug > >> this > >> >> > situation? > >> >> > >> >> if you can strace each thread for a few seconds it may give you > an > >> idea > >> >> what's happening. > >> > > >> > This is a good suggestion. > >> > > >> > It would also potentially be enlightening to attach gdb to the > >> process at > >> > various points in time and get a backtrace from all running > threads. > >> > > >> > Other than that, there is little you can do on a running version. > If > >> it were > >> > compiled for debug, and the environment setup so that we could > >> capture > >> > stdout/stderr, sending SIGUSR2 would yield to much the same > >> information as > >> > the gdb backtrace BUT when runtime instrumentation is enabled, the > >> build is > >> > operating at a third to a quarter of its normal speed. > >> > > >> >> in the meantime you need to stop the system from getting further > >> >> behind, > >> >> can you redirect or reconfigure the senders so that they are not > >> >> sending > >> >> new logs to this box so that it can dig itself out (stopping the > >> input > >> >> may > >> >> be enough by itself, rsyslog has historicly done a LOT of locking > >> >> around > >> >> the main queue, and if you have a full queue with more data > trying > >> to > >> >> be > >> >> delivered it can really slow things down. I wouldn't expect it to > be > >> >> this > >> >> much, but it could be part of it) > >> >> > >> > > >> > This may clean up things, but I really doubt it, given the > magnitude > >> of > >> > delays. Also, Aaron runs 4.4.5, which already has lots of the > locking > >> > removed/restructure (not to compare with the recent v5-devel, but > >> much more > >> > effcient than early v4 and v3). > >> > > >> > Rainer > >> >> David Lang > >> >> > >> >> > (We're up to 18:46:51 now!) > >> >> > > >> >> > -Aaron > >> >> > > >> >> > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards > >> >> > wrote: > >> >> >> mhhh... This is obviously not intended behavior. There are > some > >> >> rate-limiting > >> >> >> features that can deliberately slow down a queue, but I guess > you > >> >> have not > >> >> >> configured these. So there either is a bug that activates them > at > >> >> some point > >> >> >> during processing (e.g. an invalid memory access could do > that) > >> or > >> >> there is > >> >> >> some other bug that causes the dequeue to happen very slowly. > In > >> any > >> >> case, it > >> >> >> is hard to guess. > >> >> >> > >> >> >> Given the volume of the messages, a debug log probably does > not > >> >> help. We > >> >> >> could gain a little insight in at least the queue sizes that > >> rsyslog > >> >> sees via > >> >> >> imdiag plus the (very rudamentary) v5 debug front-end (it > doesn't > >> >> require the > >> >> >> engine to be v5!). I would need to spend at least a little > work > >> on > >> >> the > >> >> >> front-end to enable remote access, but that's not really a > >> problem. > >> >> >> > >> >> >> One other thing is that I am still holding a v4-beta with > >> additional > >> >> fixes as > >> >> >> I didn't want to put these in the middle of some other > debugging. > >> >> However, > >> >> >> these fixes address potential memory problems, so the most > >> >> appropriate course > >> >> >> of action would be to give that version at least a try. It > needs > >> to > >> >> be > >> >> >> released in the next days in any case. > >> >> >> > >> >> >> I have uploaded that (pre-4.5.6) version so that you can give > it > >> a > >> >> try if you > >> >> >> like: > >> >> >> > >> >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog- > 4.5.6.tar.gz > >> >> >> > >> >> >> I think it would good if you could try it at least once, > because > >> I > >> >> think any > >> >> >> other troubleshooting will boil down to looking at the fixes > this > >> >> version > >> >> >> contains. > >> >> >> > >> >> >> Rainer > >> >> >> > >> >> >>> -----Original Message----- > >> >> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> >> >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > >> >> >>> Sent: Monday, November 02, 2009 11:52 PM > >> >> >>> To: rsyslog-users > >> >> >>> Subject: [rsyslog] Queuing bug (4.5.5) > >> >> >>> > >> >> >>> Greetings all, > >> >> >>> > >> >> >>> I appear to have run into an issue with 4.5.5 where queues > are > >> not > >> >> >>> being flushed in a timely manner. ?In this specific case, I > have > >> >> data > >> >> >>> from last wednesday that is being written to disk in small > >> chunks > >> >> >>> today since last wednesday. ?Unfortunately I cannot be too > >> specific > >> >> in > >> >> >>> details, but here is what I can see: > >> >> >>> > >> >> >>> Using omfile+gzip, there appears to have been a decent burst > in > >> >> >>> traffic sometime last wednesday. ?The rsyslog instance has > grown > >> to > >> >> >>> 1.8GB of memory and stayed there for a while. ?I have both > main > >> >> >>> message and action queues defined using fixedarray, and I see > no > >> >> >>> on-disk queues (appears to be entirely in memory). > >> >> >>> > >> >> >>> I've got templates writing out to filenames using an hourly > >> >> timestamp > >> >> >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some > of > >> >> those > >> >> >>> files, all of them less than 5k in size, there are between 5 > and > >> 15 > >> >> >>> lines of content, all of them from last wednesday, and within > a > >> few > >> >> >>> seconds of each other. ?It's almost like there was a > significant > >> >> queue > >> >> >>> built up, the hour rolled over, and only the first block of > >> lines > >> >> were > >> >> >>> pulled from the queue. ?Then the hour rolled over again, and > >> >> another > >> >> >>> block of lines were pulled from the queue. ?Then the next > hour, > >> >> then > >> >> >>> another 5-15 lines. > >> >> >>> > >> >> >>> Is it possible that one of the queues still has a good chunk > of > >> >> data > >> >> >>> built up, and is flushing it out very slowly? ?It hasn't been > >> >> >>> consistantly at the top of the hour either, and not every > hour. > >> >> ?But > >> >> >>> the log lines themselves are sequentially written out, and > >> usually > >> >> the > >> >> >>> lines are within a few seconds of each other. > >> >> >>> > >> >> >>> For example: > >> >> >>> > >> >> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 > >> >> 18:46:34 > >> >> >>> and one 18:46:35 > >> >> >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 > 18:46:35, > >> 14 > >> >> >>> lines Oct 21 18:46:36 > >> >> >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 > 18:46:36, > >> 4 > >> >> >>> lines Oct 21 18:46:37 > >> >> >>> > >> >> >>> Thoughts? > >> >> >>> -Aaron > >> >> >>> _______________________________________________ > >> >> >>> rsyslog mailing list > >> >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> >>> http://www.rsyslog.com > >> >> >> _______________________________________________ > >> >> >> rsyslog mailing list > >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> >> http://www.rsyslog.com > >> >> >> > >> >> > _______________________________________________ > >> >> > rsyslog mailing list > >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> > http://www.rsyslog.com > >> >> > > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com > >> > > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From epiphani at gmail.com Tue Nov 3 18:41:46 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 3 Nov 2009 12:41:46 -0500 Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> Message-ID: I'll see what I can do. I'm actually somewhat tied up with other stuff this week, but I may be able to find time to update. -Aaron On Tue, Nov 3, 2009 at 12:38 PM, Rainer Gerhards wrote: > It really looks like a full queue. The 2-second wait is the default wait > interval before discarding a message when a queue is full. So for some reason > the action queue seems to think it is full, so that the main queue worker can > no longer add any data to it. Out of the strace, I unfortunately do not see > why this happens. > > If possible, it would be good to try the 4.5.6 release, as this may actually > be caused by the memory bug that is solved in that release. If it doesn't > help, we would need to think about a way to get more in-depth info when the > problem happens. I have an idea, but need to check if it can be done. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> Sent: Tuesday, November 03, 2009 6:34 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> Here's one with times: >> >> 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >> 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL >> >> 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> >> 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, ? >> 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, >> 3889000}) = 0 <0.000047> >> 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> >> 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, >> 4147000}) = 0 <0.000013> >> 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} >> >> 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection >> timed out) <2.002462> >> 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 >> WT-GARULF6"..., 250) = 250 <0.000305> >> 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> >> 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, >> 7292000}) = 0 <0.000027> >> 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL >> >> 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> >> 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, ? >> 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, >> 414932000}) = 0 <0.000048> >> 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> >> 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, >> 415191000}) = 0 <0.000013> >> 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} >> >> 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection >> timed out) <2.002220> >> 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 >> WT-GARULF6"..., 197) = 197 <0.000279> >> 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> >> 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, >> 417992000}) = 0 <0.000027> >> 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL >> >> 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> >> 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, ? >> 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, >> 608226000}) = 0 <0.000047> >> 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> >> 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, >> 608486000}) = 0 <0.000014> >> 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} >> >> >> It looks like the locks are waiting a -very- long time. >> >> -Aaron >> >> On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards >> wrote: >> > first shot at the data: this looks like a full queue where the >> enqueue >> > operation is waiting on a drain that does not happen (fast enough). >> Of >> > course, that doesn't explain why this happens... >> > >> > Rainer >> > >> >> -----Original Message----- >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> >> Sent: Tuesday, November 03, 2009 6:27 PM >> >> To: rsyslog-users >> >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> >> >> Ok - I captured an strace. ?This appears to be the action thread. >> >> This specific set of calls took minutes. ?I'll re-run this with -t >> >> and/or -T. >> >> >> >> Note that this syslog instance is not actually receiving any data >> right >> >> now. >> >> >> >> -Aaron >> >> >> >> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} > ...> >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> >> timed out) >> >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} > ...> >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> >> timed out) >> >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, ? >> >> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} > ...> >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> >> timed out) >> >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource >> >> temporarily unavailable) >> >> 26365 clock_gettime(CLOCK_REALTIME, ? >> >> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, ? >> >> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} > ...> >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> >> timed out) >> >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL >> >> >> >> >> >> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards >> >> wrote: >> >> >> -----Original Message----- >> >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >> >> Sent: Tuesday, November 03, 2009 4:06 PM >> >> >> To: rsyslog-users >> >> >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> >> >> >> >> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >> >> >> >> >> >> > This is still taking place this hour - though I obviously can't >> >> >> > restart onto a newer version without losing this case. ?Is >> there >> >> >> > anything I can do in the current configuration to try and debug >> >> this >> >> >> > situation? >> >> >> >> >> >> if you can strace each thread for a few seconds it may give you >> an >> >> idea >> >> >> what's happening. >> >> > >> >> > This is a good suggestion. >> >> > >> >> > It would also potentially be enlightening to attach gdb to the >> >> process at >> >> > various points in time and get a backtrace from all running >> threads. >> >> > >> >> > Other than that, there is little you can do on a running version. >> If >> >> it were >> >> > compiled for debug, and the environment setup so that we could >> >> capture >> >> > stdout/stderr, sending SIGUSR2 would yield to much the same >> >> information as >> >> > the gdb backtrace BUT when runtime instrumentation is enabled, the >> >> build is >> >> > operating at a third to a quarter of its normal speed. >> >> > >> >> >> in the meantime you need to stop the system from getting further >> >> >> behind, >> >> >> can you redirect or reconfigure the senders so that they are not >> >> >> sending >> >> >> new logs to this box so that it can dig itself out (stopping the >> >> input >> >> >> may >> >> >> be enough by itself, rsyslog has historicly done a LOT of locking >> >> >> around >> >> >> the main queue, and if you have a full queue with more data >> trying >> >> to >> >> >> be >> >> >> delivered it can really slow things down. I wouldn't expect it to >> be >> >> >> this >> >> >> much, but it could be part of it) >> >> >> >> >> > >> >> > This may clean up things, but I really doubt it, given the >> magnitude >> >> of >> >> > delays. Also, Aaron runs 4.4.5, which already has lots of the >> locking >> >> > removed/restructure (not to compare with the recent v5-devel, but >> >> much more >> >> > effcient than early v4 and v3). >> >> > >> >> > Rainer >> >> >> David Lang >> >> >> >> >> >> > (We're up to 18:46:51 now!) >> >> >> > >> >> >> > -Aaron >> >> >> > >> >> >> > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >> >> >> > wrote: >> >> >> >> mhhh... This is obviously not intended behavior. There are >> some >> >> >> rate-limiting >> >> >> >> features that can deliberately slow down a queue, but I guess >> you >> >> >> have not >> >> >> >> configured these. So there either is a bug that activates them >> at >> >> >> some point >> >> >> >> during processing (e.g. an invalid memory access could do >> that) >> >> or >> >> >> there is >> >> >> >> some other bug that causes the dequeue to happen very slowly. >> In >> >> any >> >> >> case, it >> >> >> >> is hard to guess. >> >> >> >> >> >> >> >> Given the volume of the messages, a debug log probably does >> not >> >> >> help. We >> >> >> >> could gain a little insight in at least the queue sizes that >> >> rsyslog >> >> >> sees via >> >> >> >> imdiag plus the (very rudamentary) v5 debug front-end (it >> doesn't >> >> >> require the >> >> >> >> engine to be v5!). I would need to spend at least a little >> work >> >> on >> >> >> the >> >> >> >> front-end to enable remote access, but that's not really a >> >> problem. >> >> >> >> >> >> >> >> One other thing is that I am still holding a v4-beta with >> >> additional >> >> >> fixes as >> >> >> >> I didn't want to put these in the middle of some other >> debugging. >> >> >> However, >> >> >> >> these fixes address potential memory problems, so the most >> >> >> appropriate course >> >> >> >> of action would be to give that version at least a try. It >> needs >> >> to >> >> >> be >> >> >> >> released in the next days in any case. >> >> >> >> >> >> >> >> I have uploaded that (pre-4.5.6) version so that you can give >> it >> >> a >> >> >> try if you >> >> >> >> like: >> >> >> >> >> >> >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog- >> 4.5.6.tar.gz >> >> >> >> >> >> >> >> I think it would good if you could try it at least once, >> because >> >> I >> >> >> think any >> >> >> >> other troubleshooting will boil down to looking at the fixes >> this >> >> >> version >> >> >> >> contains. >> >> >> >> >> >> >> >> Rainer >> >> >> >> >> >> >> >>> -----Original Message----- >> >> >> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> >> >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> >> >> >>> Sent: Monday, November 02, 2009 11:52 PM >> >> >> >>> To: rsyslog-users >> >> >> >>> Subject: [rsyslog] Queuing bug (4.5.5) >> >> >> >>> >> >> >> >>> Greetings all, >> >> >> >>> >> >> >> >>> I appear to have run into an issue with 4.5.5 where queues >> are >> >> not >> >> >> >>> being flushed in a timely manner. ?In this specific case, I >> have >> >> >> data >> >> >> >>> from last wednesday that is being written to disk in small >> >> chunks >> >> >> >>> today since last wednesday. ?Unfortunately I cannot be too >> >> specific >> >> >> in >> >> >> >>> details, but here is what I can see: >> >> >> >>> >> >> >> >>> Using omfile+gzip, there appears to have been a decent burst >> in >> >> >> >>> traffic sometime last wednesday. ?The rsyslog instance has >> grown >> >> to >> >> >> >>> 1.8GB of memory and stayed there for a while. ?I have both >> main >> >> >> >>> message and action queues defined using fixedarray, and I see >> no >> >> >> >>> on-disk queues (appears to be entirely in memory). >> >> >> >>> >> >> >> >>> I've got templates writing out to filenames using an hourly >> >> >> timestamp >> >> >> >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some >> of >> >> >> those >> >> >> >>> files, all of them less than 5k in size, there are between 5 >> and >> >> 15 >> >> >> >>> lines of content, all of them from last wednesday, and within >> a >> >> few >> >> >> >>> seconds of each other. ?It's almost like there was a >> significant >> >> >> queue >> >> >> >>> built up, the hour rolled over, and only the first block of >> >> lines >> >> >> were >> >> >> >>> pulled from the queue. ?Then the hour rolled over again, and >> >> >> another >> >> >> >>> block of lines were pulled from the queue. ?Then the next >> hour, >> >> >> then >> >> >> >>> another 5-15 lines. >> >> >> >>> >> >> >> >>> Is it possible that one of the queues still has a good chunk >> of >> >> >> data >> >> >> >>> built up, and is flushing it out very slowly? ?It hasn't been >> >> >> >>> consistantly at the top of the hour either, and not every >> hour. >> >> >> ?But >> >> >> >>> the log lines themselves are sequentially written out, and >> >> usually >> >> >> the >> >> >> >>> lines are within a few seconds of each other. >> >> >> >>> >> >> >> >>> For example: >> >> >> >>> >> >> >> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >> >> >> 18:46:34 >> >> >> >>> and one 18:46:35 >> >> >> >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 >> 18:46:35, >> >> 14 >> >> >> >>> lines Oct 21 18:46:36 >> >> >> >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 >> 18:46:36, >> >> 4 >> >> >> >>> lines Oct 21 18:46:37 >> >> >> >>> >> >> >> >>> Thoughts? >> >> >> >>> -Aaron >> >> >> >>> _______________________________________________ >> >> >> >>> rsyslog mailing list >> >> >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> >>> http://www.rsyslog.com >> >> >> >> _______________________________________________ >> >> >> >> rsyslog mailing list >> >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> >> http://www.rsyslog.com >> >> >> >> >> >> >> > _______________________________________________ >> >> >> > rsyslog mailing list >> >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> > http://www.rsyslog.com >> >> >> > >> >> > _______________________________________________ >> >> > rsyslog mailing list >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> > http://www.rsyslog.com >> >> > >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From Luis.Fernando.Munoz.Mejias at cern.ch Tue Nov 3 19:04:36 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?utf-8?q?Mu=C3=B1oz_Mej=C3=ADas?=) Date: Tue, 3 Nov 2009 19:04:36 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5? Message-ID: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> Hello, world I must be doing something really dumb. I need to send some output to a named pipe that I have already created, but rsyslog is refusing to open it. My configuration states: *.* |/var/log/external/all/unknown;RSYSLOG_FileFormat And, of course, the named pipe exists: # ls -Z /var/log/external/all/unknown prw-r----- root logreaders user_u:object_r:var_log_t /var/log/external/all/unknown But when I start the syslog daemon it complains with this message: rsyslogd-2039: Could no open output file '|/var/log/external/all/unknown' [try http://www.rsyslog.com/e/2039 ] It may be a SELinux problem, but the logs show nothing about any denied requests. And the context for the named pipe is the exact same I have for the log files, which lie in the same directory as well. Am I the only one experiencing such problems? Thanks. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From david at lang.hm Tue Nov 3 19:20:57 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 3 Nov 2009 10:20:57 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> Message-ID: what is the configuration? does it have seperate action queues defined? if you have some way to stop the input, it would let the main queue drop down from being full (and eliminate the input modules from needing to lock it to try and deliver messages) I did see some times during my testing of 4.x stuff where the throughput would drop drasticly when the queue was full and there ws more incoming traffic hammering on it. it would drop from 500K logs/min to ~3k logs/min with a trivial config, slowing down the input would let it get out of this mode and speed up again after a little while. so much other stuff was happening at the time that I don't think that I reported it. David Lang On Tue, 3 Nov 2009, Rainer Gerhards wrote: > Date: Tue, 3 Nov 2009 18:38:40 +0100 > From: Rainer Gerhards > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) > > It really looks like a full queue. The 2-second wait is the default wait > interval before discarding a message when a queue is full. So for some reason > the action queue seems to think it is full, so that the main queue worker can > no longer add any data to it. Out of the strace, I unfortunately do not see > why this happens. > > If possible, it would be good to try the 4.5.6 release, as this may actually > be caused by the memory bug that is solved in that release. If it doesn't > help, we would need to think about a way to get more in-depth info when the > problem happens. I have an idea, but need to check if it can be done. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> Sent: Tuesday, November 03, 2009 6:34 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> Here's one with times: >> >> 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >> 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL >> >> 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> >> 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, >> 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, >> 3889000}) = 0 <0.000047> >> 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> >> 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, >> 4147000}) = 0 <0.000013> >> 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} >> >> 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection >> timed out) <2.002462> >> 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 >> WT-GARULF6"..., 250) = 250 <0.000305> >> 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> >> 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, >> 7292000}) = 0 <0.000027> >> 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL >> >> 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> >> 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, >> 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, >> 414932000}) = 0 <0.000048> >> 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> >> 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, >> 415191000}) = 0 <0.000013> >> 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} >> >> 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection >> timed out) <2.002220> >> 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 >> WT-GARULF6"..., 197) = 197 <0.000279> >> 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> >> 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, >> 417992000}) = 0 <0.000027> >> 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL >> >> 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> >> 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, >> 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, >> 608226000}) = 0 <0.000047> >> 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> >> 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, >> 608486000}) = 0 <0.000014> >> 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} >> >> >> It looks like the locks are waiting a -very- long time. >> >> -Aaron >> >> On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards >> wrote: >>> first shot at the data: this looks like a full queue where the >> enqueue >>> operation is waiting on a drain that does not happen (fast enough). >> Of >>> course, that doesn't explain why this happens... >>> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>> Sent: Tuesday, November 03, 2009 6:27 PM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>> >>>> Ok - I captured an strace. ?This appears to be the action thread. >>>> This specific set of calls took minutes. ?I'll re-run this with -t >>>> and/or -T. >>>> >>>> Note that this syslog instance is not actually receiving any data >> right >>>> now. >>>> >>>> -Aaron >>>> >>>> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} > ...> >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>> timed out) >>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} > ...> >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>> timed out) >>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} > ...> >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>> timed out) >>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource >>>> temporarily unavailable) >>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} > ...> >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>> timed out) >>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL >>>> >>>> >>>> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards >>>> wrote: >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>> Sent: Tuesday, November 03, 2009 4:06 PM >>>>>> To: rsyslog-users >>>>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>>>> >>>>>> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >>>>>> >>>>>>> This is still taking place this hour - though I obviously can't >>>>>>> restart onto a newer version without losing this case. ?Is >> there >>>>>>> anything I can do in the current configuration to try and debug >>>> this >>>>>>> situation? >>>>>> >>>>>> if you can strace each thread for a few seconds it may give you >> an >>>> idea >>>>>> what's happening. >>>>> >>>>> This is a good suggestion. >>>>> >>>>> It would also potentially be enlightening to attach gdb to the >>>> process at >>>>> various points in time and get a backtrace from all running >> threads. >>>>> >>>>> Other than that, there is little you can do on a running version. >> If >>>> it were >>>>> compiled for debug, and the environment setup so that we could >>>> capture >>>>> stdout/stderr, sending SIGUSR2 would yield to much the same >>>> information as >>>>> the gdb backtrace BUT when runtime instrumentation is enabled, the >>>> build is >>>>> operating at a third to a quarter of its normal speed. >>>>> >>>>>> in the meantime you need to stop the system from getting further >>>>>> behind, >>>>>> can you redirect or reconfigure the senders so that they are not >>>>>> sending >>>>>> new logs to this box so that it can dig itself out (stopping the >>>> input >>>>>> may >>>>>> be enough by itself, rsyslog has historicly done a LOT of locking >>>>>> around >>>>>> the main queue, and if you have a full queue with more data >> trying >>>> to >>>>>> be >>>>>> delivered it can really slow things down. I wouldn't expect it to >> be >>>>>> this >>>>>> much, but it could be part of it) >>>>>> >>>>> >>>>> This may clean up things, but I really doubt it, given the >> magnitude >>>> of >>>>> delays. Also, Aaron runs 4.4.5, which already has lots of the >> locking >>>>> removed/restructure (not to compare with the recent v5-devel, but >>>> much more >>>>> effcient than early v4 and v3). >>>>> >>>>> Rainer >>>>>> David Lang >>>>>> >>>>>>> (We're up to 18:46:51 now!) >>>>>>> >>>>>>> -Aaron >>>>>>> >>>>>>> On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >>>>>>> wrote: >>>>>>>> mhhh... This is obviously not intended behavior. There are >> some >>>>>> rate-limiting >>>>>>>> features that can deliberately slow down a queue, but I guess >> you >>>>>> have not >>>>>>>> configured these. So there either is a bug that activates them >> at >>>>>> some point >>>>>>>> during processing (e.g. an invalid memory access could do >> that) >>>> or >>>>>> there is >>>>>>>> some other bug that causes the dequeue to happen very slowly. >> In >>>> any >>>>>> case, it >>>>>>>> is hard to guess. >>>>>>>> >>>>>>>> Given the volume of the messages, a debug log probably does >> not >>>>>> help. We >>>>>>>> could gain a little insight in at least the queue sizes that >>>> rsyslog >>>>>> sees via >>>>>>>> imdiag plus the (very rudamentary) v5 debug front-end (it >> doesn't >>>>>> require the >>>>>>>> engine to be v5!). I would need to spend at least a little >> work >>>> on >>>>>> the >>>>>>>> front-end to enable remote access, but that's not really a >>>> problem. >>>>>>>> >>>>>>>> One other thing is that I am still holding a v4-beta with >>>> additional >>>>>> fixes as >>>>>>>> I didn't want to put these in the middle of some other >> debugging. >>>>>> However, >>>>>>>> these fixes address potential memory problems, so the most >>>>>> appropriate course >>>>>>>> of action would be to give that version at least a try. It >> needs >>>> to >>>>>> be >>>>>>>> released in the next days in any case. >>>>>>>> >>>>>>>> I have uploaded that (pre-4.5.6) version so that you can give >> it >>>> a >>>>>> try if you >>>>>>>> like: >>>>>>>> >>>>>>>> http://www.rsyslog.com/download/rsyslog/pre/rsyslog- >> 4.5.6.tar.gz >>>>>>>> >>>>>>>> I think it would good if you could try it at least once, >> because >>>> I >>>>>> think any >>>>>>>> other troubleshooting will boil down to looking at the fixes >> this >>>>>> version >>>>>>>> contains. >>>>>>>> >>>>>>>> Rainer >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>>>>>>> Sent: Monday, November 02, 2009 11:52 PM >>>>>>>>> To: rsyslog-users >>>>>>>>> Subject: [rsyslog] Queuing bug (4.5.5) >>>>>>>>> >>>>>>>>> Greetings all, >>>>>>>>> >>>>>>>>> I appear to have run into an issue with 4.5.5 where queues >> are >>>> not >>>>>>>>> being flushed in a timely manner. ?In this specific case, I >> have >>>>>> data >>>>>>>>> from last wednesday that is being written to disk in small >>>> chunks >>>>>>>>> today since last wednesday. ?Unfortunately I cannot be too >>>> specific >>>>>> in >>>>>>>>> details, but here is what I can see: >>>>>>>>> >>>>>>>>> Using omfile+gzip, there appears to have been a decent burst >> in >>>>>>>>> traffic sometime last wednesday. ?The rsyslog instance has >> grown >>>> to >>>>>>>>> 1.8GB of memory and stayed there for a while. ?I have both >> main >>>>>>>>> message and action queues defined using fixedarray, and I see >> no >>>>>>>>> on-disk queues (appears to be entirely in memory). >>>>>>>>> >>>>>>>>> I've got templates writing out to filenames using an hourly >>>>>> timestamp >>>>>>>>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some >> of >>>>>> those >>>>>>>>> files, all of them less than 5k in size, there are between 5 >> and >>>> 15 >>>>>>>>> lines of content, all of them from last wednesday, and within >> a >>>> few >>>>>>>>> seconds of each other. ?It's almost like there was a >> significant >>>>>> queue >>>>>>>>> built up, the hour rolled over, and only the first block of >>>> lines >>>>>> were >>>>>>>>> pulled from the queue. ?Then the hour rolled over again, and >>>>>> another >>>>>>>>> block of lines were pulled from the queue. ?Then the next >> hour, >>>>>> then >>>>>>>>> another 5-15 lines. >>>>>>>>> >>>>>>>>> Is it possible that one of the queues still has a good chunk >> of >>>>>> data >>>>>>>>> built up, and is flushing it out very slowly? ?It hasn't been >>>>>>>>> consistantly at the top of the hour either, and not every >> hour. >>>>>> ?But >>>>>>>>> the log lines themselves are sequentially written out, and >>>> usually >>>>>> the >>>>>>>>> lines are within a few seconds of each other. >>>>>>>>> >>>>>>>>> For example: >>>>>>>>> >>>>>>>>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >>>>>> 18:46:34 >>>>>>>>> and one 18:46:35 >>>>>>>>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 >> 18:46:35, >>>> 14 >>>>>>>>> lines Oct 21 18:46:36 >>>>>>>>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 >> 18:46:36, >>>> 4 >>>>>>>>> lines Oct 21 18:46:37 >>>>>>>>> >>>>>>>>> Thoughts? >>>>>>>>> -Aaron >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Tue Nov 3 19:52:12 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 3 Nov 2009 10:52:12 -0800 (PST) Subject: [rsyslog] queue configuration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103318@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103318@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 3 Nov 2009, Rainer Gerhards wrote: > David and all, > > I have created a new output plugin, omruleset, which permits to nest > rulesets. What it does is copy a message over from one ruleset to another and > make it process in parallel. Some more details in the doc: > > http://www.rsyslog.com/doc-omruleset.html this looks very interesting, however there is one thing mentioned here that concerns me. it sounds like you are saying that if you have multiple rules that write to the same file that you may get the content of various lines mixed togeather. am I understanding this correctly? if so, is there any way to work around this? David Lang > It permits to do what you asked for below. It also permits to do many more > things and create very advanced configurations. But one must be VERY careful > to receive the intended results. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Friday, October 23, 2009 10:23 PM >> To: rsyslog-users >> Subject: [rsyslog] queue configuration >> >> I know that you can create an action queue for a specific output. >> >> is there any way to create an action queue for multiple outputs? >> >> for example, in my configuration I have >> >> :fromhost, !isequal, "127.0.0.1" >> /var/log/messages;TraditionalFormat >> :fromhost, isequal, "127.0.0.1" >> @192.168.1.1;TraditionalForwardFormat >> *.* @192.168.1.115 >> *.* @192.168.1.241 >> *.* @192.168.1.242 >> *.* @192.168.1.6 >> *.* @192.168.1.7 >> *.* @192.168.1.122 >> :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 >> & @192.168.1.1;fixsnareForwardFormat2 >> & ~ >> :syslogtag, startswith, "MSWinEventLog#011" >> /var/log/messages;fixsnareFormat >> & @192.168.1.1;fixsnareForwardFormat >> & ~ >> *.* /var/log/messages;TraditionalFormat >> *.* @192.168.1.1 >> >> it would be nice to move the outbound relays off to a different queue, >> and >> to put the list of rules that have order dependancies (because they >> output >> in different formats to fix up formatting problem) to a seperate queue >> to >> spread the work across multiple processes and not slow down the basic >> writing to file. >> >> but as far as I can tell I would have to create a queue for each >> individual item, and I can't create a queue for the part of the config >> where I need to discard. >> >> am I missing something (including a better way to do everything ;-) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 3 20:20:38 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 20:20:38 +0100 Subject: [rsyslog] queue configuration Message-ID: <000201ca5cba$d08238ff$100013ac@intern.adiscon.com> Yes, thats right if you use the new buffered mode. The cure is either not to do that or use the new omruleset. ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 03.11.09 19:52 Betreff: Re: [rsyslog] queue configuration On Tue, 3 Nov 2009, Rainer Gerhards wrote: > David and all, > > I have created a new output plugin, omruleset, which permits to nest > rulesets. What it does is copy a message over from one ruleset to another and > make it process in parallel. Some more details in the doc: > > http://www.rsyslog.com/doc-omruleset.html this looks very interesting, however there is one thing mentioned here that concerns me. it sounds like you are saying that if you have multiple rules that write to the same file that you may get the content of various lines mixed togeather. am I understanding this correctly? if so, is there any way to work around this? David Lang > It permits to do what you asked for below. It also permits to do many more > things and create very advanced configurations. But one must be VERY careful > to receive the intended results. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Friday, October 23, 2009 10:23 PM >> To: rsyslog-users >> Subject: [rsyslog] queue configuration >> >> I know that you can create an action queue for a specific output. >> >> is there any way to create an action queue for multiple outputs? >> >> for example, in my configuration I have >> >> :fromhost, !isequal, "127.0.0.1" >> /var/log/messages;TraditionalFormat >> :fromhost, isequal, "127.0.0.1" >> @192.168.1.1;TraditionalForwardFormat >> *.* @192.168.1.115 >> *.* @192.168.1.241 >> *.* @192.168.1.242 >> *.* @192.168.1.6 >> *.* @192.168.1.7 >> *.* @192.168.1.122 >> :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 >> & @192.168.1.1;fixsnareForwardFormat2 >> & ~ >> :syslogtag, startswith, "MSWinEventLog#011" >> /var/log/messages;fixsnareFormat >> & @192.168.1.1;fixsnareForwardFormat >> & ~ >> *.* /var/log/messages;TraditionalFormat >> *.* @192.168.1.1 >> >> it would be nice to move the outbound relays off to a different queue, >> and >> to put the list of rules that have order dependancies (because they >> output >> in different formats to fix up formatting problem) to a seperate queue >> to >> spread the work across multiple processes and not slow down the basic >> writing to file. >> >> but as far as I can tell I would have to create a queue for each >> individual item, and I can't create a queue for the part of the config >> where I need to discard. >> >> am I missing something (including a better way to do everything ;-) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Tue Nov 3 20:29:30 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 3 Nov 2009 11:29:30 -0800 (PST) Subject: [rsyslog] queue configuration In-Reply-To: <000201ca5cba$d08238ff$100013ac@intern.adiscon.com> References: <000201ca5cba$d08238ff$100013ac@intern.adiscon.com> Message-ID: On Tue, 3 Nov 2009, Rainer Gerhards wrote: > Yes, thats right if you use the new buffered mode. The cure is either not to do that or use the new omruleset. so the new buffered mode even without seperate action queues or omruleset can have this problem? David Lang > ----- Urspr?ngliche Nachricht ----- > Von: "david at lang.hm" > An: "rsyslog-users" > Gesendet: 03.11.09 19:52 > Betreff: Re: [rsyslog] queue configuration > > On Tue, 3 Nov 2009, Rainer Gerhards wrote: > >> David and all, >> >> I have created a new output plugin, omruleset, which permits to nest >> rulesets. What it does is copy a message over from one ruleset to another and >> make it process in parallel. Some more details in the doc: >> >> http://www.rsyslog.com/doc-omruleset.html > > this looks very interesting, however there is one thing mentioned here > that concerns me. > > it sounds like you are saying that if you have multiple rules that write > to the same file that you may get the content of various lines mixed > togeather. > > am I understanding this correctly? > > if so, is there any way to work around this? > > David Lang > >> It permits to do what you asked for below. It also permits to do many more >> things and create very advanced configurations. But one must be VERY careful >> to receive the intended results. >> >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Friday, October 23, 2009 10:23 PM >>> To: rsyslog-users >>> Subject: [rsyslog] queue configuration >>> >>> I know that you can create an action queue for a specific output. >>> >>> is there any way to create an action queue for multiple outputs? >>> >>> for example, in my configuration I have >>> >>> :fromhost, !isequal, "127.0.0.1" >>> /var/log/messages;TraditionalFormat >>> :fromhost, isequal, "127.0.0.1" >>> @192.168.1.1;TraditionalForwardFormat >>> *.* @192.168.1.115 >>> *.* @192.168.1.241 >>> *.* @192.168.1.242 >>> *.* @192.168.1.6 >>> *.* @192.168.1.7 >>> *.* @192.168.1.122 >>> :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 >>> & @192.168.1.1;fixsnareForwardFormat2 >>> & ~ >>> :syslogtag, startswith, "MSWinEventLog#011" >>> /var/log/messages;fixsnareFormat >>> & @192.168.1.1;fixsnareForwardFormat >>> & ~ >>> *.* /var/log/messages;TraditionalFormat >>> *.* @192.168.1.1 >>> >>> it would be nice to move the outbound relays off to a different queue, >>> and >>> to put the list of rules that have order dependancies (because they >>> output >>> in different formats to fix up formatting problem) to a seperate queue >>> to >>> spread the work across multiple processes and not slow down the basic >>> writing to file. >>> >>> but as far as I can tell I would have to create a queue for each >>> individual item, and I can't create a queue for the part of the config >>> where I need to discard. >>> >>> am I missing something (including a better way to do everything ;-) >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 3 21:25:04 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 21:25:04 +0100 Subject: [rsyslog] queue configuration Message-ID: <000301ca5cc3$d0e60e48$100013ac@intern.adiscon.com> Think about this: actions are independent of each other. If different actions write to the same file, they do that independently. If these actions now buffer before they write, they do that independently. Consequently, the writes get mixed up. Without buffering, each os write is done seperately, so the OS keeps each of the records appended to the file. ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 03.11.09 20:30 Betreff: Re: [rsyslog] queue configuration On Tue, 3 Nov 2009, Rainer Gerhards wrote: > Yes, thats right if you use the new buffered mode. The cure is either not to do that or use the new omruleset. so the new buffered mode even without seperate action queues or omruleset can have this problem? David Lang > ----- Urspr?ngliche Nachricht ----- > Von: "david at lang.hm" > An: "rsyslog-users" > Gesendet: 03.11.09 19:52 > Betreff: Re: [rsyslog] queue configuration > > On Tue, 3 Nov 2009, Rainer Gerhards wrote: > >> David and all, >> >> I have created a new output plugin, omruleset, which permits to nest >> rulesets. What it does is copy a message over from one ruleset to another and >> make it process in parallel. Some more details in the doc: >> >> http://www.rsyslog.com/doc-omruleset.html > > this looks very interesting, however there is one thing mentioned here > that concerns me. > > it sounds like you are saying that if you have multiple rules that write > to the same file that you may get the content of various lines mixed > togeather. > > am I understanding this correctly? > > if so, is there any way to work around this? > > David Lang > >> It permits to do what you asked for below. It also permits to do many more >> things and create very advanced configurations. But one must be VERY careful >> to receive the intended results. >> >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Friday, October 23, 2009 10:23 PM >>> To: rsyslog-users >>> Subject: [rsyslog] queue configuration >>> >>> I know that you can create an action queue for a specific output. >>> >>> is there any way to create an action queue for multiple outputs? >>> >>> for example, in my configuration I have >>> >>> :fromhost, !isequal, "127.0.0.1" >>> /var/log/messages;TraditionalFormat >>> :fromhost, isequal, "127.0.0.1" >>> @192.168.1.1;TraditionalForwardFormat >>> *.* @192.168.1.115 >>> *.* @192.168.1.241 >>> *.* @192.168.1.242 >>> *.* @192.168.1.6 >>> *.* @192.168.1.7 >>> *.* @192.168.1.122 >>> :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 >>> & @192.168.1.1;fixsnareForwardFormat2 >>> & ~ >>> :syslogtag, startswith, "MSWinEventLog#011" >>> /var/log/messages;fixsnareFormat >>> & @192.168.1.1;fixsnareForwardFormat >>> & ~ >>> *.* /var/log/messages;TraditionalFormat >>> *.* @192.168.1.1 >>> >>> it would be nice to move the outbound relays off to a different queue, >>> and >>> to put the list of rules that have order dependancies (because they >>> output >>> in different formats to fix up formatting problem) to a seperate queue >>> to >>> spread the work across multiple processes and not slow down the basic >>> writing to file. >>> >>> but as far as I can tell I would have to create a queue for each >>> individual item, and I can't create a queue for the part of the config >>> where I need to discard. >>> >>> am I missing something (including a better way to do everything ;-) >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From epiphani at gmail.com Wed Nov 4 00:53:05 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 3 Nov 2009 18:53:05 -0500 Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> Message-ID: There is an action queue defined per rule set.. of which there are four. Here's the tricky part: input has been stopped. For days. The queues just seem to continue to flush very very slowly. I have a feeling that if I increased the traffic flow, the queues would flush faster. I'm not 100% sure about that though, and I can't test in this environment. Unfortunately, due to outside requirements, I'm going to have to pull rsyslog out of this deployment until we can sort out these problems... I'll work on replicating the issue in another environment, if I can. -Aaron On Tue, Nov 3, 2009 at 1:20 PM, wrote: > what is the configuration? does it have seperate action queues defined? > > if you have some way to stop the input, it would let the main queue drop > down from being full (and eliminate the input modules from needing to lock > it to try and deliver messages) > > I did see some times during my testing of 4.x stuff where the throughput > would drop drasticly when the queue was full and there ws more incoming > traffic hammering on it. it would drop from 500K logs/min to ~3k logs/min > with a trivial config, slowing down the input would let it get out of this > mode and speed up again after a little while. so much other stuff was > happening at the time that I don't think that I reported it. > > David Lang > > On Tue, 3 Nov 2009, Rainer Gerhards wrote: > >> Date: Tue, 3 Nov 2009 18:38:40 +0100 >> From: Rainer Gerhards >> Reply-To: rsyslog-users >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> It really looks like a full queue. The 2-second wait is the default wait >> interval before discarding a message when a queue is full. So for some >> reason >> the action queue seems to think it is full, so that the main queue worker >> can >> no longer add any data to it. Out of the strace, I unfortunately do not >> see >> why this happens. >> >> If possible, it would be good to try the 4.5.6 release, as this may >> actually >> be caused by the memory bug that is solved in that release. If it doesn't >> help, we would need to think about a way to get more in-depth info when >> the >> problem happens. I have an idea, but need to check if it can be done. >> >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>> Sent: Tuesday, November 03, 2009 6:34 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>> >>> Here's one with times: >>> >>> 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >>> 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL >>> >>> 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> >>> 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, ? >>> 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, >>> 3889000}) = 0 <0.000047> >>> 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 >>> 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> >>> 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, >>> 4147000}) = 0 <0.000013> >>> 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} >>> >>> 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection >>> timed out) <2.002462> >>> 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 >>> WT-GARULF6"..., 250) = 250 <0.000305> >>> 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> >>> 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, >>> 7292000}) = 0 <0.000027> >>> 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL >>> >>> 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> >>> 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, ? >>> 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, >>> 414932000}) = 0 <0.000048> >>> 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 >>> 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> >>> 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, >>> 415191000}) = 0 <0.000013> >>> 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} >>> >>> 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection >>> timed out) <2.002220> >>> 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 >>> WT-GARULF6"..., 197) = 197 <0.000279> >>> 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> >>> 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, >>> 417992000}) = 0 <0.000027> >>> 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL >>> >>> 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> >>> 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, ? >>> 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, >>> 608226000}) = 0 <0.000047> >>> 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 >>> 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> >>> 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, >>> 608486000}) = 0 <0.000014> >>> 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} >>> >>> >>> It looks like the locks are waiting a -very- long time. >>> >>> -Aaron >>> >>> On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards >>> wrote: >>>> >>>> first shot at the data: this looks like a full queue where the >>> >>> enqueue >>>> >>>> operation is waiting on a drain that does not happen (fast enough). >>> >>> Of >>>> >>>> course, that doesn't explain why this happens... >>>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>>> Sent: Tuesday, November 03, 2009 6:27 PM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>>> >>>>> Ok - I captured an strace. ?This appears to be the action thread. >>>>> This specific set of calls took minutes. ?I'll re-run this with -t >>>>> and/or -T. >>>>> >>>>> Note that this syslog instance is not actually receiving any data >>> >>> right >>>>> >>>>> now. >>>>> >>>>> -Aaron >>>>> >>>>> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} >> >>> ...> >>>>> >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>> timed out) >>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} >> >>> ...> >>>>> >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>> timed out) >>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} >> >>> ...> >>>>> >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>> timed out) >>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource >>>>> temporarily unavailable) >>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} >> >>> ...> >>>>> >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>> timed out) >>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL >>>>> >>>>> >>>>> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards >>>>> wrote: >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>> Sent: Tuesday, November 03, 2009 4:06 PM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>>>>> >>>>>>> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >>>>>>> >>>>>>>> This is still taking place this hour - though I obviously can't >>>>>>>> restart onto a newer version without losing this case. ?Is >>> >>> there >>>>>>>> >>>>>>>> anything I can do in the current configuration to try and debug >>>>> >>>>> this >>>>>>>> >>>>>>>> situation? >>>>>>> >>>>>>> if you can strace each thread for a few seconds it may give you >>> >>> an >>>>> >>>>> idea >>>>>>> >>>>>>> what's happening. >>>>>> >>>>>> This is a good suggestion. >>>>>> >>>>>> It would also potentially be enlightening to attach gdb to the >>>>> >>>>> process at >>>>>> >>>>>> various points in time and get a backtrace from all running >>> >>> threads. >>>>>> >>>>>> Other than that, there is little you can do on a running version. >>> >>> If >>>>> >>>>> it were >>>>>> >>>>>> compiled for debug, and the environment setup so that we could >>>>> >>>>> capture >>>>>> >>>>>> stdout/stderr, sending SIGUSR2 would yield to much the same >>>>> >>>>> information as >>>>>> >>>>>> the gdb backtrace BUT when runtime instrumentation is enabled, the >>>>> >>>>> build is >>>>>> >>>>>> operating at a third to a quarter of its normal speed. >>>>>> >>>>>>> in the meantime you need to stop the system from getting further >>>>>>> behind, >>>>>>> can you redirect or reconfigure the senders so that they are not >>>>>>> sending >>>>>>> new logs to this box so that it can dig itself out (stopping the >>>>> >>>>> input >>>>>>> >>>>>>> may >>>>>>> be enough by itself, rsyslog has historicly done a LOT of locking >>>>>>> around >>>>>>> the main queue, and if you have a full queue with more data >>> >>> trying >>>>> >>>>> to >>>>>>> >>>>>>> be >>>>>>> delivered it can really slow things down. I wouldn't expect it to >>> >>> be >>>>>>> >>>>>>> this >>>>>>> much, but it could be part of it) >>>>>>> >>>>>> >>>>>> This may clean up things, but I really doubt it, given the >>> >>> magnitude >>>>> >>>>> of >>>>>> >>>>>> delays. Also, Aaron runs 4.4.5, which already has lots of the >>> >>> locking >>>>>> >>>>>> removed/restructure (not to compare with the recent v5-devel, but >>>>> >>>>> much more >>>>>> >>>>>> effcient than early v4 and v3). >>>>>> >>>>>> Rainer >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>>> (We're up to 18:46:51 now!) >>>>>>>> >>>>>>>> -Aaron >>>>>>>> >>>>>>>> On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> mhhh... This is obviously not intended behavior. There are >>> >>> some >>>>>>> >>>>>>> rate-limiting >>>>>>>>> >>>>>>>>> features that can deliberately slow down a queue, but I guess >>> >>> you >>>>>>> >>>>>>> have not >>>>>>>>> >>>>>>>>> configured these. So there either is a bug that activates them >>> >>> at >>>>>>> >>>>>>> some point >>>>>>>>> >>>>>>>>> during processing (e.g. an invalid memory access could do >>> >>> that) >>>>> >>>>> or >>>>>>> >>>>>>> there is >>>>>>>>> >>>>>>>>> some other bug that causes the dequeue to happen very slowly. >>> >>> In >>>>> >>>>> any >>>>>>> >>>>>>> case, it >>>>>>>>> >>>>>>>>> is hard to guess. >>>>>>>>> >>>>>>>>> Given the volume of the messages, a debug log probably does >>> >>> not >>>>>>> >>>>>>> help. We >>>>>>>>> >>>>>>>>> could gain a little insight in at least the queue sizes that >>>>> >>>>> rsyslog >>>>>>> >>>>>>> sees via >>>>>>>>> >>>>>>>>> imdiag plus the (very rudamentary) v5 debug front-end (it >>> >>> doesn't >>>>>>> >>>>>>> require the >>>>>>>>> >>>>>>>>> engine to be v5!). I would need to spend at least a little >>> >>> work >>>>> >>>>> on >>>>>>> >>>>>>> the >>>>>>>>> >>>>>>>>> front-end to enable remote access, but that's not really a >>>>> >>>>> problem. >>>>>>>>> >>>>>>>>> One other thing is that I am still holding a v4-beta with >>>>> >>>>> additional >>>>>>> >>>>>>> fixes as >>>>>>>>> >>>>>>>>> I didn't want to put these in the middle of some other >>> >>> debugging. >>>>>>> >>>>>>> However, >>>>>>>>> >>>>>>>>> these fixes address potential memory problems, so the most >>>>>>> >>>>>>> appropriate course >>>>>>>>> >>>>>>>>> of action would be to give that version at least a try. It >>> >>> needs >>>>> >>>>> to >>>>>>> >>>>>>> be >>>>>>>>> >>>>>>>>> released in the next days in any case. >>>>>>>>> >>>>>>>>> I have uploaded that (pre-4.5.6) version so that you can give >>> >>> it >>>>> >>>>> a >>>>>>> >>>>>>> try if you >>>>>>>>> >>>>>>>>> like: >>>>>>>>> >>>>>>>>> http://www.rsyslog.com/download/rsyslog/pre/rsyslog- >>> >>> 4.5.6.tar.gz >>>>>>>>> >>>>>>>>> I think it would good if you could try it at least once, >>> >>> because >>>>> >>>>> I >>>>>>> >>>>>>> think any >>>>>>>>> >>>>>>>>> other troubleshooting will boil down to looking at the fixes >>> >>> this >>>>>>> >>>>>>> version >>>>>>>>> >>>>>>>>> contains. >>>>>>>>> >>>>>>>>> Rainer >>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>>>>>>>> Sent: Monday, November 02, 2009 11:52 PM >>>>>>>>>> To: rsyslog-users >>>>>>>>>> Subject: [rsyslog] Queuing bug (4.5.5) >>>>>>>>>> >>>>>>>>>> Greetings all, >>>>>>>>>> >>>>>>>>>> I appear to have run into an issue with 4.5.5 where queues >>> >>> are >>>>> >>>>> not >>>>>>>>>> >>>>>>>>>> being flushed in a timely manner. ?In this specific case, I >>> >>> have >>>>>>> >>>>>>> data >>>>>>>>>> >>>>>>>>>> from last wednesday that is being written to disk in small >>>>> >>>>> chunks >>>>>>>>>> >>>>>>>>>> today since last wednesday. ?Unfortunately I cannot be too >>>>> >>>>> specific >>>>>>> >>>>>>> in >>>>>>>>>> >>>>>>>>>> details, but here is what I can see: >>>>>>>>>> >>>>>>>>>> Using omfile+gzip, there appears to have been a decent burst >>> >>> in >>>>>>>>>> >>>>>>>>>> traffic sometime last wednesday. ?The rsyslog instance has >>> >>> grown >>>>> >>>>> to >>>>>>>>>> >>>>>>>>>> 1.8GB of memory and stayed there for a while. ?I have both >>> >>> main >>>>>>>>>> >>>>>>>>>> message and action queues defined using fixedarray, and I see >>> >>> no >>>>>>>>>> >>>>>>>>>> on-disk queues (appears to be entirely in memory). >>>>>>>>>> >>>>>>>>>> I've got templates writing out to filenames using an hourly >>>>>>> >>>>>>> timestamp >>>>>>>>>> >>>>>>>>>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some >>> >>> of >>>>>>> >>>>>>> those >>>>>>>>>> >>>>>>>>>> files, all of them less than 5k in size, there are between 5 >>> >>> and >>>>> >>>>> 15 >>>>>>>>>> >>>>>>>>>> lines of content, all of them from last wednesday, and within >>> >>> a >>>>> >>>>> few >>>>>>>>>> >>>>>>>>>> seconds of each other. ?It's almost like there was a >>> >>> significant >>>>>>> >>>>>>> queue >>>>>>>>>> >>>>>>>>>> built up, the hour rolled over, and only the first block of >>>>> >>>>> lines >>>>>>> >>>>>>> were >>>>>>>>>> >>>>>>>>>> pulled from the queue. ?Then the hour rolled over again, and >>>>>>> >>>>>>> another >>>>>>>>>> >>>>>>>>>> block of lines were pulled from the queue. ?Then the next >>> >>> hour, >>>>>>> >>>>>>> then >>>>>>>>>> >>>>>>>>>> another 5-15 lines. >>>>>>>>>> >>>>>>>>>> Is it possible that one of the queues still has a good chunk >>> >>> of >>>>>>> >>>>>>> data >>>>>>>>>> >>>>>>>>>> built up, and is flushing it out very slowly? ?It hasn't been >>>>>>>>>> consistantly at the top of the hour either, and not every >>> >>> hour. >>>>>>> >>>>>>> ?But >>>>>>>>>> >>>>>>>>>> the log lines themselves are sequentially written out, and >>>>> >>>>> usually >>>>>>> >>>>>>> the >>>>>>>>>> >>>>>>>>>> lines are within a few seconds of each other. >>>>>>>>>> >>>>>>>>>> For example: >>>>>>>>>> >>>>>>>>>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >>>>>>> >>>>>>> 18:46:34 >>>>>>>>>> >>>>>>>>>> and one 18:46:35 >>>>>>>>>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 >>> >>> 18:46:35, >>>>> >>>>> 14 >>>>>>>>>> >>>>>>>>>> lines Oct 21 18:46:36 >>>>>>>>>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 >>> >>> 18:46:36, >>>>> >>>>> 4 >>>>>>>>>> >>>>>>>>>> lines Oct 21 18:46:37 >>>>>>>>>> >>>>>>>>>> Thoughts? >>>>>>>>>> -Aaron >>>>>>>>>> _______________________________________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > From david at lang.hm Wed Nov 4 03:18:56 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 3 Nov 2009 18:18:56 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 3 Nov 2009, Aaron Wiebe wrote: > There is an action queue defined per rule set.. of which there are four. is it possible that something blocked one of the rule outputs? as I understand it, that would cause that action queue to fill up, and then the behavior of the main queue trying, then sleeping would make sense to me > Here's the tricky part: input has been stopped. For days. The > queues just seem to continue to flush very very slowly. I have a > feeling that if I increased the traffic flow, the queues would flush > faster. I'm not 100% sure about that though, and I can't test in this > environment. Unfortunately, due to outside requirements, I'm going to > have to pull rsyslog out of this deployment until we can sort out > these problems... did you configure this with HUPisRestart off? if so you may try sending it a HUP and see if that gets a few log messages out. if it does, hammer it with HUPs to drain the queue it is really a pain to run into problems only in production. David Lang > I'll work on replicating the issue in another environment, if I can. > > -Aaron > > On Tue, Nov 3, 2009 at 1:20 PM, wrote: >> what is the configuration? does it have seperate action queues defined? >> >> if you have some way to stop the input, it would let the main queue drop >> down from being full (and eliminate the input modules from needing to lock >> it to try and deliver messages) >> >> I did see some times during my testing of 4.x stuff where the throughput >> would drop drasticly when the queue was full and there ws more incoming >> traffic hammering on it. it would drop from 500K logs/min to ~3k logs/min >> with a trivial config, slowing down the input would let it get out of this >> mode and speed up again after a little while. so much other stuff was >> happening at the time that I don't think that I reported it. >> >> David Lang >> >> On Tue, 3 Nov 2009, Rainer Gerhards wrote: >> >>> Date: Tue, 3 Nov 2009 18:38:40 +0100 >>> From: Rainer Gerhards >>> Reply-To: rsyslog-users >>> To: rsyslog-users >>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>> >>> It really looks like a full queue. The 2-second wait is the default wait >>> interval before discarding a message when a queue is full. So for some >>> reason >>> the action queue seems to think it is full, so that the main queue worker >>> can >>> no longer add any data to it. Out of the strace, I unfortunately do not >>> see >>> why this happens. >>> >>> If possible, it would be good to try the 4.5.6 release, as this may >>> actually >>> be caused by the memory bug that is solved in that release. If it doesn't >>> help, we would need to think about a way to get more in-depth info when >>> the >>> problem happens. I have an idea, but need to check if it can be done. >>> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>> Sent: Tuesday, November 03, 2009 6:34 PM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>> >>>> Here's one with times: >>>> >>>> 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >>>> 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL >>>> >>>> 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> >>>> 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, >>>> 3889000}) = 0 <0.000047> >>>> 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>> 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> >>>> 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, >>>> 4147000}) = 0 <0.000013> >>>> 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} >>>> >>>> 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection >>>> timed out) <2.002462> >>>> 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 >>>> WT-GARULF6"..., 250) = 250 <0.000305> >>>> 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> >>>> 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, >>>> 7292000}) = 0 <0.000027> >>>> 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL >>>> >>>> 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> >>>> 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, >>>> 414932000}) = 0 <0.000048> >>>> 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>> 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> >>>> 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, >>>> 415191000}) = 0 <0.000013> >>>> 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} >>>> >>>> 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection >>>> timed out) <2.002220> >>>> 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 >>>> WT-GARULF6"..., 197) = 197 <0.000279> >>>> 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> >>>> 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, >>>> 417992000}) = 0 <0.000027> >>>> 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL >>>> >>>> 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> >>>> 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, >>>> 608226000}) = 0 <0.000047> >>>> 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>> 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> >>>> 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, >>>> 608486000}) = 0 <0.000014> >>>> 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} >>>> >>>> >>>> It looks like the locks are waiting a -very- long time. >>>> >>>> -Aaron >>>> >>>> On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards >>>> wrote: >>>>> >>>>> first shot at the data: this looks like a full queue where the >>>> >>>> enqueue >>>>> >>>>> operation is waiting on a drain that does not happen (fast enough). >>>> >>>> Of >>>>> >>>>> course, that doesn't explain why this happens... >>>>> >>>>> Rainer >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>>>> Sent: Tuesday, November 03, 2009 6:27 PM >>>>>> To: rsyslog-users >>>>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>>>> >>>>>> Ok - I captured an strace. ?This appears to be the action thread. >>>>>> This specific set of calls took minutes. ?I'll re-run this with -t >>>>>> and/or -T. >>>>>> >>>>>> Note that this syslog instance is not actually receiving any data >>>> >>>> right >>>>>> >>>>>> now. >>>>>> >>>>>> -Aaron >>>>>> >>>>>> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} >>> >>>> ...> >>>>>> >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>>> timed out) >>>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} >>> >>>> ...> >>>>>> >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>>> timed out) >>>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>>> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} >>> >>>> ...> >>>>>> >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>>> timed out) >>>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource >>>>>> temporarily unavailable) >>>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>>> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>>> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} >>> >>>> ...> >>>>>> >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>>> timed out) >>>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL >>>>>> >>>>>> >>>>>> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards >>>>>> wrote: >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>>> Sent: Tuesday, November 03, 2009 4:06 PM >>>>>>>> To: rsyslog-users >>>>>>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>>>>>> >>>>>>>> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >>>>>>>> >>>>>>>>> This is still taking place this hour - though I obviously can't >>>>>>>>> restart onto a newer version without losing this case. ?Is >>>> >>>> there >>>>>>>>> >>>>>>>>> anything I can do in the current configuration to try and debug >>>>>> >>>>>> this >>>>>>>>> >>>>>>>>> situation? >>>>>>>> >>>>>>>> if you can strace each thread for a few seconds it may give you >>>> >>>> an >>>>>> >>>>>> idea >>>>>>>> >>>>>>>> what's happening. >>>>>>> >>>>>>> This is a good suggestion. >>>>>>> >>>>>>> It would also potentially be enlightening to attach gdb to the >>>>>> >>>>>> process at >>>>>>> >>>>>>> various points in time and get a backtrace from all running >>>> >>>> threads. >>>>>>> >>>>>>> Other than that, there is little you can do on a running version. >>>> >>>> If >>>>>> >>>>>> it were >>>>>>> >>>>>>> compiled for debug, and the environment setup so that we could >>>>>> >>>>>> capture >>>>>>> >>>>>>> stdout/stderr, sending SIGUSR2 would yield to much the same >>>>>> >>>>>> information as >>>>>>> >>>>>>> the gdb backtrace BUT when runtime instrumentation is enabled, the >>>>>> >>>>>> build is >>>>>>> >>>>>>> operating at a third to a quarter of its normal speed. >>>>>>> >>>>>>>> in the meantime you need to stop the system from getting further >>>>>>>> behind, >>>>>>>> can you redirect or reconfigure the senders so that they are not >>>>>>>> sending >>>>>>>> new logs to this box so that it can dig itself out (stopping the >>>>>> >>>>>> input >>>>>>>> >>>>>>>> may >>>>>>>> be enough by itself, rsyslog has historicly done a LOT of locking >>>>>>>> around >>>>>>>> the main queue, and if you have a full queue with more data >>>> >>>> trying >>>>>> >>>>>> to >>>>>>>> >>>>>>>> be >>>>>>>> delivered it can really slow things down. I wouldn't expect it to >>>> >>>> be >>>>>>>> >>>>>>>> this >>>>>>>> much, but it could be part of it) >>>>>>>> >>>>>>> >>>>>>> This may clean up things, but I really doubt it, given the >>>> >>>> magnitude >>>>>> >>>>>> of >>>>>>> >>>>>>> delays. Also, Aaron runs 4.4.5, which already has lots of the >>>> >>>> locking >>>>>>> >>>>>>> removed/restructure (not to compare with the recent v5-devel, but >>>>>> >>>>>> much more >>>>>>> >>>>>>> effcient than early v4 and v3). >>>>>>> >>>>>>> Rainer >>>>>>>> >>>>>>>> David Lang >>>>>>>> >>>>>>>>> (We're up to 18:46:51 now!) >>>>>>>>> >>>>>>>>> -Aaron >>>>>>>>> >>>>>>>>> On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> mhhh... This is obviously not intended behavior. There are >>>> >>>> some >>>>>>>> >>>>>>>> rate-limiting >>>>>>>>>> >>>>>>>>>> features that can deliberately slow down a queue, but I guess >>>> >>>> you >>>>>>>> >>>>>>>> have not >>>>>>>>>> >>>>>>>>>> configured these. So there either is a bug that activates them >>>> >>>> at >>>>>>>> >>>>>>>> some point >>>>>>>>>> >>>>>>>>>> during processing (e.g. an invalid memory access could do >>>> >>>> that) >>>>>> >>>>>> or >>>>>>>> >>>>>>>> there is >>>>>>>>>> >>>>>>>>>> some other bug that causes the dequeue to happen very slowly. >>>> >>>> In >>>>>> >>>>>> any >>>>>>>> >>>>>>>> case, it >>>>>>>>>> >>>>>>>>>> is hard to guess. >>>>>>>>>> >>>>>>>>>> Given the volume of the messages, a debug log probably does >>>> >>>> not >>>>>>>> >>>>>>>> help. We >>>>>>>>>> >>>>>>>>>> could gain a little insight in at least the queue sizes that >>>>>> >>>>>> rsyslog >>>>>>>> >>>>>>>> sees via >>>>>>>>>> >>>>>>>>>> imdiag plus the (very rudamentary) v5 debug front-end (it >>>> >>>> doesn't >>>>>>>> >>>>>>>> require the >>>>>>>>>> >>>>>>>>>> engine to be v5!). I would need to spend at least a little >>>> >>>> work >>>>>> >>>>>> on >>>>>>>> >>>>>>>> the >>>>>>>>>> >>>>>>>>>> front-end to enable remote access, but that's not really a >>>>>> >>>>>> problem. >>>>>>>>>> >>>>>>>>>> One other thing is that I am still holding a v4-beta with >>>>>> >>>>>> additional >>>>>>>> >>>>>>>> fixes as >>>>>>>>>> >>>>>>>>>> I didn't want to put these in the middle of some other >>>> >>>> debugging. >>>>>>>> >>>>>>>> However, >>>>>>>>>> >>>>>>>>>> these fixes address potential memory problems, so the most >>>>>>>> >>>>>>>> appropriate course >>>>>>>>>> >>>>>>>>>> of action would be to give that version at least a try. It >>>> >>>> needs >>>>>> >>>>>> to >>>>>>>> >>>>>>>> be >>>>>>>>>> >>>>>>>>>> released in the next days in any case. >>>>>>>>>> >>>>>>>>>> I have uploaded that (pre-4.5.6) version so that you can give >>>> >>>> it >>>>>> >>>>>> a >>>>>>>> >>>>>>>> try if you >>>>>>>>>> >>>>>>>>>> like: >>>>>>>>>> >>>>>>>>>> http://www.rsyslog.com/download/rsyslog/pre/rsyslog- >>>> >>>> 4.5.6.tar.gz >>>>>>>>>> >>>>>>>>>> I think it would good if you could try it at least once, >>>> >>>> because >>>>>> >>>>>> I >>>>>>>> >>>>>>>> think any >>>>>>>>>> >>>>>>>>>> other troubleshooting will boil down to looking at the fixes >>>> >>>> this >>>>>>>> >>>>>>>> version >>>>>>>>>> >>>>>>>>>> contains. >>>>>>>>>> >>>>>>>>>> Rainer >>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>>>>>>>>> Sent: Monday, November 02, 2009 11:52 PM >>>>>>>>>>> To: rsyslog-users >>>>>>>>>>> Subject: [rsyslog] Queuing bug (4.5.5) >>>>>>>>>>> >>>>>>>>>>> Greetings all, >>>>>>>>>>> >>>>>>>>>>> I appear to have run into an issue with 4.5.5 where queues >>>> >>>> are >>>>>> >>>>>> not >>>>>>>>>>> >>>>>>>>>>> being flushed in a timely manner. ?In this specific case, I >>>> >>>> have >>>>>>>> >>>>>>>> data >>>>>>>>>>> >>>>>>>>>>> from last wednesday that is being written to disk in small >>>>>> >>>>>> chunks >>>>>>>>>>> >>>>>>>>>>> today since last wednesday. ?Unfortunately I cannot be too >>>>>> >>>>>> specific >>>>>>>> >>>>>>>> in >>>>>>>>>>> >>>>>>>>>>> details, but here is what I can see: >>>>>>>>>>> >>>>>>>>>>> Using omfile+gzip, there appears to have been a decent burst >>>> >>>> in >>>>>>>>>>> >>>>>>>>>>> traffic sometime last wednesday. ?The rsyslog instance has >>>> >>>> grown >>>>>> >>>>>> to >>>>>>>>>>> >>>>>>>>>>> 1.8GB of memory and stayed there for a while. ?I have both >>>> >>>> main >>>>>>>>>>> >>>>>>>>>>> message and action queues defined using fixedarray, and I see >>>> >>>> no >>>>>>>>>>> >>>>>>>>>>> on-disk queues (appears to be entirely in memory). >>>>>>>>>>> >>>>>>>>>>> I've got templates writing out to filenames using an hourly >>>>>>>> >>>>>>>> timestamp >>>>>>>>>>> >>>>>>>>>>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some >>>> >>>> of >>>>>>>> >>>>>>>> those >>>>>>>>>>> >>>>>>>>>>> files, all of them less than 5k in size, there are between 5 >>>> >>>> and >>>>>> >>>>>> 15 >>>>>>>>>>> >>>>>>>>>>> lines of content, all of them from last wednesday, and within >>>> >>>> a >>>>>> >>>>>> few >>>>>>>>>>> >>>>>>>>>>> seconds of each other. ?It's almost like there was a >>>> >>>> significant >>>>>>>> >>>>>>>> queue >>>>>>>>>>> >>>>>>>>>>> built up, the hour rolled over, and only the first block of >>>>>> >>>>>> lines >>>>>>>> >>>>>>>> were >>>>>>>>>>> >>>>>>>>>>> pulled from the queue. ?Then the hour rolled over again, and >>>>>>>> >>>>>>>> another >>>>>>>>>>> >>>>>>>>>>> block of lines were pulled from the queue. ?Then the next >>>> >>>> hour, >>>>>>>> >>>>>>>> then >>>>>>>>>>> >>>>>>>>>>> another 5-15 lines. >>>>>>>>>>> >>>>>>>>>>> Is it possible that one of the queues still has a good chunk >>>> >>>> of >>>>>>>> >>>>>>>> data >>>>>>>>>>> >>>>>>>>>>> built up, and is flushing it out very slowly? ?It hasn't been >>>>>>>>>>> consistantly at the top of the hour either, and not every >>>> >>>> hour. >>>>>>>> >>>>>>>> ?But >>>>>>>>>>> >>>>>>>>>>> the log lines themselves are sequentially written out, and >>>>>> >>>>>> usually >>>>>>>> >>>>>>>> the >>>>>>>>>>> >>>>>>>>>>> lines are within a few seconds of each other. >>>>>>>>>>> >>>>>>>>>>> For example: >>>>>>>>>>> >>>>>>>>>>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >>>>>>>> >>>>>>>> 18:46:34 >>>>>>>>>>> >>>>>>>>>>> and one 18:46:35 >>>>>>>>>>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 >>>> >>>> 18:46:35, >>>>>> >>>>>> 14 >>>>>>>>>>> >>>>>>>>>>> lines Oct 21 18:46:36 >>>>>>>>>>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 >>>> >>>> 18:46:36, >>>>>> >>>>>> 4 >>>>>>>>>>> >>>>>>>>>>> lines Oct 21 18:46:37 >>>>>>>>>>> >>>>>>>>>>> Thoughts? >>>>>>>>>>> -Aaron >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> rsyslog mailing list >>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>> http://www.rsyslog.com >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Nov 4 07:22:01 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 07:22:01 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5? References: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103323@GRFEXC.intern.adiscon.com> Hi, that actually could be a regression caused by the rewrite of omfile. I'll look at it as soon as I have time, but it probably is a good idea to file a bug tracker with bugzilla. I first need to finish what I am currently implementing (custom parser interface) and then I need to look into which bugs are open and have priority. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luis Fernando Mu?oz Mej?as > Sent: Tuesday, November 03, 2009 7:05 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Output to named pipes not working on v4.5.5? > > Hello, world > > I must be doing something really dumb. I need to send some output to a > named pipe that I have already created, but rsyslog is refusing to > open it. > > My configuration states: > > *.* |/var/log/external/all/unknown;RSYSLOG_FileFormat > > And, of course, the named pipe exists: > > # ls -Z /var/log/external/all/unknown > > prw-r----- root logreaders user_u:object_r:var_log_t > /var/log/external/all/unknown > > But when I start the syslog daemon it complains with this message: > > rsyslogd-2039: Could no open output file > '|/var/log/external/all/unknown' [try http://www.rsyslog.com/e/2039 ] > > It may be a SELinux problem, but the logs show nothing about any > denied requests. And the context for the named pipe is the exact same > I have for the log files, which lie in the same directory as well. > > Am I the only one experiencing such problems? > > Thanks. > -- > Luis Fernando Mu?oz Mej?as > Luis.Fernando.Munoz.Mejias at cern.ch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Nov 4 07:34:43 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 07:34:43 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> > it is really a pain to run into problems only in production. The problem with hunting such bugs is that we cannot get the diagnostic information we need. I have thought about a solution. Could you please comment if this one would be acceptable for your environment: I add the capability to send the debug log out via UDP syslog (not via the usual channels, but rather via a special send function inside the debug printf system). Then, when a problem occurs, we could enable the debug printf (which is present in production builds as well) and make it output the data to a remote server (or a special listener, even netcat, running on the local machine). I could turn on/off this via a signal (as we currently do for sending debug message to stdout). Alternatively, one could load imdiag into a suspect instance and use that to gain more in-depth information. However, I am not sure if that would be suitable for use in production, at least it is questionable from a security POV. Any feedback is appreciated. While I like to have better ability to dig into problems as they occur, I don't think it makes sense to invest time into coding something if it is unclear whether or not it could be utilized... So unless I get sufficient positive feedback, I'll stay away from implementing that. Rainer From correo at miguelangelnieto.net Wed Nov 4 10:16:42 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Wed, 4 Nov 2009 10:16:42 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory Message-ID: Hi, I have a problem with the attached client-configuration. When I stop the server, the client computer hangs-up some minutes later and didn't write logs on $WorkDirectory /var/log/queue. Where is the mistake? Thank you and sorry for my bad english :P -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From epiphani at gmail.com Wed Nov 4 14:09:29 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Wed, 4 Nov 2009 08:09:29 -0500 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> Message-ID: On Wed, Nov 4, 2009 at 1:34 AM, Rainer Gerhards wrote: > > I add the capability to send the debug log out via UDP syslog (not via the > usual channels, but rather via a special send function inside the debug > printf system). Then, when a problem occurs, we could enable the debug printf > (which is present in production builds as well) and make it output the data > to a remote server (or a special listener, even netcat, running on the local > machine). > > I could turn on/off this via a signal (as we currently do for sending debug > message to stdout). That could work - I could see that being an acceptable step for some of our environments here. > Alternatively, one could load imdiag into a suspect instance and use that to > gain more in-depth information. However, I am not sure if that would be > suitable for use in production, at least it is questionable from a security > POV. Is there more docs on imdiag and the enhancements you've made to it recently kicking around? I'll see if I can't make use if it in my efforts to reproduce the issue (which probably won't start until next week). -Aaron From rgerhards at hq.adiscon.com Wed Nov 4 14:13:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 14:13:56 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103333@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Wednesday, November 04, 2009 2:09 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) / Problem in production > > On Wed, Nov 4, 2009 at 1:34 AM, Rainer Gerhards > wrote: > > > > I add the capability to send the debug log out via UDP syslog (not > via the > > usual channels, but rather via a special send function inside the > debug > > printf system). Then, when a problem occurs, we could enable the > debug printf > > (which is present in production builds as well) and make it output > the data > > to a remote server (or a special listener, even netcat, running on > the local > > machine). > > > > I could turn on/off this via a signal (as we currently do for sending > debug > > message to stdout). > > That could work - I could see that being an acceptable step for some > of our environments here. OK, that sounds good. Will see what I can do (I think this could be generally useful). > > > Alternatively, one could load imdiag into a suspect instance and use > that to > > gain more in-depth information. However, I am not sure if that would > be > > suitable for use in production, at least it is questionable from a > security > > POV. > > Is there more docs on imdiag and the enhancements you've made to it > recently kicking around? I'll see if I can't make use if it in my > efforts to reproduce the issue (which probably won't start until next > week). No, actually not. Initially it was intended to be a thing mostly for Rainer and the testbench ;) I'll see that I do a write-up on production troubleshooting and include it. Any more feedback from anyone on this issue is appreciated. Rainer > > -Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From Luis.Fernando.Munoz.Mejias at cern.ch Wed Nov 4 14:14:34 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?utf-8?q?Mu=C3=B1oz_Mej=C3=ADas?=) Date: Wed, 4 Nov 2009 14:14:34 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5? In-Reply-To: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> References: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <200911041414.34449.Luis.Fernando.Munoz.Mejias@cern.ch> So, this has two parts: 1) A SELinux problem on RHEL5 and similar (but the logs show nothing!!). It seems the syslog context is not allowed to access FIFOs. I'll have to fix that. 2) Even when there are no permission problems, rsyslog 4.5.5 just doesn't write to a named pipe. From the following consecutive lines: *.* |/var/log/external/fifo *.* /var/log/external/file the file receives data, the FIFO doesn't. I'll paste all this to bugzilla. Cheers. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From david at lang.hm Wed Nov 4 14:28:57 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 05:28:57 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 4 Nov 2009, Rainer Gerhards wrote: >> it is really a pain to run into problems only in production. > > The problem with hunting such bugs is that we cannot get the diagnostic > information we need. I have thought about a solution. Could you please > comment if this one would be acceptable for your environment: > > I add the capability to send the debug log out via UDP syslog (not via the > usual channels, but rather via a special send function inside the debug > printf system). Then, when a problem occurs, we could enable the debug printf > (which is present in production builds as well) and make it output the data > to a remote server (or a special listener, even netcat, running on the local > machine). > > I could turn on/off this via a signal (as we currently do for sending debug > message to stdout). that would be a LOT easier than having to run in debug mode the entire time. would this have similar performance impact to running in debug mode? or does this need less serialization, and therefor less impact? > Alternatively, one could load imdiag into a suspect instance and use that to > gain more in-depth information. However, I am not sure if that would be > suitable for use in production, at least it is questionable from a security > POV. is loading this on the fly even possible? david Lang > Any feedback is appreciated. While I like to have better ability to dig into > problems as they occur, I don't think it makes sense to invest time into > coding something if it is unclear whether or not it could be utilized... So > unless I get sufficient positive feedback, I'll stay away from implementing > that. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Nov 4 14:32:22 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 05:32:22 -0800 (PST) Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > I have a problem with the attached client-configuration. When I stop > the server, the client computer hangs-up some minutes later and didn't > write logs on $WorkDirectory /var/log/queue. this list strips attachments, please re-send with the config in the body of the message. david Lang -------------- next part -------------- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Nov 4 14:33:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 14:33:53 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103334@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, November 04, 2009 2:29 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) / Problem in production > > On Wed, 4 Nov 2009, Rainer Gerhards wrote: > > >> it is really a pain to run into problems only in production. > > > > The problem with hunting such bugs is that we cannot get the > diagnostic > > information we need. I have thought about a solution. Could you > please > > comment if this one would be acceptable for your environment: > > > > I add the capability to send the debug log out via UDP syslog (not > via the > > usual channels, but rather via a special send function inside the > debug > > printf system). Then, when a problem occurs, we could enable the > debug printf > > (which is present in production builds as well) and make it output > the data > > to a remote server (or a special listener, even netcat, running on > the local > > machine). > > > > I could turn on/off this via a signal (as we currently do for sending > debug > > message to stdout). > > that would be a LOT easier than having to run in debug mode the entire > time. The problem is that you would not get the information BEFORE you turned in on. In theory, it should even be possible with the current system and a file, but I need to check this out (I never used it that way, but the engine probably is capable of doing it). > would this have similar performance impact to running in debug mode? or > does this need less serialization, and therefor less impact? Think of it as turning on debug mode on the fly. That exactly is what it does. So before the signal, there is no impact, but after it is the same impact as usual. > > Alternatively, one could load imdiag into a suspect instance and use > that to > > gain more in-depth information. However, I am not sure if that would > be > > suitable for use in production, at least it is questionable from a > security > > POV. > > is loading this on the fly even possible? NO, that's the point. If we intend to pursue that path, imdiag always needs to be loaded. That does not have any performance impact (but a security impact!). Once imdiag is loaded, it could be used to do all sorts of "interesting" things (most of which need to be implemented first...). Rainer > > david Lang > > > Any feedback is appreciated. While I like to have better ability to > dig into > > problems as they occur, I don't think it makes sense to invest time > into > > coding something if it is unclear whether or not it could be > utilized... So > > unless I get sufficient positive feedback, I'll stay away from > implementing > > that. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Nov 4 14:55:20 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 05:55:20 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103334@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103334@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 4 Nov 2009, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >>> >>> I could turn on/off this via a signal (as we currently do for sending >> debug >>> message to stdout). >> >> that would be a LOT easier than having to run in debug mode the entire >> time. > > The problem is that you would not get the information BEFORE you turned in > on. In theory, it should even be possible with the current system and a file, > but I need to check this out (I never used it that way, but the engine > probably is capable of doing it). yes, this would not help see what triggered the problem, but could be a huge help in figuring out what the current state of things are. going to a file could work as well. in either case a config option would have to be created to specify the destination. >> would this have similar performance impact to running in debug mode? or >> does this need less serialization, and therefor less impact? > > Think of it as turning on debug mode on the fly. That exactly is what it > does. So before the signal, there is no impact, but after it is the same > impact as usual. even in a high volume production environment this can be tolorated for short time periods. currently there is no way (short of stop, reconfigure, start, stop, reconfigure, start) to try and gather this info, and each stop will loose logs, so this ability would be much better. >>> Alternatively, one could load imdiag into a suspect instance and use >> that to >>> gain more in-depth information. However, I am not sure if that would >> be >>> suitable for use in production, at least it is questionable from a >> security >>> POV. >> >> is loading this on the fly even possible? > > NO, that's the point. If we intend to pursue that path, imdiag always needs > to be loaded. That does not have any performance impact (but a security > impact!). Once imdiag is loaded, it could be used to do all sorts of > "interesting" things (most of which need to be implemented first...). it all depends on what imdiag can do, and where the control signals come from. if the contol is via normal syslog messages it is a problem, on the other hand if it creates a new socket that it listens on that can only be written to by root it's probably far less of a problem. I guess the real response is that I don't know enough about imdiag to know the risks. David Lang From correo at miguelangelnieto.net Wed Nov 4 15:00:35 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Wed, 4 Nov 2009 15:00:35 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # kernel logging (formerly provided by rklogd) $WorkDirectory /var/log/queue $MainMsgQueueFileName mainq $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "TVC" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "TVB" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "TTD" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "KCD" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "LPT" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "ABT" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "XET" @@10.10.0.100 & ~ *.* /var/log/syslog kern.* /dev/console *.info;mail.none;authpriv.none;cron.none -/var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* -/var/log/cron uucp,news.crit -/var/log/spooler local7.* /var/log/boot.log 2009/11/4 : > On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > >> I have a problem with the attached client-configuration. When I stop >> the server, the client computer hangs-up some minutes later and didn't >> write logs on $WorkDirectory /var/log/queue. > > this list strips attachments, please re-send with the config in the body of > the message. > > david Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From rgerhards at hq.adiscon.com Wed Nov 4 15:11:58 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 15:11:58 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103334@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> > > The problem is that you would not get the information BEFORE you > turned in > > on. In theory, it should even be possible with the current system and > a file, > > but I need to check this out (I never used it that way, but the > engine > > probably is capable of doing it). > > yes, this would not help see what triggered the problem, but could be a > huge help in figuring out what the current state of things are. definitely > going to a file could work as well. in either case a config option > would > have to be created to specify the destination. traditionally, this is done via environment options, please see: http://www.rsyslog.com/doc-debug.html It may very well work just be specifying a file and sending SIGUSR1 - but I have never tried that in a production run (which also means I *never* tested it). When I added this mechanism, I selected envrionment variables because that enables to change the settings without touching the config file. Obviously it wouldn't be much of a problem to create config options for them as well. > > >> would this have similar performance impact to running in debug mode? > or > >> does this need less serialization, and therefor less impact? > > > > Think of it as turning on debug mode on the fly. That exactly is what > it > > does. So before the signal, there is no impact, but after it is the > same > > impact as usual. > > even in a high volume production environment this can be tolorated for > short time periods. currently there is no way (short of stop, > reconfigure, > start, stop, reconfigure, start) to try and gather this info, and each > stop will loose logs, so this ability would be much better. > > >>> Alternatively, one could load imdiag into a suspect instance and > use > >> that to > >>> gain more in-depth information. However, I am not sure if that > would > >> be > >>> suitable for use in production, at least it is questionable from a > >> security > >>> POV. > >> > >> is loading this on the fly even possible? > > > > NO, that's the point. If we intend to pursue that path, imdiag always > needs > > to be loaded. That does not have any performance impact (but a > security > > impact!). Once imdiag is loaded, it could be used to do all sorts of > > "interesting" things (most of which need to be implemented first...). > > it all depends on what imdiag can do, and where the control signals > come > from. if the contol is via normal syslog messages it is a problem, on > the > other hand if it creates a new socket that it listens on that can only > be > written to by root it's probably far less of a problem. > > I guess the real response is that I don't know enough about imdiag to > know > the risks. Most of that needs to be written, so we can specify what is acceptable. So far it listens to a network socket. This was intended so that testcases can be build without the need to fiddle with local socket creation policies. One potential was would also to selectively enable/disable imdiag functionality inside the config file. The GUI front-end I (barely) started to write talks to imdiag, but this is not really anything decent so far. In the long term, I'd like to be able to pull things like queue status, modules loaded, action status and the like from a running instance as well as inject not only messages but change some settings on the fly. My initial need was to provide a consistent rsyslog/UDP monitoring solution (for performance testing) as well as problem analysis. Rainer > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Nov 4 15:13:37 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 15:13:37 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103336@GRFEXC.intern.adiscon.com> just a quick look: the file name is always "dbq" and that will quickly result in an abort when actions go to the queue. Make sure the file names are different. Could be the source of your problem, but not sure... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Miguel Angel Nieto > Sent: Wednesday, November 04, 2009 3:01 PM > To: rsyslog-users > Subject: Re: [rsyslog] computer hang-up and WorkDirectory > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > $WorkDirectory /var/log/queue > $MainMsgQueueFileName mainq > > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TVC" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TVB" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TTD" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "KCD" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "LPT" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "ABT" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "XET" @@10.10.0.100 > & ~ > > *.* /var/log/syslog > kern.* /dev/console > *.info;mail.none;authpriv.none;cron.none - > /var/log/messages > authpriv.* /var/log/secure > mail.* - > /var/log/maillog > cron.* -/var/log/cron > uucp,news.crit - > /var/log/spooler > local7.* > /var/log/boot.log > > > 2009/11/4 : > > On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > > > >> I have a problem with the attached client-configuration. When I stop > >> the server, the client computer hangs-up some minutes later and > didn't > >> write logs on $WorkDirectory /var/log/queue. > > > > this list strips attachments, please re-send with the config in the > body of > > the message. > > > > david Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > > > > -- > Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que > hablar. Si quer?an decirme algo, tendr?an que escribirlo en un > papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que > hablar el resto de mi vida. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Nov 4 15:27:03 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 06:27:03 -0800 (PST) Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: ok, looking at this I don't see that you have any commands that would use the work directory. now when you say the client computer locks up do you mean the following? you have a server writing logs you have a seperate client sending logs to the server you shut down the server later the client machine stops responding. is this config for the client or for the server? one possible explination for the freeze you are seeing is that if you have the client configured to send via TCP (the @@ option) and the server does not accept the message, the client will queue the message, when the client queue fills up it will not accept any more messages. many processes (including login) will block until syslog accepts the message causeing the machine to 'freeze' or 'lock up' does this match what you are seeing? if so, turning the server back on should un-freeze the client machines. if this is the case you need to decide your priorities how critical is it to get the logs off the machine? in some cases they are a real security issue and you must get them off (in which case you really should be using relp, not tcp, but that's a different discussion that rainer did a write-up on), and your only real answer is to setup multiple servers so that one is always up. in other cases you are willing to spill over to disk and risk having an intruder tamper with the logs before they get sent off to another machine and set the main queu type to disk assisted mode in other cases you are willing to loose logs rather than freezing the machine and can configure rsyslog to accept messages, even when it can't do anything with them to avoid this sort of lockup. Daivd Lang On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > $WorkDirectory /var/log/queue > $MainMsgQueueFileName mainq > > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TVC" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TVB" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TTD" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "KCD" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "LPT" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "ABT" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "XET" @@10.10.0.100 > & ~ > > *.* /var/log/syslog > kern.* /dev/console > *.info;mail.none;authpriv.none;cron.none -/var/log/messages > authpriv.* /var/log/secure > mail.* -/var/log/maillog > cron.* -/var/log/cron > uucp,news.crit -/var/log/spooler > local7.* /var/log/boot.log > > > 2009/11/4 : >> On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: >> >>> I have a problem with the attached client-configuration. When I stop >>> the server, the client computer hangs-up some minutes later and didn't >>> write logs on $WorkDirectory /var/log/queue. >> >> this list strips attachments, please re-send with the config in the body of >> the message. >> >> david Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > > > > From david at lang.hm Wed Nov 4 15:38:54 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 06:38:54 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 4 Nov 2009, Rainer Gerhards wrote: >>> The problem is that you would not get the information BEFORE you >> turned in >>> on. In theory, it should even be possible with the current system and >> a file, >>> but I need to check this out (I never used it that way, but the >> engine >>> probably is capable of doing it). >> >> yes, this would not help see what triggered the problem, but could be a >> huge help in figuring out what the current state of things are. > > definitely > >> going to a file could work as well. in either case a config option >> would >> have to be created to specify the destination. > > traditionally, this is done via environment options, please see: > > http://www.rsyslog.com/doc-debug.html > > It may very well work just be specifying a file and sending SIGUSR1 - but I > have never tried that in a production run (which also means I *never* tested > it). > > When I added this mechanism, I selected envrionment variables because that > enables to change the settings without touching the config file. Obviously it > wouldn't be much of a problem to create config options for them as well. given that in a production environment, rsyslog is started from an init script of some sort, setting environment variables before it starts is just editing a different config file. it would probably be much better to have it in the rsyslog config than to hunt down the correct (distro specific) init file and edit it to set an environemnt variable there. >>> NO, that's the point. If we intend to pursue that path, imdiag always >> needs >>> to be loaded. That does not have any performance impact (but a >> security >>> impact!). Once imdiag is loaded, it could be used to do all sorts of >>> "interesting" things (most of which need to be implemented first...). >> >> it all depends on what imdiag can do, and where the control signals >> come >> from. if the contol is via normal syslog messages it is a problem, on >> the >> other hand if it creates a new socket that it listens on that can only >> be >> written to by root it's probably far less of a problem. >> >> I guess the real response is that I don't know enough about imdiag to >> know >> the risks. > > Most of that needs to be written, so we can specify what is acceptable. So > far it listens to a network socket. This was intended so that testcases can > be build without the need to fiddle with local socket creation policies. One > potential was would also to selectively enable/disable imdiag functionality > inside the config file. > > The GUI front-end I (barely) started to write talks to imdiag, but this is > not really anything decent so far. In the long term, I'd like to be able to > pull things like queue status, modules loaded, action status and the like > from a running instance as well as inject not only messages but change some > settings on the fly. My initial need was to provide a consistent rsyslog/UDP > monitoring solution (for performance testing) as well as problem analysis. in a production environement you may not be able to use a GUI, and a network socket (on anything other than localhost) is definantly a problem. if you have it create a network listener on localhost that can be connected to via telnet it's not nearly as big of a security problem. then it can be scripted or a user can cut-n-paste to it while still allowing a nice GUI (or text based equivalent) to be created as time permits. but for now just use a simple line-based interface and it would be usable. if the listening address and port are configurable you can have the address default to 127.0.0.1 (relativly safe), but be configurable to 0.0.0.0 so that you can hit it over the network with other machines as needed. David Lang From rgerhards at hq.adiscon.com Wed Nov 4 15:57:37 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 15:57:37 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com> > given that in a production environment, rsyslog is started from an init > script of some sort, setting environment variables before it starts is > just editing a different config file. it would probably be much better > to > have it in the rsyslog config than to hunt down the correct (distro > specific) init file and edit it to set an environemnt variable there. I'll see if there is anything that prevents this, if not, I'll simply add the relevant statements. > > The GUI front-end I (barely) started to write talks to imdiag, but > this is > > not really anything decent so far. In the long term, I'd like to be > able to > > pull things like queue status, modules loaded, action status and the > like > > from a running instance as well as inject not only messages but > change some > > settings on the fly. My initial need was to provide a consistent > rsyslog/UDP > > monitoring solution (for performance testing) as well as problem > analysis. > > in a production environement you may not be able to use a GUI, and a > network socket (on anything other than localhost) is definantly a > problem. > > if you have it create a network listener on localhost that can be > connected to via telnet it's not nearly as big of a security problem. > then > it can be scripted or a user can cut-n-paste to it while still allowing > a > nice GUI (or text based equivalent) to be created as time permits. but > for > now just use a simple line-based interface and it would be usable. > > if the listening address and port are configurable you can have the > address default to 127.0.0.1 (relativly safe), but be configurable to > 0.0.0.0 so that you can hit it over the network with other machines as > needed. That's along the lines after which the current imdiag is designed. However, I was thinking to exchange some data via XML streams as it can be very lengthy (think of a complete queue stats dump for all queues in a system). XML would probably not work very well with the frontend, something readable would probably not work very well with the GUI - maybe the request should just specify the output format ;) Rainer From david at lang.hm Wed Nov 4 16:01:34 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 07:01:34 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 4 Nov 2009, Rainer Gerhards wrote: >>> The GUI front-end I (barely) started to write talks to imdiag, but >> this is >>> not really anything decent so far. In the long term, I'd like to be >> able to >>> pull things like queue status, modules loaded, action status and the >> like >>> from a running instance as well as inject not only messages but >> change some >>> settings on the fly. My initial need was to provide a consistent >> rsyslog/UDP >>> monitoring solution (for performance testing) as well as problem >> analysis. >> >> in a production environement you may not be able to use a GUI, and a >> network socket (on anything other than localhost) is definantly a >> problem. >> >> if you have it create a network listener on localhost that can be >> connected to via telnet it's not nearly as big of a security problem. >> then >> it can be scripted or a user can cut-n-paste to it while still allowing >> a >> nice GUI (or text based equivalent) to be created as time permits. but >> for >> now just use a simple line-based interface and it would be usable. >> >> if the listening address and port are configurable you can have the >> address default to 127.0.0.1 (relativly safe), but be configurable to >> 0.0.0.0 so that you can hit it over the network with other machines as >> needed. > > That's along the lines after which the current imdiag is designed. However, I > was thinking to exchange some data via XML streams as it can be very lengthy > (think of a complete queue stats dump for all queues in a system). XML would > probably not work very well with the frontend, something readable would > probably not work very well with the GUI - maybe the request should just > specify the output format ;) that can work, however XML output may not be too bad, people entering commands manually are probably not going to be asking for that much data. the biggest problem I see with XML is the need to send requests and responses via e-mail for debugging. if one end uses a HTML enabled e-mail client it may not work well to paste XML text into it. David Lang From rgerhards at hq.adiscon.com Wed Nov 4 16:06:36 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 16:06:36 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103338@GRFEXC.intern.adiscon.com> > that can work, however XML output may not be too bad, people entering > commands manually are probably not going to be asking for that much > data. If I look at time constraints, I will probably not able to define a big CLI with a real language. So I think you might send something like "queuestatus" and it will reply with all status information on all queues it has. > the biggest problem I see with XML is the need to send requests and > responses via e-mail for debugging. if one end uses a HTML enabled e- > mail > client it may not work well to paste XML text into it. That's a very good point and I really should begin to think about this right now. Some kind of "debug package" that's easily mailable would not be a bad thing ;) Rainer From correo at miguelangelnieto.net Wed Nov 4 16:14:03 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Wed, 4 Nov 2009 16:14:03 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: Hi, 2009/11/4 : > ok, looking at this I don't see that you have any commands that would use > the work directory. Ok, here I have the first bug. I supposed that rsyslog will use work directory to save queues of all TCP forwards. What I missed to get this functionality? > now when you say the client computer locks up do you mean the following? > > you have a server writing logs > you have a seperate client sending logs to the server > you shut down the server > later the client machine stops responding. The client stops responding. > is this config for the client or for the server? Client. > one possible explination for the freeze you are seeing is that if you have > the client configured to send via TCP (the @@ option) and the server does > not accept the message, the client will queue the message, when the client > queue fills up it will not accept any more messages. many processes > (including login) will block until syslog accepts the message causeing the > machine to 'freeze' or 'lock up' > > does this match what you are seeing? Yes. So, activating the disk queue should fix this problem. Is it true? I would set a limit of 5GB. > if this is the case you need to decide your priorities > > how critical is it to get the logs off the machine? We have a lot of computers logging to a centralized rsyslog server. In each computer runs a application logging diferent activities (webcams, doors, etc.) We need all the data in one computer to analize it. > in other cases you are willing to loose logs rather than freezing the > machine and can configure rsyslog to accept messages, even when it can't > do anything with them to avoid this sort of lockup. How Can I do that? Thank you! > Daivd Lang > > > On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > >> $ModLoad immark.so # provides --MARK-- message capability >> $ModLoad imuxsock.so # provides support for local system logging (e.g. >> via logger command) >> $ModLoad imklog.so # kernel logging (formerly provided by rklogd) >> >> $WorkDirectory /var/log/queue >> $MainMsgQueueFileName mainq >> >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TVC" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TVB" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TTD" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "KCD" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "LPT" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "ABT" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "XET" @@10.10.0.100 >> & ~ >> >> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? /var/log/syslog >> kern.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? /dev/console >> *.info;mail.none;authpriv.none;cron.none ? ? ? ? ? ? ? ?-/var/log/messages >> authpriv.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/secure >> mail.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/maillog >> cron.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/cron >> uucp,news.crit ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/spooler >> local7.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/boot.log >> >> >> 2009/11/4 ?: >>> On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: >>> >>>> I have a problem with the attached client-configuration. When I stop >>>> the server, the client computer hangs-up some minutes later and didn't >>>> write logs on $WorkDirectory /var/log/queue. >>> >>> this list strips attachments, please re-send with the config in the body of >>> the message. >>> >>> david Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> >> >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From mbiebl at gmail.com Wed Nov 4 19:30:20 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 4 Nov 2009 19:30:20 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 Message-ID: Hi Rainer, hi list I tried the latest v5 stable release. First thing I noticed is, that it still uses -c 4 as native mode. I also noticed problems with logging to a pipe. The default rsyslog.conf file on Debian has a daemon.*;mail.*;\ news.err;\ *.=debug;*.=info;\ *.=notice;*.=warn |/dev/xconsole But I get the following error: 9042.679173747:b7607b70: file to log to: |/dev/xconsole 9042.679183525:b7607b70: doWrite, pData->pStrm 0x814d8c0, lenBuf 131 9042.679194699:b7607b70: strm 0x814d8c0: file -1 flush, buflen 208 9042.679225429:b7607b70: strm 0x814d8c0: open error 2, file '|/dev/xconsole' I've put a complete debug log + config files at http://debs.michaelbiebl.de/rsyslog/v5 Rainer, maybe you can take a look. Thanks, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Wed Nov 4 21:17:09 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 21:17:09 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Michael, That is very probably the same regression that Luis Fernando mentioned for 4.5.5. All in all, I would be quite careful with 5.2.0, I really have the impression that folks simply didn't try it out. But as I wrote, I need to somehow get the process started to come to make v5 mainstream. It is really important to me to get rid of v4-devel, as it takes considerable time (and bug potential!) to have two branches that are in development state. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Wednesday, November 04, 2009 7:30 PM > To: rsyslog-users > Subject: [rsyslog] rsyslog v5 5.2.0 > > Hi Rainer, hi list > > I tried the latest v5 stable release. > First thing I noticed is, that it still uses -c 4 as native mode. > I also noticed problems with logging to a pipe. The default > rsyslog.conf file on Debian has a > daemon.*;mail.*;\ > news.err;\ > *.=debug;*.=info;\ > *.=notice;*.=warn |/dev/xconsole > > But I get the following error: > > 9042.679173747:b7607b70: file to log to: |/dev/xconsole > 9042.679183525:b7607b70: doWrite, pData->pStrm 0x814d8c0, lenBuf 131 > 9042.679194699:b7607b70: strm 0x814d8c0: file -1 flush, buflen 208 > 9042.679225429:b7607b70: strm 0x814d8c0: open error 2, file > '|/dev/xconsole' > > I've put a complete debug log + config files at > http://debs.michaelbiebl.de/rsyslog/v5 > > Rainer, maybe you can take a look. > > Thanks, > Michael > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From correo at miguelangelnieto.net Wed Nov 4 22:32:43 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Wed, 4 Nov 2009 22:32:43 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: Hi again, I simplified the rsyslog config file: $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # kernel logging (formerly provided by rklogd) $WorkDirectory /var/log/queue $ActionQueueType LinkedList $ActionQueueFileName prueba $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on *.* @@10.10.0.210 & ~ *.* /var/log/messages kern.* /dev/console *.info;mail.none;authpriv.none;cron.none -/var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* -/var/log/cron uucp,news.crit -/var/log/spooler local7.* /var/log/boot.log Now I have disk queue, but rsyslog only logs up to 3000 entries. 0719.224892620:imuxsock.c: --------imuxsock calling select, active file descriptors (max 3): 3 0719.233534436:imuxsock.c: Message from UNIX socket: 0000003 0719.233573550:imuxsock.c: logmsg: flags 4, from 'linux-92wq', msg Nov 4 22:38:39 punisher: pruebaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa 0719.233589475:imuxsock.c: Message has legacy syslog format. 0719.233643675:imuxsock.c: main queue: entry added, size now 3000 entries 0719.233659599:imuxsock.c: main queue: EnqueueMsg signaled condition (0) 0719.233674127:imuxsock.c: wtpAdviseMaxWorkers signals busy 0719.233688096:imuxsock.c: --------imuxsock calling select, active file descriptors (max 3): 3 0719.244104262:imuxsock.c: Message from UNIX socket: 0000003 0719.244145331:imuxsock.c: logmsg: flags 4, from 'linux-92wq', msg Nov 4 22:38:39 punisher: pruebaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa 0719.244161256:imuxsock.c: Message has legacy syslog format. 0719.244186680:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0720.818283853:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0721.819871573:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0722.828967380:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0723.832482270:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0724.839541947:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0725.848415367:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0726.855937980:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0727.858592935:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0728.862998771:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. Rsyslog 3.19.7 (Suse) 2009/11/4 : > ok, looking at this I don't see that you have any commands that would use > the work directory. > > now when you say the client computer locks up do you mean the following? > > you have a server writing logs > you have a seperate client sending logs to the server > you shut down the server > later the client machine stops responding. > > is this config for the client or for the server? > > one possible explination for the freeze you are seeing is that if you have > the client configured to send via TCP (the @@ option) and the server does > not accept the message, the client will queue the message, when the client > queue fills up it will not accept any more messages. many processes > (including login) will block until syslog accepts the message causeing the > machine to 'freeze' or 'lock up' > > does this match what you are seeing? > > if so, turning the server back on should un-freeze the client machines. > > > if this is the case you need to decide your priorities > > how critical is it to get the logs off the machine? in some cases they are > a real security issue and you must get them off (in which case you really > should be using relp, not tcp, but that's a different discussion that > rainer did a write-up on), and your only real answer is to setup multiple > servers so that one is always up. > > in other cases you are willing to spill over to disk and risk having an > intruder tamper with the logs before they get sent off to another machine > and set the main queu type to disk assisted mode > > in other cases you are willing to loose logs rather than freezing the > machine and can configure rsyslog to accept messages, even when it can't > do anything with them to avoid this sort of lockup. > > Daivd Lang > > > On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > >> $ModLoad immark.so # provides --MARK-- message capability >> $ModLoad imuxsock.so # provides support for local system logging (e.g. >> via logger command) >> $ModLoad imklog.so # kernel logging (formerly provided by rklogd) >> >> $WorkDirectory /var/log/queue >> $MainMsgQueueFileName mainq >> >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TVC" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TVB" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TTD" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "KCD" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "LPT" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "ABT" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "XET" @@10.10.0.100 >> & ~ >> >> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? /var/log/syslog >> kern.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? /dev/console >> *.info;mail.none;authpriv.none;cron.none ? ? ? ? ? ? ? ?-/var/log/messages >> authpriv.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/secure >> mail.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/maillog >> cron.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/cron >> uucp,news.crit ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/spooler >> local7.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/boot.log >> >> >> 2009/11/4 ?: >>> On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: >>> >>>> I have a problem with the attached client-configuration. When I stop >>>> the server, the client computer hangs-up some minutes later and didn't >>>> write logs on $WorkDirectory /var/log/queue. >>> >>> this list strips attachments, please re-send with the config in the body of >>> the message. >>> >>> david Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> >> >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From tbergfeld at hq.adiscon.com Thu Nov 5 08:08:00 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Thu, 5 Nov 2009 08:08:00 +0100 Subject: [rsyslog] rsyslog 5.3.4 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103340@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.3.4, a member of the v5-devel branch. Today's release offers a number of important improvements. I would like to highlight the ability to nest rulesets [1] (I already announced that) and the ability to define custom parsers [2]. Let me elaborate a bit on the later. In the syslog world exists a myriad of incompatible and invalidly formatted messages. We have had numerous support requests on how to parse this or that malformed message format. With custom parsers, we now have a vehicel to integrate all these invalid formats in an elegant way that is also offering high performance ([2] has a concrete sample). The core idea now is that parsers are plugins just like input and output modules. It is relatively easy to write such a parser (roughly a day, I expect) and custom parsers can be integrated into rsyslogd in a way that they only affect messages received on specific port. So far, I have not actually created a custom parser, but I hope that over time many of them will become available. My hope here is that we actually can build a directory, which other folks can browse so that they are able to find a solution to the mess their vendor has provided ;) This release also introduces the capability to run rulesets off their own queue (also already mentioned on the mailing list). Plus, it contains the usual set of bug fixes. We plan to make this version the basis for the next v5-stable. [1] http://www.rsyslog.com/doc-omruleset.html [2] http://www.rsyslog.com/doc-rsconf1_rulesetparser.html ChangeLog: http://www.rsyslog.com/Article423.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-185.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From david at lang.hm Thu Nov 5 08:53:12 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 23:53:12 -0800 (PST) Subject: [rsyslog] rsyslog 5.3.4 (devel) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103340@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103340@GRFEXC.intern.adiscon.com> Message-ID: yOn Thu, 5 Nov 2009, Tom Bergfeld wrote: > We have just released rsyslog 5.3.4, a member of the v5-devel branch. > Today's release offers a number of important improvements. I would like to > highlight the ability to nest rulesets [1] (I already announced that) and the > ability to define custom parsers [2]. > > Let me elaborate a bit on the later. In the syslog world exists a myriad of > incompatible and invalidly formatted messages. We have had numerous support > requests on how to parse this or that malformed message format. With custom > parsers, we now have a vehicel to integrate all these invalid formats in an > elegant way that is also offering high performance ([2] has a concrete > sample). The core idea now is that parsers are plugins just like input and > output modules. It is relatively easy to write such a parser (roughly a day, > I expect) and custom parsers can be integrated into rsyslogd in a way that > they only affect messages received on specific port. So far, I have not > actually created a custom parser, but I hope that over time many of them will > become available. My hope here is that we actually can build a directory, > which other folks can browse so that they are able to find a solution to the > mess their vendor has provided ;) this sounds very interesting. however, reading the link I'm a little confused. on the one hand it talks a lot about the priority of parsers, but then it talks about binding different parsers to different ports. It's not clear if these are two different ways to use parsers, or how these two would interact. where can these parsers be used? obviously the imudp module can use them. can all input modules use them? what are the limits of the parser? a couple of extreme examples: could you implement relp as a parser for imtcp, or since relp sends replies that can't be done (limiting it to different ways of processing a message that's already been received) could a parser detect that it had a piece of a multi-line messsage and stash the piece until it receives the rest of the message so that it could submit the complete message? a coouple questions from trying to look through the code at almost 3am local time if it can easily be done, may I suggest changing the santization flag from a yes/no boolean to an enum so that there can be more than one sanitation routine do you have the default parser broken out as a seperate file tha canbe used as an example? David Lang > This release also introduces the capability to run rulesets off their own > queue (also already mentioned on the mailing list). Plus, it contains the > usual set of bug fixes. > > We plan to make this version the basis for the next v5-stable. > > > [1] http://www.rsyslog.com/doc-omruleset.html > [2] http://www.rsyslog.com/doc-rsconf1_rulesetparser.html > > > > ChangeLog: > > http://www.rsyslog.com/Article423.phtml > > Download: > > http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-185.phtml > > As always, feedback is appreciated. > > Best regards, > Tom Bergfeld > -- > Support > ======= > > Improving rsyslog is costly, but you can help! We are looking for > organizations that find rsyslog useful and wish to contribute back. You can > contribute by reporting bugs, improve the software, or donate money or > equipment. > > Commercial support contracts for rsyslog are available, and they help finance > continued maintenance. Adiscon GmbH, a privately held German company, is > currently funding rsyslog development. We are always looking for interesting > development projects. For details on how to help, please see > http://www.rsyslog.com/doc-how2help.html . > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Nov 5 09:29:26 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 09:29:26 +0100 Subject: [rsyslog] rsyslog 5.3.4 (devel) released References: <9B6E2A8877C38245BFB15CC491A11DA7103340@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103342@GRFEXC.intern.adiscon.com> Good questions, thanks! :) I'll add some of the answers to the doc, what probably makes most sense. Just one thing: These are message parsers, not frame parsers. So you cannot parse RELP out of plain tcp, simply because that would be a frame parser (and even more, as these are totally different protocols). Message parsers work on the raw message, once it is extracted from the transport framing. I don't see any use case for permitting different frame parsers, as the framing is always bound to a specific protocol and a protocol has many more attributes than just the framing. The right way to implement a new protocol is via an input/output plugins. Currently, all message parsers are built into the core, but this is just a delivery mechanism (like with omfile, ...). The code is the same for loadable parsers. The two parsers I currently provide can be seen here: http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/pmrfc3164.c;h=5b684af56e 6d5e8ea2bcd616a06cc39e4cbb4a09;hb=6d9c54c7a2d4f07b0414082ef9681bd197ed6bde http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/pmrfc5424.c;h=07994adeba fb7e40d597e417dd1215874972971c;hb=6d9c54c7a2d4f07b0414082ef9681bd197ed6bde Rest comes either via doc or with a later reply... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, November 05, 2009 8:53 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 5.3.4 (devel) released > > yOn Thu, 5 Nov 2009, Tom Bergfeld wrote: > > > We have just released rsyslog 5.3.4, a member of the v5-devel branch. > > Today's release offers a number of important improvements. I would > like to > > highlight the ability to nest rulesets [1] (I already announced that) > and the > > ability to define custom parsers [2]. > > > > Let me elaborate a bit on the later. In the syslog world exists a > myriad of > > incompatible and invalidly formatted messages. We have had numerous > support > > requests on how to parse this or that malformed message format. With > custom > > parsers, we now have a vehicel to integrate all these invalid formats > in an > > elegant way that is also offering high performance ([2] has a > concrete > > sample). The core idea now is that parsers are plugins just like > input and > > output modules. It is relatively easy to write such a parser (roughly > a day, > > I expect) and custom parsers can be integrated into rsyslogd in a way > that > > they only affect messages received on specific port. So far, I have > not > > actually created a custom parser, but I hope that over time many of > them will > > become available. My hope here is that we actually can build a > directory, > > which other folks can browse so that they are able to find a solution > to the > > mess their vendor has provided ;) > > this sounds very interesting. however, reading the link I'm a little > confused. > > on the one hand it talks a lot about the priority of parsers, but then > it > talks about binding different parsers to different ports. It's not > clear > if these are two different ways to use parsers, or how these two would > interact. > > where can these parsers be used? > > obviously the imudp module can use them. can all input modules use > them? > > what are the limits of the parser? > > a couple of extreme examples: > > could you implement relp as a parser for imtcp, or since relp sends > replies that can't be done (limiting it to different ways of processing > a > message that's already been received) > > could a parser detect that it had a piece of a multi-line messsage and > stash the piece until it receives the rest of the message so that it > could > submit the complete message? > > > a coouple questions from trying to look through the code at almost 3am > local time > > if it can easily be done, may I suggest changing the santization flag > from > a yes/no boolean to an enum so that there can be more than one > sanitation > routine > > do you have the default parser broken out as a seperate file tha canbe > used as an example? > > David Lang > > > This release also introduces the capability to run rulesets off their > own > > queue (also already mentioned on the mailing list). Plus, it contains > the > > usual set of bug fixes. > > > > We plan to make this version the basis for the next v5-stable. > > > > > > [1] http://www.rsyslog.com/doc-omruleset.html > > [2] http://www.rsyslog.com/doc-rsconf1_rulesetparser.html > > > > > > > > ChangeLog: > > > > http://www.rsyslog.com/Article423.phtml > > > > Download: > > > > http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid- > 185.phtml > > > > As always, feedback is appreciated. > > > > Best regards, > > Tom Bergfeld > > -- > > Support > > ======= > > > > Improving rsyslog is costly, but you can help! We are looking for > > organizations that find rsyslog useful and wish to contribute back. > You can > > contribute by reporting bugs, improve the software, or donate money > or > > equipment. > > > > Commercial support contracts for rsyslog are available, and they help > finance > > continued maintenance. Adiscon GmbH, a privately held German > company, is > > currently funding rsyslog development. We are always looking for > interesting > > development projects. For details on how to help, please see > > http://www.rsyslog.com/doc-how2help.html . > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Nov 5 09:36:14 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 09:36:14 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103343@GRFEXC.intern.adiscon.com> > 0726.855937980:imuxsock.c: main queue: enqueueMsg: LightDelay mark > reached for light delayble message - blocking a bit. > 0727.858592935:imuxsock.c: main queue: enqueueMsg: LightDelay mark > reached for light delayble message - blocking a bit. > 0728.862998771:imuxsock.c: main queue: enqueueMsg: LightDelay mark > reached for light delayble message - blocking a bit. > > Rsyslog 3.19.7 (Suse) The version is the issue - that's an outdated beta version ;) Some versions had unix sockets flagged as sources that could lightly be delayed, resulting in what you see. That was made optional in later builds (and I guess nobody has turned it on since then ;)). Cure: upgrade to the current (3.22.1) v3-stable. Rainer From rgerhards at hq.adiscon.com Thu Nov 5 12:47:39 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 12:47:39 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5/5.2.0? References: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> <200911041414.34449.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103353@GRFEXC.intern.adiscon.com> Looks like a trivial regression, fix here: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=a5cd509be736fcff3c8ae310 4712d2fe85776416 I'll now do some more checks and see if I can include an automatted test for the pipe functionality so that a similar problem won't slip through in the future :) I will later apply the same fix to 5.2.0, it is the same problem. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luis Fernando Mu?oz Mej?as > Sent: Wednesday, November 04, 2009 2:15 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Output to named pipes not working on v4.5.5? > > So, this has two parts: > > 1) A SELinux problem on RHEL5 and similar (but the logs show > nothing!!). It seems the syslog context is not allowed to access > FIFOs. I'll have to fix that. > > 2) Even when there are no permission problems, rsyslog 4.5.5 just > doesn't write to a named pipe. From the following consecutive lines: > > *.* |/var/log/external/fifo > *.* /var/log/external/file > > the file receives data, the FIFO doesn't. > > I'll paste all this to bugzilla. > > Cheers. > -- > Luis Fernando Mu?oz Mej?as > Luis.Fernando.Munoz.Mejias at cern.ch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From martinmie at PartyGaming.com Thu Nov 5 13:51:38 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 5 Nov 2009 13:51:38 +0100 Subject: [rsyslog] rsyslog 5.2.0 meets signal 11. Message-ID: <0B1B9163138571439EAF804646F3F6AA08A586E8@GIBSVWIN004X.partygaming.local> Hi, Yesterday I successfully upgraded our logservers to the stable rsyslog 5.2.0. This morning I found out that the rsyslog process on the primary server was dead. Digging a bit in the logs I found this: -- # grep stop logserver.20091104.log 2009-11-04T07:32:01.970288-05:00 logserver kernel: Kernel logging (proc) stopped. -- According to the audit.log, it received a SIGSEGV (kill -11): -- type=ANOM_ABEND msg=audit(1257340663.009:19009): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=25881 comm="rsyslogd" sig=11 -- ...so, the application died on nov. 4th 2009 @ 8:17:43 EST -- # date -d @1257340663.009 Wed Nov 4 08:17:43 EST 2009 -- Has this been observed for this v5 stable branch?? And is there any fix? And, auditd was needed in order to determine the reason why rsyslogd was stopped. Could it be possible to have a more verbose message when the process stops due to an anomaly? Cheers, Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From Luis.Fernando.Munoz.Mejias at cern.ch Thu Nov 5 15:49:40 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?utf-8?q?Mu=C3=B1oz_Mej=C3=ADas?=) Date: Thu, 5 Nov 2009 15:49:40 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5/5.2.0? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103353@GRFEXC.intern.adiscon.com> References: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> <200911041414.34449.Luis.Fernando.Munoz.Mejias@cern.ch> <9B6E2A8877C38245BFB15CC491A11DA7103353@GRFEXC.intern.adiscon.com> Message-ID: <200911051549.40352.Luis.Fernando.Munoz.Mejias@cern.ch> Rainer, > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=a5cd509be736fcff3c8ae3 >10 4712d2fe85776416 I've cherry-picked this commit and I confirm it's fixed on v4.5.X. Thanks a lot for the quick action. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From rgerhards at hq.adiscon.com Thu Nov 5 15:51:15 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 15:51:15 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5/5.2.0? References: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch><200911041414.34449.Luis.Fernando.Munoz.Mejias@cern.ch><9B6E2A8877C38245BFB15CC491A11DA7103353@GRFEXC.intern.adiscon.com> <200911051549.40352.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103357@GRFEXC.intern.adiscon.com> Thanks for the feedback, much appreciated. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luis Fernando Mu?oz Mej?as > Sent: Thursday, November 05, 2009 3:50 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Output to named pipes not working on > v4.5.5/5.2.0? > > Rainer, > > > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=a5cd509be736fcff3c > 8ae3 > >10 4712d2fe85776416 > > I've cherry-picked this commit and I confirm it's fixed on > v4.5.X. Thanks a lot for the quick action. > -- > Luis Fernando Mu?oz Mej?as > Luis.Fernando.Munoz.Mejias at cern.ch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Thu Nov 5 16:50:16 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 5 Nov 2009 16:50:16 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: 2009/11/4 Rainer Gerhards : > Michael, > > That is very probably the same regression that Luis Fernando mentioned for > 4.5.5. All in all, I would be quite careful with 5.2.0, I really have the > impression that folks simply didn't try it out. But as I wrote, I need to > somehow get the process started to come to make v5 mainstream. It is really > important to me to get rid of v4-devel, as it takes considerable time (and > bug potential!) to have two branches that are in development state. I can confirm that a5cd509be736fcff3c8ae3104712d2fe85776416 fixes this issue for me using 5.2.0. What about the -c compat flag? Is it intentional that there is no -c 5? I will probably upload a 5.2.* to unstable (or experimental) soon, which should give it some wider exposure. Cheers, Michael > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Michael Biebl >> Sent: Wednesday, November 04, 2009 7:30 PM >> To: rsyslog-users >> Subject: [rsyslog] rsyslog v5 5.2.0 >> >> Hi Rainer, hi list >> >> I tried the latest v5 stable release. >> First thing I noticed is, that it still uses -c 4 as native mode. >> I also noticed problems with logging to a pipe. The default >> rsyslog.conf file on Debian has a >> daemon.*;mail.*;\ >> ? ? ? ? news.err;\ >> ? ? ? ? *.=debug;*.=info;\ >> ? ? ? ? *.=notice;*.=warn ? ? ? |/dev/xconsole >> >> But I get the following error: >> >> 9042.679173747:b7607b70: file to log to: |/dev/xconsole >> 9042.679183525:b7607b70: doWrite, pData->pStrm 0x814d8c0, lenBuf 131 >> 9042.679194699:b7607b70: strm 0x814d8c0: file -1 flush, buflen 208 >> 9042.679225429:b7607b70: strm 0x814d8c0: open error 2, file >> '|/dev/xconsole' >> >> I've put a complete debug log + config files at >> http://debs.michaelbiebl.de/rsyslog/v5 >> >> Rainer, maybe you can take a look. >> >> Thanks, >> Michael >> >> -- >> Why is it that all of the instruments seeking intelligent life in the >> universe are pointed away from Earth? >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Nov 5 16:53:34 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 16:53:34 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, November 05, 2009 4:50 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog v5 5.2.0 > > 2009/11/4 Rainer Gerhards : > > Michael, > > > > That is very probably the same regression that Luis Fernando > mentioned for > > 4.5.5. All in all, I would be quite careful with 5.2.0, I really have > the > > impression that folks simply didn't try it out. But as I wrote, I > need to > > somehow get the process started to come to make v5 mainstream. It is > really > > important to me to get rid of v4-devel, as it takes considerable time > (and > > bug potential!) to have two branches that are in development state. > > I can confirm that a5cd509be736fcff3c8ae3104712d2fe85776416 fixes this > issue for me using 5.2.0. > What about the -c compat flag? Is it intentional that there is no -c 5? > I will probably upload a 5.2.* to unstable (or experimental) soon, > which should give it some wider exposure. I'll look into that, soon. My hopes are to get past 5.2.0 as quickly as possible. I have done a lot of work on the queue engine in the recent development version and there are definitely a couple of issues in 5.2.0 which cannot be fixed without upgrading the 5.3 version. That just FYI. I will probably not work on any queue-related issues in 5.2 but rather recommend to move on to a newer version. However, for normal operations that do not require intensive use of the queues, 5.2.0 should work OK and be faster than pervious versions. Rainer > Cheers, > Michael > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Michael > Biebl > >> Sent: Wednesday, November 04, 2009 7:30 PM > >> To: rsyslog-users > >> Subject: [rsyslog] rsyslog v5 5.2.0 > >> > >> Hi Rainer, hi list > >> > >> I tried the latest v5 stable release. > >> First thing I noticed is, that it still uses -c 4 as native mode. > >> I also noticed problems with logging to a pipe. The default > >> rsyslog.conf file on Debian has a > >> daemon.*;mail.*;\ > >> ? ? ? ? news.err;\ > >> ? ? ? ? *.=debug;*.=info;\ > >> ? ? ? ? *.=notice;*.=warn ? ? ? |/dev/xconsole > >> > >> But I get the following error: > >> > >> 9042.679173747:b7607b70: file to log to: |/dev/xconsole > >> 9042.679183525:b7607b70: doWrite, pData->pStrm 0x814d8c0, lenBuf 131 > >> 9042.679194699:b7607b70: strm 0x814d8c0: file -1 flush, buflen 208 > >> 9042.679225429:b7607b70: strm 0x814d8c0: open error 2, file > >> '|/dev/xconsole' > >> > >> I've put a complete debug log + config files at > >> http://debs.michaelbiebl.de/rsyslog/v5 > >> > >> Rainer, maybe you can take a look. > >> > >> Thanks, > >> Michael > >> > >> -- > >> Why is it that all of the instruments seeking intelligent life in > the > >> universe are pointed away from Earth? > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Thu Nov 5 17:02:32 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 5 Nov 2009 17:02:32 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710335A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710335A@GRFEXC.intern.adiscon.com> Message-ID: Another issue I noticed with 5.2.0 (+said patch) is, that it no longer shuts down cleany on SIGTERM/SIGINT If I run it with -c 4 -d and press STRG+C, I get ... 6895.847489380:b7692940: Terminating outputs... 6895.847516758:b7692940: destructing ruleset 0x89c7648, name 0x89c7678 6895.847553634:b7692940: strm 0x89c97b8: file -1 flush, buflen 0 6895.847595539:b7692940: strm 0x89ca1d8: file 5 flush, buflen 0 6895.847627386:b7692940: strm 0x89ca1d8: file 5 closing 6895.847688567:b7692940: strm 0x89cac48: file -1 flush, buflen 0 6895.847728796:b7692940: strm 0x89cb680: file 6 flush, buflen 0 6895.847758967:b7692940: strm 0x89cb680: file 6 closing 6895.847805062:b7692940: strm 0x89cc108: file -1 flush, buflen 0 6895.847846129:b7692940: strm 0x89ccb78: file -1 flush, buflen 0 6895.847886916:b7692940: strm 0x89cd5c0: file -1 flush, buflen 0 6895.847927145:b7692940: strm 0x89ce060: file -1 flush, buflen 0 6895.847967653:b7692940: strm 0x89cea98: file -1 flush, buflen 0 6895.848007882:b7692940: strm 0x89cf500: file -1 flush, buflen 0 6895.848048110:b7692940: strm 0x89cff68: file -1 flush, buflen 0 6895.848088897:b7692940: strm 0x89d09d8: file -1 flush, buflen 0 6895.848129126:b7692940: strm 0x89d1478: file -1 flush, buflen 0 6895.848169355:b7692940: strm 0x89d1ee8: file -1 flush, buflen 0 6895.848220758:b7692940: strm 0x89d2990: file 7 flush, buflen 0 6895.848251488:b7692940: strm 0x89d2990: file 7 closing 6895.848300098:b7692940: strm 0x89d38c0: file -1 flush, buflen 77 pressing STRG+C a second time will then terminate rsyslogd. From mrdemeanour at jackpot.uk.net Thu Nov 5 17:14:05 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Thu, 05 Nov 2009 16:14:05 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls Message-ID: <4AF2F9CD.9000402@jackpot.uk.net> Hi, I'm running a central rsyslog server with a couple of remote WAN (internet) clients and several remote LAN clients. Traffic is low - of the order of 10,000 messages per day. Internet clients communicate with the server using gnutls. LAN clients are currently using UDP. The server writes client logs to mysql, and also writes messages of local origin to disk. After some interval x::x <= 24h, the server consumes all memory (and swap) in the system. It looks as if the OS then tries to evict rsyslogd, and hangs - there's what looks like a stacktrace displayed on the (dead) console. I've stopped using gnutls for now, because the server machine is also a NFS file-server, and other systems depend on it not shutting down like this. It seems the problem doesn't occur with plain tcp. # uname -a Linux prajna 2.6.30-2-686 #1 SMP Sat Sep 26 01:16:22 UTC 2009 i686 GNU/Linux # rsyslogd -v rsyslogd 4.4.2, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: Yes FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: Yes FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: Yes Runtime Instrumentation (slow code): No Here's some log output from around the time of one of these incidents; I don't know if it helps, but the data on the console after the incident looks a lot like the end of this set of entries. Nov 4 08:10:05 prajna rsyslogd: [origin software="rsyslogd" swVersion="4.4.2" x-pid="15425" x-info="http://www.rsyslog.com"] (re)start Nov 4 08:10:05 prajna kernel: [47633.000709] Killed process 13665 (rsyslogd) Nov 4 08:10:05 prajna kernel: [47633.000612] 191505 pages non-shared Nov 4 08:10:05 prajna kernel: [47633.000636] Out of memory: kill process 13665 (rsyslogd) score 14137 or a child Nov 4 08:10:05 prajna kernel: [47633.000575] 2870 pages reserved Nov 4 08:10:05 prajna kernel: [47633.000594] 221 pages shared Nov 4 08:10:05 prajna kernel: [47633.000529] 196608 pages RAM Nov 4 08:10:05 prajna kernel: [47633.000557] 0 pages HighMem Nov 4 08:10:05 prajna kernel: [47632.970129] Free swap = 0kB Nov 4 08:10:05 prajna kernel: [47632.970148] Total swap = 1951856kB Nov 4 08:10:05 prajna kernel: [47632.970081] 0 pages in swap cache Nov 4 08:10:05 prajna kernel: [47632.970103] Swap cache stats: add 497948, delete 497948, find 68248/68574 Nov 4 08:10:05 prajna kernel: [47632.970061] 209 total pagecache pages Nov 4 08:10:05 prajna kernel: [47632.969982] Normal: 6*4kB 1*8kB 1*16kB 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3440kB Nov 4 08:10:05 prajna kernel: [47632.969873] lowmem_reserve[]: 0 0 0 0 Nov 4 08:10:05 prajna kernel: [47632.969905] DMA: 1*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3052kB Nov 4 08:10:05 prajna kernel: [47632.969763] lowmem_reserve[]: 0 746 746 746 Nov 4 08:10:05 prajna kernel: [47632.969804] Normal free:3456kB min:3460kB low:4324kB high:5188kB active_anon:366232kB inactive_anon:366324kB active_file:608kB inactive_file:156kB unevictable:64kB present:764032kB pages_scanned:1512018 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47632.968951] free:1627 slab:1995 mapped:1 pagetables:944 bounce:0 Nov 4 08:10:05 prajna kernel: [47632.969024] DMA free:3052kB min:68kB low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB active_file:0kB inactive_file:16kB unevictable:0kB present:15868kB pages_scanned:15745 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47632.968946] inactive_file:43 unevictable:16 dirty:0 writeback:0 unstable:0 Nov 4 08:10:05 prajna kernel: [47632.968909] CPU 0: hi: 186, btch: 31 usd: 171 Nov 4 08:10:05 prajna kernel: [47632.968941] Active_anon:92646 active_file:152 inactive_anon:92691 Nov 4 08:10:05 prajna kernel: [47632.968889] Normal per-cpu: Nov 4 08:10:05 prajna kernel: [47632.968827] Mem-Info: Nov 4 08:10:05 prajna kernel: [47632.968846] DMA per-cpu: Nov 4 08:10:05 prajna kernel: [47632.968866] CPU 0: hi: 0, btch: 1 usd: 0 Nov 4 08:10:05 prajna kernel: [47632.968803] [] ? do_page_fault+0x0/0x1e7 Nov 4 08:10:05 prajna kernel: [47632.968731] [] ? do_page_fault+0x0/0x1e7 Nov 4 08:10:05 prajna kernel: [47632.968774] [] ? error_code+0x6d/0x74 Nov 4 08:10:05 prajna kernel: [47632.968702] [] ? do_page_fault+0x1d8/0x1e7 Nov 4 08:10:05 prajna kernel: [47632.968623] [] ? handle_mm_fault+0x2bb/0x652 Nov 4 08:10:05 prajna kernel: [47632.968658] [] ? fd_install+0x1e/0x3c Nov 4 08:10:05 prajna kernel: [47632.968592] [] ? irq_exit+0x31/0x53 Nov 4 08:10:05 prajna kernel: [47632.968549] [] ? __do_fault+0x47/0x355 Nov 4 08:10:05 prajna kernel: [47632.968508] [] ? kunmap_atomic+0x63/0x72 Nov 4 08:10:05 prajna kernel: [47632.968442] [] ? do_page_cache_readahead+0x3d/0x47 Nov 4 08:10:05 prajna kernel: [47632.968474] [] ? filemap_fault+0x154/0x315 Nov 4 08:10:05 prajna kernel: [47632.968410] [] ? __do_page_cache_readahead+0x98/0x16b Nov 4 08:10:05 prajna kernel: [47632.968376] [] ? __alloc_pages_internal+0x2ee/0x39d Nov 4 08:10:05 prajna kernel: [47632.968342] [] ? out_of_memory+0x5a/0x7c Nov 4 08:10:05 prajna kernel: [47632.968311] [] ? __out_of_memory+0x33/0x10b Nov 4 08:10:05 prajna kernel: [47632.968231] Call Trace: Nov 4 08:10:05 prajna kernel: [47632.968279] [] ? oom_kill_process+0x79/0x1f7 Nov 4 08:10:05 prajna kernel: [47632.968173] hald-addon-stor cpuset=/ mems_allowed=0 Nov 4 08:10:05 prajna kernel: [47632.968204] Pid: 1965, comm: hald-addon-stor Not tainted 2.6.30-2-686 #1 Nov 4 08:10:05 prajna kernel: [47632.968114] hald-addon-stor invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 Nov 4 08:10:05 prajna kernel: [47631.204598] Out of memory: kill process 6420 (rsyslogd) score 21205 or a child Nov 4 08:10:05 prajna kernel: [47631.204667] Killed process 6420 (rsyslogd) Nov 4 08:10:05 prajna kernel: [47631.204555] 207 pages shared Nov 4 08:10:05 prajna kernel: [47631.204573] 191533 pages non-shared Nov 4 08:10:05 prajna kernel: [47631.204518] 0 pages HighMem Nov 4 08:10:05 prajna kernel: [47631.204536] 2870 pages reserved Nov 4 08:10:05 prajna kernel: [47631.174161] Total swap = 1951856kB Nov 4 08:10:05 prajna kernel: [47631.204489] 196608 pages RAM Nov 4 08:10:05 prajna kernel: [47631.174142] Free swap = 0kB Nov 4 08:10:05 prajna kernel: [47631.174093] 14 pages in swap cache Nov 4 08:10:05 prajna kernel: [47631.174116] Swap cache stats: add 497939, delete 497925, find 68248/68574 Nov 4 08:10:05 prajna kernel: [47631.173995] Normal: 4*4kB 1*8kB 1*16kB 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3432kB Nov 4 08:10:05 prajna kernel: [47631.174073] 230 total pagecache pages Nov 4 08:10:05 prajna kernel: [47631.173917] DMA: 1*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3052kB Nov 4 08:10:05 prajna kernel: [47631.173817] Normal free:3448kB min:3460kB low:4324kB high:5188kB active_anon:366252kB inactive_anon:366360kB active_file:280kB inactive_file:584kB unevictable:64kB present:764032kB pages_scanned:1830820 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47631.173885] lowmem_reserve[]: 0 0 0 0 Nov 4 08:10:05 prajna kernel: [47631.173775] lowmem_reserve[]: 0 746 746 746 Nov 4 08:10:05 prajna kernel: [47631.173710] DMA free:3052kB min:68kB low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB active_file:0kB inactive_file:0kB unevictable:0kB present:15868kB pages_scanned:15745 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47631.173637] free:1625 slab:2000 mapped:1 pagetables:944 bounce:0 Nov 4 08:10:05 prajna kernel: [47631.173627] Active_anon:92651 active_file:70 inactive_anon:92700 Nov 4 08:10:05 prajna kernel: [47631.173632] inactive_file:146 unevictable:16 dirty:0 writeback:0 unstable:0 Nov 4 08:10:05 prajna kernel: [47631.173575] Normal per-cpu: Nov 4 08:10:05 prajna kernel: [47631.173596] CPU 0: hi: 186, btch: 31 usd: 154 Nov 4 08:10:05 prajna kernel: [47631.173532] DMA per-cpu: Nov 4 08:10:05 prajna kernel: [47631.173552] CPU 0: hi: 0, btch: 1 usd: 0 Nov 4 08:10:05 prajna kernel: [47631.173514] Mem-Info: Nov 4 08:10:05 prajna kernel: [47631.173461] [] ? error_code+0x6d/0x74 Nov 4 08:10:05 prajna kernel: [47631.173490] [] ? do_page_fault+0x0/0x1e7 Nov 4 08:10:05 prajna kernel: [47631.173420] [] ? do_page_fault+0x0/0x1e7 Nov 4 08:10:05 prajna kernel: [47631.173348] [] ? sys_socketcall+0x103/0x19f Nov 4 08:10:05 prajna kernel: [47631.173390] [] ? do_page_fault+0x1d8/0x1e7 Nov 4 08:10:05 prajna kernel: [47631.173319] [] ? sys_recv+0x19/0x1d Nov 4 08:10:05 prajna kernel: [47631.173280] [] ? handle_mm_fault+0x2bb/0x652 Nov 4 08:10:05 prajna kernel: [47631.173208] [] ? kunmap_atomic+0x63/0x72 Nov 4 08:10:05 prajna kernel: [47631.173248] [] ? __do_fault+0x47/0x355 Nov 4 08:10:05 prajna kernel: [47631.173173] [] ? filemap_fault+0x154/0x315 Nov 4 08:10:05 prajna kernel: [47631.173110] [] ? __do_page_cache_readahead+0x98/0x16b Nov 4 08:10:05 prajna kernel: [47631.173142] [] ? do_page_cache_readahead+0x3d/0x47 Nov 4 08:10:05 prajna kernel: [47631.173011] [] ? __out_of_memory+0x33/0x10b Nov 4 08:10:05 prajna kernel: [47631.173042] [] ? out_of_memory+0x5a/0x7c Nov 4 08:10:05 prajna kernel: [47631.173076] [] ? __alloc_pages_internal+0x2ee/0x39d Nov 4 08:10:05 prajna kernel: [47631.172979] [] ? oom_kill_process+0x79/0x1f7 Nov 4 08:10:05 prajna kernel: [47631.172935] Call Trace: Nov 4 08:10:05 prajna kernel: [47631.172909] Pid: 13665, comm: rsyslogd Not tainted 2.6.30-2-686 #1 Nov 4 08:10:05 prajna kernel: [47631.172829] rsyslogd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 Nov 4 08:10:05 prajna kernel: [47631.172881] rsyslogd cpuset=/ mems_allowed=0 Nov 4 08:10:05 prajna kernel: [47631.136403] Killed process 13664 (rsyslogd) Nov 4 08:10:05 prajna kernel: [47631.136310] 191499 pages non-shared Nov 4 08:10:05 prajna kernel: [47631.136334] Out of memory: kill process 13664 (rsyslogd) score 42411 or a child Nov 4 08:10:05 prajna kernel: [47631.136272] 2870 pages reserved Nov 4 08:10:05 prajna kernel: [47631.136291] 203 pages shared Nov 4 08:10:05 prajna kernel: [47631.136226] 196608 pages RAM Nov 4 08:10:05 prajna kernel: [47631.136254] 0 pages HighMem Nov 4 08:10:05 prajna kernel: [47631.105740] Total swap = 1951856kB Nov 4 08:10:05 prajna kernel: [47631.105722] Free swap = 0kB Nov 4 08:10:05 prajna kernel: [47631.105696] Swap cache stats: add 497920, delete 497920, find 68247/68572 Nov 4 08:10:05 prajna kernel: [47631.105674] 0 pages in swap cache Nov 4 08:10:05 prajna kernel: [47631.105653] 152 total pagecache pages Nov 4 08:10:05 prajna kernel: [47631.105574] Normal: 33*4kB 2*8kB 1*16kB 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3556kB Nov 4 08:10:05 prajna kernel: [47631.105496] DMA: 1*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3052kB Nov 4 08:10:05 prajna kernel: [47631.105465] lowmem_reserve[]: 0 0 0 0 Nov 4 08:10:05 prajna kernel: [47631.105396] Normal free:3556kB min:3460kB low:4324kB high:5188kB active_anon:366288kB inactive_anon:366268kB active_file:280kB inactive_file:324kB unevictable:64kB present:764032kB pages_scanned:1830304 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47631.105355] lowmem_reserve[]: 0 746 746 746 Nov 4 08:10:05 prajna kernel: [47631.105289] DMA free:3052kB min:68kB low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB active_file:0kB inactive_file:0kB unevictable:0kB present:15868kB pages_scanned:15745 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47631.105217] free:1652 slab:2000 mapped:1 pagetables:944 bounce:0 Nov 4 08:10:05 prajna kernel: [47631.105207] Active_anon:92660 active_file:70 inactive_anon:92677 Nov 4 08:10:05 prajna kernel: [47631.105212] inactive_file:81 unevictable:16 dirty:0 writeback:0 unstable:0 Nov 4 08:10:05 prajna kernel: [47631.105155] Normal per-cpu: Nov 4 08:10:05 prajna kernel: [47631.105175] CPU 0: hi: 186, btch: 31 usd: 161 Nov 4 08:10:05 prajna kernel: [47631.105132] CPU 0: hi: 0, btch: 1 usd: 0 Nov 4 08:10:05 prajna kernel: [47631.105093] Mem-Info: Nov 4 08:10:05 prajna kernel: [47631.105112] DMA per-cpu: Nov 4 08:10:05 prajna kernel: [47631.105069] [] ? do_page_fault+0x0/0x1e7 Nov 4 08:10:05 prajna kernel: imklog 4.4.2, log source = /proc/kmsg started. Nov 4 08:10:05 prajna kernel: [] ? error_code+0x6d/0x74 From rgerhards at hq.adiscon.com Thu Nov 5 17:17:16 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:17:16 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710335A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335B@GRFEXC.intern.adiscon.com> Hi Michael, Is this a standard debian config? I ask because it works well for me... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, November 05, 2009 5:03 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog v5 5.2.0 > > Another issue I noticed with 5.2.0 (+said patch) is, that it no longer > shuts down cleany on SIGTERM/SIGINT > If I run it with -c 4 -d and press STRG+C, I get > ... > 6895.847489380:b7692940: Terminating outputs... > 6895.847516758:b7692940: destructing ruleset 0x89c7648, name 0x89c7678 > 6895.847553634:b7692940: strm 0x89c97b8: file -1 flush, buflen 0 > 6895.847595539:b7692940: strm 0x89ca1d8: file 5 flush, buflen 0 > 6895.847627386:b7692940: strm 0x89ca1d8: file 5 closing > 6895.847688567:b7692940: strm 0x89cac48: file -1 flush, buflen 0 > 6895.847728796:b7692940: strm 0x89cb680: file 6 flush, buflen 0 > 6895.847758967:b7692940: strm 0x89cb680: file 6 closing > 6895.847805062:b7692940: strm 0x89cc108: file -1 flush, buflen 0 > 6895.847846129:b7692940: strm 0x89ccb78: file -1 flush, buflen 0 > 6895.847886916:b7692940: strm 0x89cd5c0: file -1 flush, buflen 0 > 6895.847927145:b7692940: strm 0x89ce060: file -1 flush, buflen 0 > 6895.847967653:b7692940: strm 0x89cea98: file -1 flush, buflen 0 > 6895.848007882:b7692940: strm 0x89cf500: file -1 flush, buflen 0 > 6895.848048110:b7692940: strm 0x89cff68: file -1 flush, buflen 0 > 6895.848088897:b7692940: strm 0x89d09d8: file -1 flush, buflen 0 > 6895.848129126:b7692940: strm 0x89d1478: file -1 flush, buflen 0 > 6895.848169355:b7692940: strm 0x89d1ee8: file -1 flush, buflen 0 > 6895.848220758:b7692940: strm 0x89d2990: file 7 flush, buflen 0 > 6895.848251488:b7692940: strm 0x89d2990: file 7 closing > 6895.848300098:b7692940: strm 0x89d38c0: file -1 flush, buflen 77 > > > pressing STRG+C a second time will then terminate rsyslogd. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Nov 5 17:23:32 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:23:32 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335C@GRFEXC.intern.adiscon.com> Michael, > I can confirm that a5cd509be736fcff3c8ae3104712d2fe85776416 fixes this > issue for me using 5.2.0. > What about the -c compat flag? Is it intentional that there is no -c 5? > I will probably upload a 5.2.* to unstable (or experimental) soon, > which should give it some wider exposure. one more side-note: to do this, do you need a "stable" tag attached to the version by the upstream? I am asking because it would be much more useful to have wider exposure for 5.3.4, whereas 5.2.0 will probably not provide that much useful feedback (the most interesting pieces inside the queue engine already have been rewritten in 5.3). Still, it is good to see that the stable tag now draws attention to the version, I begin to think that nobody ever tried the beta ;) Maybe I should attach stable tags earlier in the future. Rainer From david at lang.hm Thu Nov 5 17:19:17 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 5 Nov 2009 08:19:17 -0800 (PST) Subject: [rsyslog] rsyslog v5 5.2.0 In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 5 Nov 2009, Michael Biebl wrote: > 2009/11/4 Rainer Gerhards : >> Michael, >> >> That is very probably the same regression that Luis Fernando mentioned for >> 4.5.5. All in all, I would be quite careful with 5.2.0, I really have the >> impression that folks simply didn't try it out. But as I wrote, I need to >> somehow get the process started to come to make v5 mainstream. It is really >> important to me to get rid of v4-devel, as it takes considerable time (and >> bug potential!) to have two branches that are in development state. > > I can confirm that a5cd509be736fcff3c8ae3104712d2fe85776416 fixes this > issue for me using 5.2.0. > What about the -c compat flag? Is it intentional that there is no -c 5? are you sure there isn't? I've been running dev versions of 5.x and hav eneeded to use -c5 with them instead of -c4 > I will probably upload a 5.2.* to unstable (or experimental) soon, > which should give it some wider exposure. I would not use 5.2.0, it had many problems under load (and problems where invalid syslog messages could cause it to crash) I'm currently running a slightly later git snapshot reliably under fairly heavy load (>10k logs/sec sustained for hours at a time). unfortunantly I am 3 timezones away from the office or I would upgrade to the 5.3 that was just released. I will proabaly do so tuesday when I am back in the office. David Lang > Cheers, > Michael > > >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com >>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Michael Biebl >>> Sent: Wednesday, November 04, 2009 7:30 PM >>> To: rsyslog-users >>> Subject: [rsyslog] rsyslog v5 5.2.0 >>> >>> Hi Rainer, hi list >>> >>> I tried the latest v5 stable release. >>> First thing I noticed is, that it still uses -c 4 as native mode. >>> I also noticed problems with logging to a pipe. The default >>> rsyslog.conf file on Debian has a >>> daemon.*;mail.*;\ >>> ? ? ? ? news.err;\ >>> ? ? ? ? *.=debug;*.=info;\ >>> ? ? ? ? *.=notice;*.=warn ? ? ? |/dev/xconsole >>> >>> But I get the following error: >>> >>> 9042.679173747:b7607b70: file to log to: |/dev/xconsole >>> 9042.679183525:b7607b70: doWrite, pData->pStrm 0x814d8c0, lenBuf 131 >>> 9042.679194699:b7607b70: strm 0x814d8c0: file -1 flush, buflen 208 >>> 9042.679225429:b7607b70: strm 0x814d8c0: open error 2, file >>> '|/dev/xconsole' >>> >>> I've put a complete debug log + config files at >>> http://debs.michaelbiebl.de/rsyslog/v5 >>> >>> Rainer, maybe you can take a look. >>> >>> Thanks, >>> Michael >>> >>> -- >>> Why is it that all of the instruments seeking intelligent life in the >>> universe are pointed away from Earth? >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > > > From rgerhards at hq.adiscon.com Thu Nov 5 17:29:52 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:29:52 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335D@GRFEXC.intern.adiscon.com> > > I will probably upload a 5.2.* to unstable (or experimental) soon, > > which should give it some wider exposure. > > I would not use 5.2.0, it had many problems under load (and problems > where > invalid syslog messages could cause it to crash) The invalid message problem should be fixed in 5.2.0, this was merged in to all v5 versions from the v4 fix. But I won't touch any queue problems, and I am sure there exist some during shutdown. > I'm currently running a slightly later git snapshot reliably under > fairly > heavy load (>10k logs/sec sustained for hours at a time). unfortunantly > I > am 3 timezones away from the office or I would upgrade to the 5.3 that > was > just released. I will proabaly do so tuesday when I am back in the > office. Looking forward to the results. I intend to promote that version to beta soon and try to push it through a very short beta cycle. I do not plan any more enhancements within the short-term timeframe. I looks like the advanced features begin to get used and so finally bug reports come in, so it is bug fixing and instrumentation time :) Rainer From correo at miguelangelnieto.net Thu Nov 5 17:31:33 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Thu, 5 Nov 2009 17:31:33 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103343@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103343@GRFEXC.intern.adiscon.com> Message-ID: Thank you! It works now! :) This is the final rsyslog.conf ############## $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # kernel logging (formerly provided by rklogd) $WorkDirectory /var/log/queue $ActionQueueType LinkedList $ActionQueueFileName SL $ActionQueueMaxDiskSpace 5g $ActionQueueMaxFileSize 10m $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 if $msg contains 'TL' or $msg contains 'CD' or $msg contains 'TR' then @@192.168.1.222 & ~ *.* /var/log/syslog kern.* /dev/console *.info;mail.none;authpriv.none;cron.none -/var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* -/var/log/cron uucp,news.crit -/var/log/spooler local7.* /var/log/boot.log ############# 2009/11/5 Rainer Gerhards : >> 0726.855937980:imuxsock.c: main queue: enqueueMsg: LightDelay mark >> reached for light delayble message - blocking a bit. >> 0727.858592935:imuxsock.c: main queue: enqueueMsg: LightDelay mark >> reached for light delayble message - blocking a bit. >> 0728.862998771:imuxsock.c: main queue: enqueueMsg: LightDelay mark >> reached for light delayble message - blocking a bit. >> >> Rsyslog 3.19.7 (Suse) > > The version is the issue - that's an outdated beta version ;) Some versions > had unix sockets flagged as sources that could lightly be delayed, resulting > in what you see. That was made optional in later builds (and I guess nobody > has turned it on since then ;)). > > Cure: upgrade to the current (3.22.1) v3-stable. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From rgerhards at hq.adiscon.com Thu Nov 5 17:37:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:37:44 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> Can you send me your rsyslog.conf, so that I can run it under the memory debugger in my lab. I'll also take this as a motivation to finally add multi-daemon tests to the testbench (what may take me a little while...). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Thursday, November 05, 2009 5:14 PM > To: rsyslog at lists.adiscon.com >> rsyslog-users > Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Hi, > > I'm running a central rsyslog server with a couple of remote WAN > (internet) clients and several remote LAN clients. Traffic is low - of > the order of 10,000 messages per day. Internet clients communicate with > the server using gnutls. LAN clients are currently using UDP. The > server > writes client logs to mysql, and also writes messages of local origin > to > disk. > > After some interval x::x <= 24h, the server consumes all memory (and > swap) in the system. It looks as if the OS then tries to evict > rsyslogd, > and hangs - there's what looks like a stacktrace displayed on the > (dead) > console. > > I've stopped using gnutls for now, because the server machine is also a > NFS file-server, and other systems depend on it not shutting down like > this. It seems the problem doesn't occur with plain tcp. > > # uname -a > Linux prajna 2.6.30-2-686 #1 SMP Sat Sep 26 01:16:22 UTC 2009 i686 > GNU/Linux > > # rsyslogd -v > rsyslogd 4.4.2, compiled with: > FEATURE_REGEXP: Yes > FEATURE_LARGEFILE: Yes > FEATURE_NETZIP (message compression): Yes > GSSAPI Kerberos 5 support: Yes > FEATURE_DEBUG (debug build, slow code): No > Atomic operations supported: Yes > Runtime Instrumentation (slow code): No > > Here's some log output from around the time of one of these incidents; > I > don't know if it helps, but the data on the console after the incident > looks a lot like the end of this set of entries. > > Nov 4 08:10:05 prajna rsyslogd: [origin software="rsyslogd" > swVersion="4.4.2" x-pid="15425" x-info="http://www.rsyslog.com"] > (re)start > Nov 4 08:10:05 prajna kernel: [47633.000709] Killed process 13665 > (rsyslogd) > Nov 4 08:10:05 prajna kernel: [47633.000612] 191505 pages non-shared > Nov 4 08:10:05 prajna kernel: [47633.000636] Out of memory: kill > process 13665 (rsyslogd) score 14137 or a child > Nov 4 08:10:05 prajna kernel: [47633.000575] 2870 pages reserved > Nov 4 08:10:05 prajna kernel: [47633.000594] 221 pages shared > Nov 4 08:10:05 prajna kernel: [47633.000529] 196608 pages RAM > Nov 4 08:10:05 prajna kernel: [47633.000557] 0 pages HighMem > Nov 4 08:10:05 prajna kernel: [47632.970129] Free swap = 0kB > Nov 4 08:10:05 prajna kernel: [47632.970148] Total swap = 1951856kB > Nov 4 08:10:05 prajna kernel: [47632.970081] 0 pages in swap cache > Nov 4 08:10:05 prajna kernel: [47632.970103] Swap cache stats: add > 497948, delete 497948, find 68248/68574 > Nov 4 08:10:05 prajna kernel: [47632.970061] 209 total pagecache pages > Nov 4 08:10:05 prajna kernel: [47632.969982] Normal: 6*4kB 1*8kB > 1*16kB > 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = > 3440kB > Nov 4 08:10:05 prajna kernel: [47632.969873] lowmem_reserve[]: 0 0 0 0 > Nov 4 08:10:05 prajna kernel: [47632.969905] DMA: 1*4kB 1*8kB 0*16kB > 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = > 3052kB > Nov 4 08:10:05 prajna kernel: [47632.969763] lowmem_reserve[]: 0 746 > 746 746 > Nov 4 08:10:05 prajna kernel: [47632.969804] Normal free:3456kB > min:3460kB low:4324kB high:5188kB active_anon:366232kB > inactive_anon:366324kB active_file:608kB inactive_file:156kB > unevictable:64kB present:764032kB pages_scanned:1512018 > all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47632.968951] free:1627 slab:1995 > mapped:1 pagetables:944 bounce:0 > Nov 4 08:10:05 prajna kernel: [47632.969024] DMA free:3052kB min:68kB > low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB > active_file:0kB inactive_file:16kB unevictable:0kB present:15868kB > pages_scanned:15745 all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47632.968946] inactive_file:43 > unevictable:16 dirty:0 writeback:0 unstable:0 > Nov 4 08:10:05 prajna kernel: [47632.968909] CPU 0: hi: 186, btch: > 31 usd: 171 > Nov 4 08:10:05 prajna kernel: [47632.968941] Active_anon:92646 > active_file:152 inactive_anon:92691 > Nov 4 08:10:05 prajna kernel: [47632.968889] Normal per-cpu: > Nov 4 08:10:05 prajna kernel: [47632.968827] Mem-Info: > Nov 4 08:10:05 prajna kernel: [47632.968846] DMA per-cpu: > Nov 4 08:10:05 prajna kernel: [47632.968866] CPU 0: hi: 0, btch: > 1 usd: 0 > Nov 4 08:10:05 prajna kernel: [47632.968803] [] ? > do_page_fault+0x0/0x1e7 > Nov 4 08:10:05 prajna kernel: [47632.968731] [] ? > do_page_fault+0x0/0x1e7 > Nov 4 08:10:05 prajna kernel: [47632.968774] [] ? > error_code+0x6d/0x74 > Nov 4 08:10:05 prajna kernel: [47632.968702] [] ? > do_page_fault+0x1d8/0x1e7 > Nov 4 08:10:05 prajna kernel: [47632.968623] [] ? > handle_mm_fault+0x2bb/0x652 > Nov 4 08:10:05 prajna kernel: [47632.968658] [] ? > fd_install+0x1e/0x3c > Nov 4 08:10:05 prajna kernel: [47632.968592] [] ? > irq_exit+0x31/0x53 > Nov 4 08:10:05 prajna kernel: [47632.968549] [] ? > __do_fault+0x47/0x355 > Nov 4 08:10:05 prajna kernel: [47632.968508] [] ? > kunmap_atomic+0x63/0x72 > Nov 4 08:10:05 prajna kernel: [47632.968442] [] ? > do_page_cache_readahead+0x3d/0x47 > Nov 4 08:10:05 prajna kernel: [47632.968474] [] ? > filemap_fault+0x154/0x315 > Nov 4 08:10:05 prajna kernel: [47632.968410] [] ? > __do_page_cache_readahead+0x98/0x16b > Nov 4 08:10:05 prajna kernel: [47632.968376] [] ? > __alloc_pages_internal+0x2ee/0x39d > Nov 4 08:10:05 prajna kernel: [47632.968342] [] ? > out_of_memory+0x5a/0x7c > Nov 4 08:10:05 prajna kernel: [47632.968311] [] ? > __out_of_memory+0x33/0x10b > Nov 4 08:10:05 prajna kernel: [47632.968231] Call Trace: > Nov 4 08:10:05 prajna kernel: [47632.968279] [] ? > oom_kill_process+0x79/0x1f7 > Nov 4 08:10:05 prajna kernel: [47632.968173] hald-addon-stor cpuset=/ > mems_allowed=0 > Nov 4 08:10:05 prajna kernel: [47632.968204] Pid: 1965, comm: > hald-addon-stor Not tainted 2.6.30-2-686 #1 > Nov 4 08:10:05 prajna kernel: [47632.968114] hald-addon-stor invoked > oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 > Nov 4 08:10:05 prajna kernel: [47631.204598] Out of memory: kill > process 6420 (rsyslogd) score 21205 or a child > Nov 4 08:10:05 prajna kernel: [47631.204667] Killed process 6420 > (rsyslogd) > Nov 4 08:10:05 prajna kernel: [47631.204555] 207 pages shared > Nov 4 08:10:05 prajna kernel: [47631.204573] 191533 pages non-shared > Nov 4 08:10:05 prajna kernel: [47631.204518] 0 pages HighMem > Nov 4 08:10:05 prajna kernel: [47631.204536] 2870 pages reserved > Nov 4 08:10:05 prajna kernel: [47631.174161] Total swap = 1951856kB > Nov 4 08:10:05 prajna kernel: [47631.204489] 196608 pages RAM > Nov 4 08:10:05 prajna kernel: [47631.174142] Free swap = 0kB > Nov 4 08:10:05 prajna kernel: [47631.174093] 14 pages in swap cache > Nov 4 08:10:05 prajna kernel: [47631.174116] Swap cache stats: add > 497939, delete 497925, find 68248/68574 > Nov 4 08:10:05 prajna kernel: [47631.173995] Normal: 4*4kB 1*8kB > 1*16kB > 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = > 3432kB > Nov 4 08:10:05 prajna kernel: [47631.174073] 230 total pagecache pages > Nov 4 08:10:05 prajna kernel: [47631.173917] DMA: 1*4kB 1*8kB 0*16kB > 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = > 3052kB > Nov 4 08:10:05 prajna kernel: [47631.173817] Normal free:3448kB > min:3460kB low:4324kB high:5188kB active_anon:366252kB > inactive_anon:366360kB active_file:280kB inactive_file:584kB > unevictable:64kB present:764032kB pages_scanned:1830820 > all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47631.173885] lowmem_reserve[]: 0 0 0 0 > Nov 4 08:10:05 prajna kernel: [47631.173775] lowmem_reserve[]: 0 746 > 746 746 > Nov 4 08:10:05 prajna kernel: [47631.173710] DMA free:3052kB min:68kB > low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB > active_file:0kB inactive_file:0kB unevictable:0kB present:15868kB > pages_scanned:15745 all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47631.173637] free:1625 slab:2000 > mapped:1 pagetables:944 bounce:0 > Nov 4 08:10:05 prajna kernel: [47631.173627] Active_anon:92651 > active_file:70 inactive_anon:92700 > Nov 4 08:10:05 prajna kernel: [47631.173632] inactive_file:146 > unevictable:16 dirty:0 writeback:0 unstable:0 > Nov 4 08:10:05 prajna kernel: [47631.173575] Normal per-cpu: > Nov 4 08:10:05 prajna kernel: [47631.173596] CPU 0: hi: 186, btch: > 31 usd: 154 > Nov 4 08:10:05 prajna kernel: [47631.173532] DMA per-cpu: > Nov 4 08:10:05 prajna kernel: [47631.173552] CPU 0: hi: 0, btch: > 1 usd: 0 > Nov 4 08:10:05 prajna kernel: [47631.173514] Mem-Info: > Nov 4 08:10:05 prajna kernel: [47631.173461] [] ? > error_code+0x6d/0x74 > Nov 4 08:10:05 prajna kernel: [47631.173490] [] ? > do_page_fault+0x0/0x1e7 > Nov 4 08:10:05 prajna kernel: [47631.173420] [] ? > do_page_fault+0x0/0x1e7 > Nov 4 08:10:05 prajna kernel: [47631.173348] [] ? > sys_socketcall+0x103/0x19f > Nov 4 08:10:05 prajna kernel: [47631.173390] [] ? > do_page_fault+0x1d8/0x1e7 > Nov 4 08:10:05 prajna kernel: [47631.173319] [] ? > sys_recv+0x19/0x1d > Nov 4 08:10:05 prajna kernel: [47631.173280] [] ? > handle_mm_fault+0x2bb/0x652 > Nov 4 08:10:05 prajna kernel: [47631.173208] [] ? > kunmap_atomic+0x63/0x72 > Nov 4 08:10:05 prajna kernel: [47631.173248] [] ? > __do_fault+0x47/0x355 > Nov 4 08:10:05 prajna kernel: [47631.173173] [] ? > filemap_fault+0x154/0x315 > Nov 4 08:10:05 prajna kernel: [47631.173110] [] ? > __do_page_cache_readahead+0x98/0x16b > Nov 4 08:10:05 prajna kernel: [47631.173142] [] ? > do_page_cache_readahead+0x3d/0x47 > Nov 4 08:10:05 prajna kernel: [47631.173011] [] ? > __out_of_memory+0x33/0x10b > Nov 4 08:10:05 prajna kernel: [47631.173042] [] ? > out_of_memory+0x5a/0x7c > Nov 4 08:10:05 prajna kernel: [47631.173076] [] ? > __alloc_pages_internal+0x2ee/0x39d > Nov 4 08:10:05 prajna kernel: [47631.172979] [] ? > oom_kill_process+0x79/0x1f7 > Nov 4 08:10:05 prajna kernel: [47631.172935] Call Trace: > Nov 4 08:10:05 prajna kernel: [47631.172909] Pid: 13665, comm: > rsyslogd > Not tainted 2.6.30-2-686 #1 > Nov 4 08:10:05 prajna kernel: [47631.172829] rsyslogd invoked > oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 > Nov 4 08:10:05 prajna kernel: [47631.172881] rsyslogd cpuset=/ > mems_allowed=0 > Nov 4 08:10:05 prajna kernel: [47631.136403] Killed process 13664 > (rsyslogd) > Nov 4 08:10:05 prajna kernel: [47631.136310] 191499 pages non-shared > Nov 4 08:10:05 prajna kernel: [47631.136334] Out of memory: kill > process 13664 (rsyslogd) score 42411 or a child > Nov 4 08:10:05 prajna kernel: [47631.136272] 2870 pages reserved > Nov 4 08:10:05 prajna kernel: [47631.136291] 203 pages shared > Nov 4 08:10:05 prajna kernel: [47631.136226] 196608 pages RAM > Nov 4 08:10:05 prajna kernel: [47631.136254] 0 pages HighMem > Nov 4 08:10:05 prajna kernel: [47631.105740] Total swap = 1951856kB > Nov 4 08:10:05 prajna kernel: [47631.105722] Free swap = 0kB > Nov 4 08:10:05 prajna kernel: [47631.105696] Swap cache stats: add > 497920, delete 497920, find 68247/68572 > Nov 4 08:10:05 prajna kernel: [47631.105674] 0 pages in swap cache > Nov 4 08:10:05 prajna kernel: [47631.105653] 152 total pagecache pages > Nov 4 08:10:05 prajna kernel: [47631.105574] Normal: 33*4kB 2*8kB > 1*16kB 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB > = 3556kB > Nov 4 08:10:05 prajna kernel: [47631.105496] DMA: 1*4kB 1*8kB 0*16kB > 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = > 3052kB > Nov 4 08:10:05 prajna kernel: [47631.105465] lowmem_reserve[]: 0 0 0 0 > Nov 4 08:10:05 prajna kernel: [47631.105396] Normal free:3556kB > min:3460kB low:4324kB high:5188kB active_anon:366288kB > inactive_anon:366268kB active_file:280kB inactive_file:324kB > unevictable:64kB present:764032kB pages_scanned:1830304 > all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47631.105355] lowmem_reserve[]: 0 746 > 746 746 > Nov 4 08:10:05 prajna kernel: [47631.105289] DMA free:3052kB min:68kB > low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB > active_file:0kB inactive_file:0kB unevictable:0kB present:15868kB > pages_scanned:15745 all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47631.105217] free:1652 slab:2000 > mapped:1 pagetables:944 bounce:0 > Nov 4 08:10:05 prajna kernel: [47631.105207] Active_anon:92660 > active_file:70 inactive_anon:92677 > Nov 4 08:10:05 prajna kernel: [47631.105212] inactive_file:81 > unevictable:16 dirty:0 writeback:0 unstable:0 > Nov 4 08:10:05 prajna kernel: [47631.105155] Normal per-cpu: > Nov 4 08:10:05 prajna kernel: [47631.105175] CPU 0: hi: 186, btch: > 31 usd: 161 > Nov 4 08:10:05 prajna kernel: [47631.105132] CPU 0: hi: 0, btch: > 1 usd: 0 > Nov 4 08:10:05 prajna kernel: [47631.105093] Mem-Info: > Nov 4 08:10:05 prajna kernel: [47631.105112] DMA per-cpu: > Nov 4 08:10:05 prajna kernel: [47631.105069] [] ? > do_page_fault+0x0/0x1e7 > Nov 4 08:10:05 prajna kernel: imklog 4.4.2, log source = /proc/kmsg > started. > Nov 4 08:10:05 prajna kernel: [] ? error_code+0x6d/0x74 > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mrdemeanour at jackpot.uk.net Thu Nov 5 17:41:15 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Thu, 05 Nov 2009 16:41:15 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> References: <4AF2F9CD.9000402@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> Message-ID: <4AF3002B.9060302@jackpot.uk.net> Rainer Gerhards wrote: > Can you send me your rsyslog.conf, so that I can run it under the memory > debugger in my lab. I'll also take this as a motivation to finally add > multi-daemon tests to the testbench (what may take me a little while...). This is the server config (some of the remarks are misleading). # /etc/rsyslog.conf Configuration file for rsyslog v3. # # For more information see # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html # $DebugPrintTemplateList on # $ActionFileDefaultTemplate mysql-template ################# #### MODULES #### ################# $ModLoad imuxsock # provides support for local system logging # $ModLoad ommysql.so # provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 $ModLoad imklog # provides kernel logging support (previously done by rklogd) # provides TCP syslog reception $ModLoad imtcp # make gtls driver the default # $DefaultNetstreamDriver gtls $DefaultNetstreamDriver ptcp # certificate files $DefaultNetstreamDriverCAFile /etc/rsyslog.d/.ssl/gnu-ca-cert.pem $DefaultNetstreamDriverCertFile /etc/rsyslog.d/.ssl/saraha-rsyslog-cert.pem $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/.ssl/saraha-rsyslog-key.pem # $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode # $InputTCPServerStreamDriverAuthMode anon # client is NOT authenticated $InputTCPServerRun 10514 # start up listener at port 10514 $ModLoad MySQL ########################### ########################### #### GLOBAL DIRECTIVES #### ########################### # # Use default timestamp format. # To enable high precision timestamps, comment out the following line. # $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # # Set the default permissions for all log files. # $FileOwner root $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $WorkDirectory /var/log/rsyslog $template mysql-template, "insert into logs(host, facility, priority, level, tag, datetime, msg) values ('%source%', '%syslogfacility-text%', '%syslogpriority-text%', '%syslogseverity-text%', '%programname%', '%timereported:::date-mysql%', '%msg%')", sql, mysql # $template DEBUG,"Debug line with all properties:\nFROMHOST: '%FROMHOST%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nrawmsg: '%rawmsg%'\n\n" ############### #### RULES #### ############### #Discard some dross messages # authpriv.info ~ :HOSTNAME, isequal, "last" ~ #Discard router access messages for the script on Prajna that collects the router logs. if $msg contains 'User logged in on TELNET (192.168.1.2)' then ~ if $msg contains 'User logged out on TELNET (192.168.1.2)' then ~ # Log everything else to mysql. $ActionQueueType LinkedList # Number of elements... $ActionQueueSize 100 # $ActionQueueFileName mysql # $ActionQueueMaxDiskSpace 1M # $ActionQueueHighWaterMark 40 # $ActionQueueLowWaterMark 5 *.* >127.0.0.1,syslog,syslog,syslog;mysql-template $ActionExecOnlyWhenPreviousIsSuspended on & ~ $ActionExecOnlyWhenPreviousIsSuspended off #Log local stuff ONLY to /var/log/syslog :HOSTNAME, isequal, "prajna" -/var/log/syslog -- Jack. From rgerhards at hq.adiscon.com Thu Nov 5 17:42:01 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:42:01 +0100 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com><200910201639.43143.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> <200911031158.02666.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335F@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, November 03, 2009 11:58 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Hello Rainer, > > is there some news about this issue? Unfortuantely not at the moment (I guess you see it is a busy time). Could you give 5.3.4 a try? It would be very interesting (even for a v4-fix) to see if the problem persists or not... Rainer > > TIA > -Marc > > Am Dienstag, 20. Oktober 2009 18:38:16 schrieb Rainer Gerhards: > > thanks! > > > > Interesting, I see that only one of the messages that should be in > the .qi > > are actually present. I wonder if this is related to the fact that > ompgsql > > emits an error message itself while being down (the others don't do > this > > AFIK). Will try to dig down to this (but I have to do so as time > permits). > > > > Rainer > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Nov 5 17:45:07 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:45:07 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> Thanks, but I wasn't specific enough. For TLS, I also need to client config, because I need two machines to reproduce any issues (these two instances are also the challenge for the current testbench, what requires hopefully fewer than I expect changes ;)). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Thursday, November 05, 2009 5:41 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Rainer Gerhards wrote: > > Can you send me your rsyslog.conf, so that I can run it under the > memory > > debugger in my lab. I'll also take this as a motivation to finally > add > > multi-daemon tests to the testbench (what may take me a little > while...). > > This is the server config (some of the remarks are misleading). > > # /etc/rsyslog.conf Configuration file for rsyslog v3. > # > # For more information see > # /usr/share/doc/rsyslog- > doc/html/rsyslog_conf.html > > # $DebugPrintTemplateList on > # $ActionFileDefaultTemplate mysql-template > ################# > #### MODULES #### > ################# > > $ModLoad imuxsock # provides support for local system logging > > # $ModLoad ommysql.so > > # provides UDP syslog reception > $ModLoad imudp > $UDPServerRun 514 > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > > # provides TCP syslog reception > $ModLoad imtcp > > # make gtls driver the default > # $DefaultNetstreamDriver gtls > $DefaultNetstreamDriver ptcp > > # certificate files > $DefaultNetstreamDriverCAFile /etc/rsyslog.d/.ssl/gnu-ca-cert.pem > $DefaultNetstreamDriverCertFile /etc/rsyslog.d/.ssl/saraha-rsyslog- > cert.pem > $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/.ssl/saraha-rsyslog- > key.pem > > # $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode > # $InputTCPServerStreamDriverAuthMode anon # client is NOT > authenticated > $InputTCPServerRun 10514 # start up listener at port 10514 > > $ModLoad MySQL > > ########################### > ########################### > #### GLOBAL DIRECTIVES #### > ########################### > > # > # Use default timestamp format. > # To enable high precision timestamps, comment out the following line. > # > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > # > # Set the default permissions for all log files. > # > $FileOwner root > $FileGroup adm > $FileCreateMode 0640 > $DirCreateMode 0755 > $WorkDirectory /var/log/rsyslog > > $template mysql-template, "insert into logs(host, facility, priority, > level, tag, datetime, msg) values ('%source%', '%syslogfacility-text%', > '%syslogpriority-text%', '%syslogseverity-text%', '%programname%', > '%timereported:::date-mysql%', '%msg%')", sql, mysql > > # $template DEBUG,"Debug line with all properties:\nFROMHOST: > '%FROMHOST%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag > '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', > PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', > STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nrawmsg: > '%rawmsg%'\n\n" > > ############### > #### RULES #### > ############### > #Discard some dross messages > # authpriv.info ~ > :HOSTNAME, isequal, "last" ~ > > #Discard router access messages for the script on Prajna that collects > the router logs. > if $msg contains 'User logged in on TELNET (192.168.1.2)' then ~ > if $msg contains 'User logged out on TELNET (192.168.1.2)' then ~ > > # Log everything else to mysql. > $ActionQueueType LinkedList > # Number of elements... > $ActionQueueSize 100 > # $ActionQueueFileName mysql > # $ActionQueueMaxDiskSpace 1M > # $ActionQueueHighWaterMark 40 > # $ActionQueueLowWaterMark 5 > > > *.* > >127.0.0.1,syslog,syslog,syslog;mysql-template > > $ActionExecOnlyWhenPreviousIsSuspended on > & ~ > $ActionExecOnlyWhenPreviousIsSuspended off > > #Log local stuff ONLY to /var/log/syslog > :HOSTNAME, isequal, "prajna" -/var/log/syslog > > > -- > Jack. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Thu Nov 5 17:50:14 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 5 Nov 2009 17:50:14 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: 2009/11/5 : > On Thu, 5 Nov 2009, Michael Biebl wrote: > >> What about the -c compat flag? Is it intentional that there is no -c 5? > > are you sure there isn't? I've been running dev versions of 5.x and hav > eneeded to use -c5 with them instead of -c4 Well, at least with 5.2.0, rsyslogd does not complain if I start it with -c4 (complain in the sense, that rsyslog says it's running in non-native mode) >> I will probably upload a 5.2.* to unstable (or experimental) soon, >> which should give it some wider exposure. > > I would not use 5.2.0, it had many problems under load (and problems where > invalid syslog messages could cause it to crash) > > I'm currently running a slightly later git snapshot reliably under fairly > heavy load (>10k logs/sec sustained for hours at a time). unfortunantly I am > 3 timezones away from the office or I would upgrade to the 5.3 that was just > released. I will proabaly do so tuesday when I am back in the office. That's good to know. It seems the 5.2.* series is some kind of neglected so maybe it's best to just skip that branch. and wait for 5.4.* or stable 5.3.* beta releases. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Thu Nov 5 17:51:05 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 5 Nov 2009 17:51:05 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710335B@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710335A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710335B@GRFEXC.intern.adiscon.com> Message-ID: 2009/11/5 Rainer Gerhards : > Hi Michael, > > Is this a standard debian config? I ask because it works well for me... It's a standard Debian config as shipped with version 4.4.1-1 > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Michael Biebl >> Sent: Thursday, November 05, 2009 5:03 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] rsyslog v5 5.2.0 >> >> Another issue I noticed with 5.2.0 (+said patch) is, that it no longer >> shuts down cleany on SIGTERM/SIGINT >> If I run it with -c 4 -d and press STRG+C, I get >> ... >> 6895.847489380:b7692940: Terminating outputs... >> 6895.847516758:b7692940: destructing ruleset 0x89c7648, name 0x89c7678 >> 6895.847553634:b7692940: strm 0x89c97b8: file -1 flush, buflen 0 >> 6895.847595539:b7692940: strm 0x89ca1d8: file 5 flush, buflen 0 >> 6895.847627386:b7692940: strm 0x89ca1d8: file 5 closing >> 6895.847688567:b7692940: strm 0x89cac48: file -1 flush, buflen 0 >> 6895.847728796:b7692940: strm 0x89cb680: file 6 flush, buflen 0 >> 6895.847758967:b7692940: strm 0x89cb680: file 6 closing >> 6895.847805062:b7692940: strm 0x89cc108: file -1 flush, buflen 0 >> 6895.847846129:b7692940: strm 0x89ccb78: file -1 flush, buflen 0 >> 6895.847886916:b7692940: strm 0x89cd5c0: file -1 flush, buflen 0 >> 6895.847927145:b7692940: strm 0x89ce060: file -1 flush, buflen 0 >> 6895.847967653:b7692940: strm 0x89cea98: file -1 flush, buflen 0 >> 6895.848007882:b7692940: strm 0x89cf500: file -1 flush, buflen 0 >> 6895.848048110:b7692940: strm 0x89cff68: file -1 flush, buflen 0 >> 6895.848088897:b7692940: strm 0x89d09d8: file -1 flush, buflen 0 >> 6895.848129126:b7692940: strm 0x89d1478: file -1 flush, buflen 0 >> 6895.848169355:b7692940: strm 0x89d1ee8: file -1 flush, buflen 0 >> 6895.848220758:b7692940: strm 0x89d2990: file 7 flush, buflen 0 >> 6895.848251488:b7692940: strm 0x89d2990: file 7 closing >> 6895.848300098:b7692940: strm 0x89d38c0: file -1 flush, buflen 77 >> >> >> pressing STRG+C a second time will then terminate rsyslogd. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Nov 5 17:54:10 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:54:10 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103361@GRFEXC.intern.adiscon.com> > That's good to know. It seems the 5.2.* series is some kind of > neglected so maybe it's best to just skip that branch. and wait for > 5.4.* or stable 5.3.* beta releases. As I said on-list (and within the announcement), I needed to get a 5.2.0 version out. The problem was that there was so few feedback on 5.1.x that it looked like it worked. However, I knew that there are problems with queue termination. That all is solved (hopefully ;)) in the newer v5 versions, but I can't renumber them to a lower version to make them 5.2.x stable). Still it would be good if we could push out some of the new 5.3.x builds, maybe after they become beta, because otherwise we may end up with the same problem... (no testers, no feedback, no bugs, but still not working while I pursue for other things, thus never arriving at a really stable build...) Rainer From mrdemeanour at jackpot.uk.net Thu Nov 5 18:02:23 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Thu, 05 Nov 2009 17:02:23 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> Message-ID: <4AF3051F.5020501@jackpot.uk.net> Rainer Gerhards wrote: > Thanks, but I wasn't specific enough. For TLS, I also need to client config, > because I need two machines to reproduce any issues (these two instances are > also the challenge for the current testbench, what requires hopefully fewer > than I expect changes ;)). Sorry, Rainer. Anyway, I sent you the *current* config, i.e. using ptls. Here are the two configs using gnutls. But NOTE: I'm not using your default MySQL schema; you can't just drop this into your testlab. It should work if you ignore the custom MySQL template. I *really* doubt this has anything to do with MySQL - I've been using this MySQL setup for a year. Also note that there's nothing included from /etc/rsyslog.d - that directory is empty. These are the complete configs. There are a bunch of *Queue* directives in these files, both active and commented-out; I started playing around with queues to see if I could straighten it out that way, but it didn't work. That is, the problem should occur with a default queueing setup. ====== Start Server ========= # /etc/rsyslog.conf Configuration file for rsyslog v3. # # For more information see # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html # $DebugPrintTemplateList on # $ActionFileDefaultTemplate mysql-template ################# #### MODULES #### ################# $ModLoad imuxsock # provides support for local system logging # $ModLoad ommysql.so # provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 $ModLoad imklog # provides kernel logging support (previously done by rklogd) # provides TCP syslog reception $ModLoad imtcp # make gtls driver the default $DefaultNetstreamDriver gtls # $DefaultNetstreamDriver ptcp # certificate files $DefaultNetstreamDriverCAFile /etc/rsyslog.d/.ssl/gnu-ca-cert.pem $DefaultNetstreamDriverCertFile /etc/rsyslog.d/.ssl/saraha-rsyslog-cert.pem $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/.ssl/saraha-rsyslog-key.pem $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode $InputTCPServerStreamDriverAuthMode anon # client is NOT authenticated $InputTCPServerRun 10514 # start up listener at port 10514 $ModLoad MySQL ########################### #### GLOBAL DIRECTIVES #### ########################### # # Use default timestamp format. # To enable high precision timestamps, comment out the following line. # $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # # Set the default permissions for all log files. # $FileOwner root $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $WorkDirectory /var/log/rsyslog $template mysql-template, "insert into logs(host, facility, priority, level, tag, datetime, msg) values ('%source%', '%syslogfacility-text%', '%syslogpriority-text%', '%syslogseverity-text%', '%programname%', '%timereported:::date-mysql%', '%msg%')", sql, mysql # $template DEBUG,"Debug line with all properties:\nFROMHOST: '%FROMHOST%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nrawmsg: '%rawmsg%'\n\n" ############### #### RULES #### ############### #Discard some dross messages # authpriv.info ~ :HOSTNAME, isequal, "last" ~ #Discard router access messages for the script on Prajna that collects the router logs. if $msg contains 'User logged in on TELNET (192.168.1.2)' then ~ if $msg contains 'User logged out on TELNET (192.168.1.2)' then ~ # Log everything else to mysql. $ActionQueueType LinkedList # Number of elements... $ActionQueueSize 100 # $ActionQueueFileName mysql # $ActionQueueMaxDiskSpace 1M # $ActionQueueHighWaterMark 40 # $ActionQueueLowWaterMark 5 *.* >127.0.0.1,syslog,syslog,syslog;mysql-template $ActionExecOnlyWhenPreviousIsSuspended on & ~ $ActionExecOnlyWhenPreviousIsSuspended off #Log local stuff ONLY to /var/log/syslog :HOSTNAME, isequal, "prajna" -/var/log/syslog ====== End Server ========= ====== Start Client ========= # /etc/rsyslog.conf Configuration file for rsyslog v3. # # For more information see # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html ################# #### MODULES #### ################# $ModLoad imuxsock # provides support for local system logging $ModLoad imklog # provides kernel logging support (previously done by rklogd) #$ModLoad immark # provides --MARK-- message capability # provides UDP syslog reception #$ModLoad imudp #$UDPServerRun 514 # provides TCP syslog reception # $ModLoad imtcp # $InputTCPServerRun 514 # certificate files - just CA for a client $DefaultNetstreamDriverCAFile /etc/rsyslog.d/.ssl/gnu-ca-cert.pem # set up the action $DefaultNetstreamDriver gtls # use gtls netstream driver # $DefaultNetstreamDriver ptcp # use default netstream driver $ActionSendStreamDriverMode 1 # require TLS for the connection $ActionSendStreamDriverAuthMode anon # server is NOT authenticated ########################### #### GLOBAL DIRECTIVES #### ########################### # # Use traditional timestamp format. # To enable high precision timestamps, comment out the following line. # $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # # Set the default permissions for all log files. # $FileOwner root $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 # # Include all config files in /etc/rsyslog.d/ # $IncludeConfig /etc/rsyslog.d/*.conf ############### #### RULES #### ############### #Encrypted TCP log to database on Prajna *.* @@87.194.213.229:10514 # *.* -/var/log/syslog ====== End Client ========= > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour >> Sent: Thursday, November 05, 2009 5:41 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls >> >> Rainer Gerhards wrote: >>> Can you send me your rsyslog.conf, so that I can run it under the >> memory >>> debugger in my lab. I'll also take this as a motivation to finally >> add >>> multi-daemon tests to the testbench (what may take me a little >> while...). >> >> This is the server config (some of the remarks are misleading). >> >> # /etc/rsyslog.conf Configuration file for rsyslog v3. >> # >> # For more information see >> # /usr/share/doc/rsyslog- >> doc/html/rsyslog_conf.html >> >> # $DebugPrintTemplateList on >> # $ActionFileDefaultTemplate mysql-template >> ################# >> #### MODULES #### >> ################# >> >> $ModLoad imuxsock # provides support for local system logging >> >> # $ModLoad ommysql.so >> >> # provides UDP syslog reception >> $ModLoad imudp >> $UDPServerRun 514 >> $ModLoad imklog # provides kernel logging support (previously done by >> rklogd) >> >> # provides TCP syslog reception >> $ModLoad imtcp >> >> # make gtls driver the default >> # $DefaultNetstreamDriver gtls >> $DefaultNetstreamDriver ptcp >> >> # certificate files >> $DefaultNetstreamDriverCAFile /etc/rsyslog.d/.ssl/gnu-ca-cert.pem >> $DefaultNetstreamDriverCertFile /etc/rsyslog.d/.ssl/saraha-rsyslog- >> cert.pem >> $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/.ssl/saraha-rsyslog- >> key.pem >> >> # $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode >> # $InputTCPServerStreamDriverAuthMode anon # client is NOT >> authenticated >> $InputTCPServerRun 10514 # start up listener at port 10514 >> >> $ModLoad MySQL >> >> ########################### >> ########################### >> #### GLOBAL DIRECTIVES #### >> ########################### >> >> # >> # Use default timestamp format. >> # To enable high precision timestamps, comment out the following line. >> # >> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >> >> # >> # Set the default permissions for all log files. >> # >> $FileOwner root >> $FileGroup adm >> $FileCreateMode 0640 >> $DirCreateMode 0755 >> $WorkDirectory /var/log/rsyslog >> >> $template mysql-template, "insert into logs(host, facility, priority, >> level, tag, datetime, msg) values ('%source%', '%syslogfacility-text%', >> '%syslogpriority-text%', '%syslogseverity-text%', '%programname%', >> '%timereported:::date-mysql%', '%msg%')", sql, mysql >> >> # $template DEBUG,"Debug line with all properties:\nFROMHOST: >> '%FROMHOST%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag >> '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', >> PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', >> STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nrawmsg: >> '%rawmsg%'\n\n" >> >> ############### >> #### RULES #### >> ############### >> #Discard some dross messages >> # authpriv.info ~ >> :HOSTNAME, isequal, "last" ~ >> >> #Discard router access messages for the script on Prajna that collects >> the router logs. >> if $msg contains 'User logged in on TELNET (192.168.1.2)' then ~ >> if $msg contains 'User logged out on TELNET (192.168.1.2)' then ~ >> >> # Log everything else to mysql. >> $ActionQueueType LinkedList >> # Number of elements... >> $ActionQueueSize 100 >> # $ActionQueueFileName mysql >> # $ActionQueueMaxDiskSpace 1M >> # $ActionQueueHighWaterMark 40 >> # $ActionQueueLowWaterMark 5 >> >> >> *.* >> >127.0.0.1,syslog,syslog,syslog;mysql-template >> >> $ActionExecOnlyWhenPreviousIsSuspended on >> & ~ >> $ActionExecOnlyWhenPreviousIsSuspended off >> >> #Log local stuff ONLY to /var/log/syslog >> :HOSTNAME, isequal, "prajna" -/var/log/syslog >> >> >> -- >> Jack. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Thu Nov 5 23:22:04 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Thu, 5 Nov 2009 23:22:04 +0100 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710335F@GRFEXC.intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200911031158.02666.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA710335F@GRFEXC.intern.adiscon.com> Message-ID: <200911052322.04174.marc.schiffbauer@mightycare.de> Hello Rainer, I will give it a try and send you the feedback as soon as I am back from holiday. This will be in the beginning of december... Thanks -Marc Am Donnerstag, 5. November 2009 17:42:01 schrieb Rainer Gerhards: > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > > Sent: Tuesday, November 03, 2009 11:58 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > > Hello Rainer, > > > > is there some news about this issue? > > Unfortuantely not at the moment (I guess you see it is a busy time). Could > you give 5.3.4 a try? It would be very interesting (even for a v4-fix) to > see if the problem persists or not... > > Rainer > From rgerhards at hq.adiscon.com Fri Nov 6 11:54:50 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 6 Nov 2009 11:54:50 +0100 Subject: [rsyslog] message parser info - was: rsyslog 5.3.4 (devel) released References: <9B6E2A8877C38245BFB15CC491A11DA7103340@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710336D@GRFEXC.intern.adiscon.com> David, all, I hope I have answered all questions in this new document: http://www.rsyslog.com/doc-messageparser.html I would also appreciate feedback on that part: ====== Note that it is currently under evaluation if rsyslog will support binding parser chains to specific inputs directly, without depending on the ruleset. There are some concerns that this may not be necessary but adds considerable complexity to the configuration. So this may or may not be possible in the future. In any case, if we decide to add it, input modules need to support it, so this functionality would require some time to implement. ====== If I implement this, a listener would probably have a parser chain assigned, it not, the parser chain from the ruleset is used, if not, the parser chain from the default ruleset is used. I it (fairly...) trivial to add this capability, but I am really concerned that the config options get more and more complex... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, November 05, 2009 8:53 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 5.3.4 (devel) released > > yOn Thu, 5 Nov 2009, Tom Bergfeld wrote: > > > We have just released rsyslog 5.3.4, a member of the v5-devel branch. > > Today's release offers a number of important improvements. I would > like to > > highlight the ability to nest rulesets [1] (I already announced that) > and the > > ability to define custom parsers [2]. > > > > Let me elaborate a bit on the later. In the syslog world exists a > myriad of > > incompatible and invalidly formatted messages. We have had numerous > support > > requests on how to parse this or that malformed message format. With > custom > > parsers, we now have a vehicel to integrate all these invalid formats > in an > > elegant way that is also offering high performance ([2] has a > concrete > > sample). The core idea now is that parsers are plugins just like > input and > > output modules. It is relatively easy to write such a parser (roughly > a day, > > I expect) and custom parsers can be integrated into rsyslogd in a way > that > > they only affect messages received on specific port. So far, I have > not > > actually created a custom parser, but I hope that over time many of > them will > > become available. My hope here is that we actually can build a > directory, > > which other folks can browse so that they are able to find a solution > to the > > mess their vendor has provided ;) > > this sounds very interesting. however, reading the link I'm a little > confused. > > on the one hand it talks a lot about the priority of parsers, but then > it > talks about binding different parsers to different ports. It's not > clear > if these are two different ways to use parsers, or how these two would > interact. > > where can these parsers be used? > > obviously the imudp module can use them. can all input modules use > them? > > what are the limits of the parser? > > a couple of extreme examples: > > could you implement relp as a parser for imtcp, or since relp sends > replies that can't be done (limiting it to different ways of processing > a > message that's already been received) > > could a parser detect that it had a piece of a multi-line messsage and > stash the piece until it receives the rest of the message so that it > could > submit the complete message? > > > a coouple questions from trying to look through the code at almost 3am > local time > > if it can easily be done, may I suggest changing the santization flag > from > a yes/no boolean to an enum so that there can be more than one > sanitation > routine > > do you have the default parser broken out as a seperate file tha canbe > used as an example? > > David Lang > > > This release also introduces the capability to run rulesets off their > own > > queue (also already mentioned on the mailing list). Plus, it contains > the > > usual set of bug fixes. > > > > We plan to make this version the basis for the next v5-stable. > > > > > > [1] http://www.rsyslog.com/doc-omruleset.html > > [2] http://www.rsyslog.com/doc-rsconf1_rulesetparser.html > > > > > > > > ChangeLog: > > > > http://www.rsyslog.com/Article423.phtml > > > > Download: > > > > http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid- > 185.phtml > > > > As always, feedback is appreciated. > > > > Best regards, > > Tom Bergfeld > > -- > > Support > > ======= > > > > Improving rsyslog is costly, but you can help! We are looking for > > organizations that find rsyslog useful and wish to contribute back. > You can > > contribute by reporting bugs, improve the software, or donate money > or > > equipment. > > > > Commercial support contracts for rsyslog are available, and they help > finance > > continued maintenance. Adiscon GmbH, a privately held German > company, is > > currently funding rsyslog development. We are always looking for > interesting > > development projects. For details on how to help, please see > > http://www.rsyslog.com/doc-how2help.html . > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From correo at miguelangelnieto.net Fri Nov 6 12:31:04 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Fri, 6 Nov 2009 12:31:04 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: I made rpm packages with the stable v3 of rsyslog for suse 10.3. If anyone need them, tell me. > in other cases you are willing to loose logs rather than freezing the > machine and can configure rsyslog to accept messages, even when it can't > do anything with them to avoid this sort of lockup. How can I do to tell rsyslog to accept all messages and not use any queue? -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From rgerhards at hq.adiscon.com Fri Nov 6 14:27:20 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 6 Nov 2009 14:27:20 +0100 Subject: [rsyslog] potential server problems on rsyslog.com Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103373@GRFEXC.intern.adiscon.com> Hi all, I was just told that http://www.rsyslog.com, associated services and some other sites (like the knowledge base) will probably experience some temporary outage within the next couple of hours. Looks like the sysadmins need to fix something urgently. So if you got a connection problem, please retry a little bit later. Thanks, Rainer From jbondc at openmv.com Fri Nov 6 16:18:55 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Fri, 6 Nov 2009 10:18:55 -0500 Subject: [rsyslog] Postgresql module: standard_conforming_strings Message-ID: <002801ca5ef4$74e7cd20$5eb76760$@com> I have two questions: 1) How does rsyslog espace SQL commands in the plugin ompgsql.c? I've been staring at the code but I can't figure out where the backslash escaping happens. My problem is I've set: standard_conforming_strings=On This means that the backslash espace ' gfgfdg \' ' is ignored and causes errors. There are many ways to fix this. a) Have rsyslog issue (SET standard_conforming_strings = off;) for postgresql 8.2+ (quick fix) b) Change default sql espacing to use doubles quotes '' --- so 'test \' ' becomes 'test '' ' http://www.postgresql.org/docs/8.1/static/libpq-exec.html#LIBPQ-EXEC-ESCAPE- STRING 2) Is there an SVN repository of rsyslog? Or does anyone know of a good way or using GIT on windows (turtoisegit is currently buggy) ? From rgerhards at hq.adiscon.com Fri Nov 6 16:45:33 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 6 Nov 2009 16:45:33 +0100 Subject: [rsyslog] Postgresql module: standard_conforming_strings References: <002801ca5ef4$74e7cd20$5eb76760$@com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103377@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Jonathan Bond-Caron > Sent: Friday, November 06, 2009 4:19 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Postgresql module: standard_conforming_strings > > I have two questions: > > > > 1) How does rsyslog espace SQL commands in the plugin ompgsql.c? > > > > I've been staring at the code but I can't figure out where the > backslash > escaping happens. > > > > My problem is I've set: standard_conforming_strings=On > > This means that the backslash espace ' gfgfdg \' ' is ignored and > causes > errors. There are many ways to fix this. > > > > a) Have rsyslog issue (SET standard_conforming_strings = off;) for > postgresql 8.2+ (quick fix) > > b) Change default sql espacing to use doubles quotes '' --- so > 'test \' > ' becomes 'test '' ' > > http://www.postgresql.org/docs/8.1/static/libpq-exec.html#LIBPQ-EXEC- > ESCAPE- > STRING > This is controlled via the template. For postgre SQL, I think it needs the STDSQL option. I will check if ompgsql requests that option. $template name,"insert ...",STDSQL The string must be sanitized before it is passed down to the module, because the module does not know any longer what a field is. > > > 2) Is there an SVN repository of rsyslog? No > Or does anyone know of a > good > way or using GIT on windows (turtoisegit is currently buggy) ? > My co-workers use http://code.google.com/p/msysgit/ and don't complain (too much ;)) about it. HTH Rainer > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From epiphani at gmail.com Fri Nov 6 19:18:37 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Fri, 6 Nov 2009 13:18:37 -0500 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103338@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103338@GRFEXC.intern.adiscon.com> Message-ID: Wanted to point out that I've observed this now in a second environment - so we can eliminate the possibility that this is specific to that situation. I'll be working to reproduce the issue in a separate system next week, and hopefully I can get something reproducable so I can test the updated version. Thanks! -Aaron On Wed, Nov 4, 2009 at 10:06 AM, Rainer Gerhards wrote: > >> that can work, however XML output may not be too bad, people entering >> commands manually are probably not going to be asking for that much >> data. > > If I look at time constraints, I will probably not able to define a big CLI > with a real language. So I think you might send something like "queuestatus" > and it will reply with all status information on all queues it has. > >> the biggest problem I see with XML is the need to send requests and >> responses via e-mail for debugging. if one end uses a HTML enabled e- >> mail >> client it may not work well to paste XML text into it. > > That's a very good point and I really should begin to think about this right > now. Some kind of "debug package" that's easily mailable would not be a bad > thing ;) > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Nov 6 19:36:03 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 6 Nov 2009 19:36:03 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103338@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710337D@GRFEXC.intern.adiscon.com> excellent news. I have reviewed the debug log handling. It looks like we can use the current version, mabe just slightly modified, to generate debug output when a problem occurs by sending SIGUSR1 to it. I hope to be able to complete testing on monday. If it works out like I expect, I'll do a write-up of how it can be done. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Friday, November 06, 2009 7:19 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) / Problem in production > > Wanted to point out that I've observed this now in a second > environment - so we can eliminate the possibility that this is > specific to that situation. > > I'll be working to reproduce the issue in a separate system next week, > and hopefully I can get something reproducable so I can test the > updated version. > > Thanks! > -Aaron > > On Wed, Nov 4, 2009 at 10:06 AM, Rainer Gerhards > wrote: > > > >> that can work, however XML output may not be too bad, people > entering > >> commands manually are probably not going to be asking for that much > >> data. > > > > If I look at time constraints, I will probably not able to define a > big CLI > > with a real language. So I think you might send something like > "queuestatus" > > and it will reply with all status information on all queues it has. > > > >> the biggest problem I see with XML is the need to send requests and > >> responses via e-mail for debugging. if one end uses a HTML enabled > e- > >> mail > >> client it may not work well to paste XML text into it. > > > > That's a very good point and I really should begin to think about > this right > > now. Some kind of "debug package" that's easily mailable would not be > a bad > > thing ;) > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Nov 6 19:47:41 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 6 Nov 2009 10:47:41 -0800 (PST) Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: On Fri, 6 Nov 2009, Miguel Angel Nieto wrote: >> in other cases you are willing to loose logs rather than freezing the >> machine and can configure rsyslog to accept messages, even when it can't >> do anything with them to avoid this sort of lockup. > > How can I do to tell rsyslog to accept all messages and not use any queue? you cannot tell rsyslog to not use a queue. the fundamental architecture of rsyslog is that the input modules receive messages, parse minimal info out of them, and put them in a queue.worker threads then pull messages from this queue and send them to their destination. this queue can be in ram, ram + disk file, or disk-only classic syslog doesn't use any queue and must fully process each message before receiving the next message. syslog-ng has the ability to delay writing the logs to disk a bit to do them in batches, but otherwise is the same. in rsyslog, significantly more time is spent in the output side of things than in the input side. historicly, syslog was intended to be a reliable logging mechanism, so applications do not complete their write until the syslog daemon has the log safe on disk. the performance of doing this is so horrible that everybody has some way of relaxing this requirement (the - option in sysklogd, the batch option in syslog-ng, the memory queue in rsyslog), but all of these options only allow a finite number of items to be buffered before the syslog daemon will stop to wait for them to really get to the destination. this is why your systems locked up, they were trying to write to rsyslog on the client, rsyslog on the client was configured to not consider the logs processed until they were acknowledged by the TCP stack of the server. so when the server is not available the clients keep accepting messages and queue them up, but when the queue filled up they stopped accepting new messages. the reason to use TCP for syslog instead of UDP is so that you can have the log server push back on the sender and tell the sender that it can't process the log message right now, the sender should hang on to it and tray again to send it later. if you are really willing to loose logs if the log server is down or backed up, why not just use UDP for the logs? look at the MainMsgQueue* options for setting high watermark, low watermark, discard options, etc if you want to change how rsyslog acts when the queue fills up. I haven't used these, so all I can do is to point you in the right direction. David Lang From mrdemeanour at jackpot.uk.net Fri Nov 6 20:10:15 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 06 Nov 2009 19:10:15 +0000 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: <4AF47497.4070407@jackpot.uk.net> david at lang.hm wrote: > On Fri, 6 Nov 2009, Miguel Angel Nieto wrote: > >>> in other cases you are willing to loose logs rather than freezing >>> the machine and can configure rsyslog to accept messages, even >>> when it can't do anything with them to avoid this sort of lockup. >>> >> How can I do to tell rsyslog to accept all messages and not use any >> queue? > > you cannot tell rsyslog to not use a queue. Is that really true? "Direct queues are non-queuing queues. A queue in direct mode does neither queue nor buffer any of the queue elements but rather passes the element directly (and immediately) from the producer to the consumer. This sounds strange, but there is a good reason for this queue type." [...] "To create a direct queue, use the "$QueueType Direct" config directive." e.g. (I suppose): $MainQueueType Direct http://www.rsyslog.com/doc-queues.html -- Jack. From ryan.b.lynch at gmail.com Fri Nov 6 22:02:40 2009 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Fri, 6 Nov 2009 16:02:40 -0500 Subject: [rsyslog] What backslash-escape sequences are supported in templates? Message-ID: <115906d10911061302h56d1b56cs50247fd010de57a5@mail.gmail.com> The man page gives '\7' (bell) as an example (in the section 'Templates'), and then it says "The set in rsyslog is a bit restricted currently." I couldn't find any more info in the online documentation, in either of the 'Templates' or 'Property Replacer' sections. I just tried to use '\t' (tab) in a log format template, though, and it didn't work--it prints as just 't'. Is there a doc that lists what escapes are and aren't accepted? If not, can somebody who knows the answer enlighten me? Ryan B. Lynch ryan.b.lynch at gmail.com From philr at jaspers.co.nz Sat Nov 7 21:37:59 2009 From: philr at jaspers.co.nz (Phil Reilly) Date: Sun, 08 Nov 2009 09:37:59 +1300 Subject: [rsyslog] Alerting rules via a database Message-ID: <4AF5DAA7.5010001@jaspers.co.nz> I attempting to allow for flexible rule matches on Syslogs from a web front end (rather than entires into the rsyslog config files) I want to get regexp filters from a db to alert upon messages. Not sure the best way to achieve this. I've so far though of. * Outputting to a pipe and runing it via an alerting script. * Having file watch the messages. * Recieving the messages then passing them to rsyslog (yuck) Can the rule engine allow for match rules outside of the config? is there an elegant way of doing this? From david at lang.hm Sun Nov 8 08:44:36 2009 From: david at lang.hm (david at lang.hm) Date: Sat, 7 Nov 2009 23:44:36 -0800 (PST) Subject: [rsyslog] Alerting rules via a database In-Reply-To: <4AF5DAA7.5010001@jaspers.co.nz> References: <4AF5DAA7.5010001@jaspers.co.nz> Message-ID: On Sun, 8 Nov 2009, Phil Reilly wrote: > I attempting to allow for flexible rule matches on Syslogs from a web > front end (rather than entires into the rsyslog config files) > > I want to get regexp filters from a db to alert upon messages. Not sure > the best way to achieve this. I've so far though of. > > * Outputting to a pipe and runing it via an alerting script. > * Having file watch the messages. > * Recieving the messages then passing them to rsyslog (yuck) > > Can the rule engine allow for match rules outside of the config? is > there an elegant way of doing this? rsyslog doesn't give you this ability, but it's not really the best approach to use for alerting anyway. what are you trying to achieve by having the alert definitions in a database? there are several tools out there to do alerting (SEC, Simple Event Correlator) is one of the leading ones, but I'm not aware of any of them that use a database for their rulesets. I'm also scratching my head trying to figure out what the advantage of doing so would be. David Lang From philr at jaspers.co.nz Sun Nov 8 09:29:30 2009 From: philr at jaspers.co.nz (Phil Reilly) Date: Sun, 08 Nov 2009 21:29:30 +1300 Subject: [rsyslog] Alerting rules via a database In-Reply-To: References: <4AF5DAA7.5010001@jaspers.co.nz> Message-ID: <4AF6816A.6050901@jaspers.co.nz> david at lang.hm wrote: > On Sun, 8 Nov 2009, Phil Reilly wrote: > > >> I attempting to allow for flexible rule matches on Syslogs from a web >> front end (rather than entires into the rsyslog config files) >> >> I want to get regexp filters from a db to alert upon messages. Not sure >> the best way to achieve this. I've so far though of. >> >> * Outputting to a pipe and runing it via an alerting script. >> * Having file watch the messages. >> * Recieving the messages then passing them to rsyslog (yuck) >> >> Can the rule engine allow for match rules outside of the config? is >> there an elegant way of doing this? >> > > rsyslog doesn't give you this ability, but it's not really the best > approach to use for alerting anyway. > > what are you trying to achieve by having the alert definitions in a > database? there are several tools out there to do alerting (SEC, Simple > Event Correlator) is one of the leading ones, but I'm not aware of any of > them that use a database for their rulesets. > > I'm also scratching my head trying to figure out what the advantage of > doing so would be. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > Thanks David, We have a networked environment. We also have a web page that allows you to configure regexp to match certain syslog messages. These patterns are compiled and kept in a table. The current syslog process we use listens for udp. When it gets a syslog message, we examine the patterns (which are re-read upon addition or change) and pass them to an alertering process before writing the logs to disk. The existing system works well, but we now want to scale it over a few machines and I'm examining what syslog products out there cater for alerting. So a database will make configuring alerts far more dynamic than statically entering them into a config file. It will also allow for grouped views so different groups have the ability to have custom alerts based upon their own interpretation of syslog messages. From rgerhards at hq.adiscon.com Sun Nov 8 11:48:19 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 8 Nov 2009 11:48:19 +0100 Subject: [rsyslog] Alerting rules via a database References: <4AF5DAA7.5010001@jaspers.co.nz> <4AF6816A.6050901@jaspers.co.nz> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> So what you are actually looking for is a system that can work with dynamically changable alert definitions? As David said, there is no such thing currently, but the best road to approach is is to write a custom output plugin, that you pass each message to. That plugin can even decide if messages should be discarded and not further processed. I envisioned such a plugin, but had not yet time to write, for a similar use case. If you intend to write one AND contribute it to the project, I can help you get started with the interface, would even be willing to create you a custom skeleton that you can fill in your logic ;) HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phil Reilly > Sent: Sunday, November 08, 2009 9:30 AM > To: rsyslog-users > Subject: Re: [rsyslog] Alerting rules via a database > > david at lang.hm wrote: > > On Sun, 8 Nov 2009, Phil Reilly wrote: > > > > > >> I attempting to allow for flexible rule matches on Syslogs > from a web > >> front end (rather than entires into the rsyslog config files) > >> > >> I want to get regexp filters from a db to alert upon > messages. Not sure > >> the best way to achieve this. I've so far though of. > >> > >> * Outputting to a pipe and runing it via an alerting script. > >> * Having file watch the messages. > >> * Recieving the messages then passing them to rsyslog (yuck) > >> > >> Can the rule engine allow for match rules outside of the config? is > >> there an elegant way of doing this? > >> > > > > rsyslog doesn't give you this ability, but it's not really the best > > approach to use for alerting anyway. > > > > what are you trying to achieve by having the alert definitions in a > > database? there are several tools out there to do alerting > (SEC, Simple > > Event Correlator) is one of the leading ones, but I'm not > aware of any of > > them that use a database for their rulesets. > > > > I'm also scratching my head trying to figure out what the > advantage of > > doing so would be. > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > Thanks David, > > We have a networked environment. We also have a web page that > allows you > to configure regexp to match certain syslog messages. These > patterns are > compiled and kept in a table. The current syslog process we > use listens > for udp. When it gets a syslog message, we examine the > patterns (which > are re-read upon addition or change) and pass them to an alertering > process before writing the logs to disk. The existing system > works well, > but we now want to scale it over a few machines and I'm > examining what > syslog products out there cater for alerting. > > So a database will make configuring alerts far more dynamic than > statically entering them into a config file. It will also allow for > grouped views so different groups have the ability to have > custom alerts > based upon their own interpretation of syslog messages. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Sun Nov 8 16:40:50 2009 From: david at lang.hm (david at lang.hm) Date: Sun, 8 Nov 2009 07:40:50 -0800 (PST) Subject: [rsyslog] Alerting rules via a database In-Reply-To: <4AF6816A.6050901@jaspers.co.nz> References: <4AF5DAA7.5010001@jaspers.co.nz> <4AF6816A.6050901@jaspers.co.nz> Message-ID: On Sun, 8 Nov 2009, Phil Reilly wrote: > > david at lang.hm wrote: >> On Sun, 8 Nov 2009, Phil Reilly wrote: >> >> >>> I attempting to allow for flexible rule matches on Syslogs from a web >>> front end (rather than entires into the rsyslog config files) >>> >>> I want to get regexp filters from a db to alert upon messages. Not sure >>> the best way to achieve this. I've so far though of. >>> >>> * Outputting to a pipe and runing it via an alerting script. >>> * Having file watch the messages. >>> * Recieving the messages then passing them to rsyslog (yuck) >>> >>> Can the rule engine allow for match rules outside of the config? is >>> there an elegant way of doing this? >>> >> >> rsyslog doesn't give you this ability, but it's not really the best >> approach to use for alerting anyway. >> >> what are you trying to achieve by having the alert definitions in a >> database? there are several tools out there to do alerting (SEC, Simple >> Event Correlator) is one of the leading ones, but I'm not aware of any of >> them that use a database for their rulesets. >> >> I'm also scratching my head trying to figure out what the advantage of >> doing so would be. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > Thanks David, > > We have a networked environment. We also have a web page that allows you > to configure regexp to match certain syslog messages. These patterns are > compiled and kept in a table. The current syslog process we use listens > for udp. When it gets a syslog message, we examine the patterns (which > are re-read upon addition or change) and pass them to an alertering > process before writing the logs to disk. The existing system works well, > but we now want to scale it over a few machines and I'm examining what > syslog products out there cater for alerting. > > So a database will make configuring alerts far more dynamic than > statically entering them into a config file. It will also allow for > grouped views so different groups have the ability to have custom alerts > based upon their own interpretation of syslog messages. I don't know anything that will read a database like you are lookng for. I think you would be better off having your web gui create SEC rules or something like that (you can still store the basic info in a database) David Lang From jbondc at openmv.com Sun Nov 8 17:42:34 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Sun, 8 Nov 2009 11:42:34 -0500 Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing Message-ID: <003001ca6092$792833d0$6b789b70$@com> Hi all, Attached is a patch for 3 things: a) Check that TCP connection is still alive when using TLS b) Improve TAG parsing so that it ends when it sees a tab or escape control character. c) Added configuration directive: escapecontrolcharactertab In rsyslog.conf, you can set: $escapeControlCharactersOnReceive on $escapeControlCharacterTab off This avoids having a tab being escaped since it is after a viewable character J I'd prefer this being the default behavior but I've left $escapeControlCharacterTab on as default in case people expect tabs to be escaped. This is my first GIT patch, hope it works well ;) From mrdemeanour at jackpot.uk.net Sun Nov 8 18:23:31 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Sun, 08 Nov 2009 17:23:31 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> Message-ID: <4AF6FE93.6080805@jackpot.uk.net> Rainer Gerhards wrote: > Thanks, but I wasn't specific enough. For TLS, I also need to client > config, because I need two machines to reproduce any issues (these > two instances are also the challenge for the current testbench, what > requires hopefully fewer than I expect changes ;)). Perhaps someone could advise me on an alternative to resolving this problem by means of bug-hunting. While 4.4.2 is the version of rsyslog currently shipping in Debian squeeze, it's obviously a bit out-of-date from the point of view of the developers. Well, the reason I'm using that version is precisely that it ships with Debian. Or to be more precise, I upgraded to squeeze because it ships with a version of rsyslog that includes encryption support. So perhaps I should just cut free, and install 5.x stable? -- Jack. From jbondc at openmv.com Sun Nov 8 20:18:43 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Sun, 8 Nov 2009 14:18:43 -0500 Subject: [rsyslog] Postgresql module: standard_conforming_strings In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103377@GRFEXC.intern.adiscon.com> References: <002801ca5ef4$74e7cd20$5eb76760$@com> <9B6E2A8877C38245BFB15CC491A11DA7103377@GRFEXC.intern.adiscon.com> Message-ID: <000601ca60a8$49f44d40$dddce7c0$@com> On Fri Nov 6 10:45 AM, Rainer Gerhards wrote: > > > > > > My problem is I've set: standard_conforming_strings=On > > > > This means that the backslash espace ' gfgfdg \' ' is ignored and > > causes errors. There are many ways to fix this. > > > > This is controlled via the template. For postgre SQL, I think it needs > the STDSQL option. I will check if ompgsql requests that option. > > $template name,"insert ...",STDSQL > Thanks that fixed it, I was using $template name,"insert ...",SQL From philr at jaspers.co.nz Sun Nov 8 20:36:07 2009 From: philr at jaspers.co.nz (Phil Reilly) Date: Mon, 09 Nov 2009 08:36:07 +1300 Subject: [rsyslog] Alerting rules via a database In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> References: <4AF5DAA7.5010001@jaspers.co.nz> <4AF6816A.6050901@jaspers.co.nz> <9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> Message-ID: <4AF71DA7.2090107@jaspers.co.nz> More than happy to Rainer. A Skeleton will be welcome. Rainer Gerhards wrote: > So what you are actually looking for is a system that can work with > dynamically changable alert definitions? As David said, there is no such > thing currently, but the best road to approach is is to write a custom output > plugin, that you pass each message to. That plugin can even decide if > messages should be discarded and not further processed. I envisioned such a > plugin, but had not yet time to write, for a similar use case. > > If you intend to write one AND contribute it to the project, I can help you > get started with the interface, would even be willing to create you a custom > skeleton that you can fill in your logic ;) > > HTH > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phil Reilly >> Sent: Sunday, November 08, 2009 9:30 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Alerting rules via a database >> >> david at lang.hm wrote: >> >>> On Sun, 8 Nov 2009, Phil Reilly wrote: >>> >>> >>> >>>> I attempting to allow for flexible rule matches on Syslogs >>>> >> from a web >> >>>> front end (rather than entires into the rsyslog config files) >>>> >>>> I want to get regexp filters from a db to alert upon >>>> >> messages. Not sure >> >>>> the best way to achieve this. I've so far though of. >>>> >>>> * Outputting to a pipe and runing it via an alerting script. >>>> * Having file watch the messages. >>>> * Recieving the messages then passing them to rsyslog (yuck) >>>> >>>> Can the rule engine allow for match rules outside of the config? is >>>> there an elegant way of doing this? >>>> >>>> >>> rsyslog doesn't give you this ability, but it's not really the best >>> approach to use for alerting anyway. >>> >>> what are you trying to achieve by having the alert definitions in a >>> database? there are several tools out there to do alerting >>> >> (SEC, Simple >> >>> Event Correlator) is one of the leading ones, but I'm not >>> >> aware of any of >> >>> them that use a database for their rulesets. >>> >>> I'm also scratching my head trying to figure out what the >>> >> advantage of >> >>> doing so would be. >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> Thanks David, >> >> We have a networked environment. We also have a web page that >> allows you >> to configure regexp to match certain syslog messages. These >> patterns are >> compiled and kept in a table. The current syslog process we >> use listens >> for udp. When it gets a syslog message, we examine the >> patterns (which >> are re-read upon addition or change) and pass them to an alertering >> process before writing the logs to disk. The existing system >> works well, >> but we now want to scale it over a few machines and I'm >> examining what >> syslog products out there cater for alerting. >> >> So a database will make configuring alerts far more dynamic than >> statically entering them into a config file. It will also allow for >> grouped views so different groups have the ability to have >> custom alerts >> based upon their own interpretation of syslog messages. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Mon Nov 9 05:48:27 2009 From: david at lang.hm (david at lang.hm) Date: Sun, 8 Nov 2009 20:48:27 -0800 (PST) Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <4AF6FE93.6080805@jackpot.uk.net> References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net> Message-ID: On Sun, 8 Nov 2009, Mr. Demeanour wrote: > Rainer Gerhards wrote: >> Thanks, but I wasn't specific enough. For TLS, I also need to client >> config, because I need two machines to reproduce any issues (these >> two instances are also the challenge for the current testbench, what >> requires hopefully fewer than I expect changes ;)). > > Perhaps someone could advise me on an alternative to resolving this > problem by means of bug-hunting. While 4.4.2 is the version of rsyslog > currently shipping in Debian squeeze, it's obviously a bit out-of-date > from the point of view of the developers. the rate of change in rsyslog over the last year is dizzying. This makes rsyslog a much better progam, but it also means that it's very unlikly that the debian maintainers will be able to backport all the fixes. > Well, the reason I'm using that version is precisely that it ships with > Debian. Or to be more precise, I upgraded to squeeze because it ships > with a version of rsyslog that includes encryption support. So perhaps I > should just cut free, and install 5.x stable? unfortunantly the current 5.2 stable release has some known problems. the 5.x development is looking pretty good right now, but it is very much the bleeding edge of things. if you need a newer stable version _now_ go with the latest 4.x if you can affort to test the 5.x development branch, it will probably work (I've found it stable over the last couple of weeks, running at loads of ~10,000 logs/sec), but as always you may run into unexpected problems (like you did with 4.4.2). I don't use encryption so I can't vouch for that part of things. David Lang From rgerhards at hq.adiscon.com Mon Nov 9 06:38:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 9 Nov 2009 06:38:29 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls Message-ID: <000501ca60fe$fac9ae3a$100013ac@intern.adiscon.com> More info follows, but v5 has exactly the same tls subsystem as v4. So we need to track that bug down. ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 09.11.09 05:49 Betreff: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls On Sun, 8 Nov 2009, Mr. Demeanour wrote: > Rainer Gerhards wrote: >> Thanks, but I wasn't specific enough. For TLS, I also need to client >> config, because I need two machines to reproduce any issues (these >> two instances are also the challenge for the current testbench, what >> requires hopefully fewer than I expect changes ;)). > > Perhaps someone could advise me on an alternative to resolving this > problem by means of bug-hunting. While 4.4.2 is the version of rsyslog > currently shipping in Debian squeeze, it's obviously a bit out-of-date > from the point of view of the developers. the rate of change in rsyslog over the last year is dizzying. This makes rsyslog a much better progam, but it also means that it's very unlikly that the debian maintainers will be able to backport all the fixes. > Well, the reason I'm using that version is precisely that it ships with > Debian. Or to be more precise, I upgraded to squeeze because it ships > with a version of rsyslog that includes encryption support. So perhaps I > should just cut free, and install 5.x stable? unfortunantly the current 5.2 stable release has some known problems. the 5.x development is looking pretty good right now, but it is very much the bleeding edge of things. if you need a newer stable version _now_ go with the latest 4.x if you can affort to test the 5.x development branch, it will probably work (I've found it stable over the last couple of weeks, running at loads of ~10,000 logs/sec), but as always you may run into unexpected problems (like you did with 4.4.2). I don't use encryption so I can't vouch for that part of things. David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Nov 9 10:02:40 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 9 Nov 2009 10:02:40 +0100 Subject: [rsyslog] rsyslog priorities - was: RE: Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710338D@GRFEXC.intern.adiscon.com> >From my POV the real issue is that there is very much going on at the moment (I don't want to complain about that). But that also means I need to prioritize the things I do. While I would like to have bugfixing always on top of the list, it is plain truth that there are also some other factors (like how to obtain bread for breakfast ;)), so priorities are also dictated by commercial activities that need to run alongside. As a reminder, rsyslog development is still very largely depending on Adiscon funding and that also means things like EventReporter actually pay for it (some people may think well about that fact and its implications ;)). If I worked on rsyslog just as a byline hobby activity, we were nowhere close to where we are right now. But, as I said: the bottom line is that priorities come with a twist of "as time permits". Anyhow, I am quite concerned about this bug and hope to be able to look at it as soon as possible. As another side-node running development versions and providing good bug reports is also an excellent way to contribute to the project... If you followed the mailing list, my priority scheme also slightly involves that bug reports/feature whishes from those that contribute frequently have a somewhat higher priority than from those who do not. I find this is only fair. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Sunday, November 08, 2009 6:24 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Rainer Gerhards wrote: > > Thanks, but I wasn't specific enough. For TLS, I also need to client > > config, because I need two machines to reproduce any issues (these > > two instances are also the challenge for the current testbench, what > > requires hopefully fewer than I expect changes ;)). > > Perhaps someone could advise me on an alternative to resolving this > problem by means of bug-hunting. While 4.4.2 is the version of rsyslog > currently shipping in Debian squeeze, it's obviously a bit out-of-date > from the point of view of the developers. > > Well, the reason I'm using that version is precisely that it ships with > Debian. Or to be more precise, I upgraded to squeeze because it ships > with a version of rsyslog that includes encryption support. So perhaps > I > should just cut free, and install 5.x stable? > > -- > Jack. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mrdemeanour at jackpot.uk.net Mon Nov 9 10:13:56 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Mon, 09 Nov 2009 09:13:56 +0000 Subject: [rsyslog] rsyslog priorities - was: RE: Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710338D@GRFEXC.intern.adiscon.com> References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA710338D@GRFEXC.intern.adiscon.com> Message-ID: <4AF7DD54.6080001@jackpot.uk.net> Rainer Gerhards wrote: > While I would like to have bugfixing always on top of the list, it is > plain truth that there are also some other factors (like how to > obtain bread for breakfast ;)), so priorities are also dictated by > commercial activities that need to run alongside. I wouldn't dream of questioning your priorities. I don't know how you manage to do so much development, while at the same time attending to this list, participating in general syslog standards-related actvities and so on. Frankly, I'm rather surprised to learn that you actually have time for breakfast! I think I shall try 5.2 on the server; I'll let you know if the trouble recurs. -- Jack. From martinmie at PartyGaming.com Mon Nov 9 10:15:13 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Mon, 9 Nov 2009 10:15:13 +0100 Subject: [rsyslog] rsyslog 5.2.0 is a zombie Message-ID: <0B1B9163138571439EAF804646F3F6AA08A58AA7@GIBSVWIN004X.partygaming.local> Hi, Just for the records... uur rsyslog servers, both running on the stable 5.2.0, died over the weekend. Well, to be more precise, they became zombies: # ps -efl | grep rsyslog 5 Z root 2565 1 73 77 0 - 0 exit Nov05 ? 2-20:44:14 [rsyslogd] ...so we sent from an "uninterruptible sleep" on 4.2.0 to a more "eternal sleep" LOL. I think it's time to revert to the latest v4 branch... suggestions? Cheers, Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From rgerhards at hq.adiscon.com Mon Nov 9 11:35:26 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 9 Nov 2009 11:35:26 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory References: <4AF47497.4070407@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103392@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Friday, November 06, 2009 8:10 PM > To: rsyslog-users > Subject: Re: [rsyslog] computer hang-up and WorkDirectory > > david at lang.hm wrote: > > On Fri, 6 Nov 2009, Miguel Angel Nieto wrote: > > > >>> in other cases you are willing to loose logs rather than freezing > >>> the machine and can configure rsyslog to accept messages, even > >>> when it can't do anything with them to avoid this sort of lockup. > >>> > >> How can I do to tell rsyslog to accept all messages and not use any > >> queue? > > > > you cannot tell rsyslog to not use a queue. > > Is that really true? > > "Direct queues are non-queuing queues. A queue in direct mode does > neither queue nor buffer any of the queue elements but rather passes > the > element directly (and immediately) from the producer to the consumer. > This sounds strange, but there is a good reason for this queue type." > > [...] > > "To create a direct queue, use the "$QueueType Direct" config > directive." > > e.g. (I suppose): > $MainQueueType Direct > > > http://www.rsyslog.com/doc-queues.html Sorry, I overlooked that message first. You are right. While rsyslog needs queues (more precisely: queue objects), one can configure queue objects not to queue. This is the default case for actions. However, you most often do not want to set the main queue to direct. But you can and there are use cases for it. The bottom line than is that sender and action processing are tightly couple, so with inputs like UDP you will most probably lose a lot of messages, especially when using slow backends like databases. One cure is to run those slow backends then on async action queues. HTH Rainer From rgerhards at hq.adiscon.com Mon Nov 9 12:24:04 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 9 Nov 2009 12:24:04 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103395@GRFEXC.intern.adiscon.com> Jack, Quick update: I was able to run a (relatively quick) test on a configuration that hopefully later becomes part of the testbench (I am working towards that goal and thought doing a manual check at that stage would not hurt). It is not based your config, and it is relatively simple and straightforward. Still, it uses TLS, and uses it in anon mode like you did. I used 4.4.2 on the server. I processed around 500,000 message (not kept track of the actual number). I did three such runs, all runing under the excellent valgrind[1] memory debugger. For none of the runs, valgrind reported any memory leaks. While this may not be an ultimate indication, valgrind is *very* effective in finding leaks and based on what you wrote I would have expected a small chunk of memory to be lost per message. So the bottom line is that I currently cannot reproduce the bug. This may change when I finally import your config. However, it would be useful for me if you could run valgrind on rsyslogd in your environment and let me know if valgrind reports any memory leaks. Doing so considerably slows down rsyslogd, but given your load, I'd expect that it would be acceptable. To run under valgrind is relatively simply. Valgrind is available as a package on almost all distros. All you need to do is run valgrind and specify your usual rsyslogd command line as the parameter. It is recommended to do this in the foreground (see rsyslog troubleshooting doc). So, for example, if you start rsyslogd usually by /sbin/rsyslogd -c4 -... you simply do valgrind /sbin/rsyslogd -c4 -... instead. By default, valgrind message go to stdout. At the end of the run (kill `rsyslog.pid`) valgrind prints out memory leaks it detects. During the run, it prints out any anomalies it sees. Would be great if you could give that a try. Rainer [1] http://www.valgrind.org > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Sunday, November 08, 2009 6:24 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Rainer Gerhards wrote: > > Thanks, but I wasn't specific enough. For TLS, I also need to client > > config, because I need two machines to reproduce any issues (these > > two instances are also the challenge for the current testbench, what > > requires hopefully fewer than I expect changes ;)). > > Perhaps someone could advise me on an alternative to resolving this > problem by means of bug-hunting. While 4.4.2 is the version of rsyslog > currently shipping in Debian squeeze, it's obviously a bit out-of-date > from the point of view of the developers. > > Well, the reason I'm using that version is precisely that it ships with > Debian. Or to be more precise, I upgraded to squeeze because it ships > with a version of rsyslog that includes encryption support. So perhaps > I > should just cut free, and install 5.x stable? > > -- > Jack. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mrdemeanour at jackpot.uk.net Mon Nov 9 17:24:45 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Mon, 09 Nov 2009 16:24:45 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103395@GRFEXC.intern.adiscon.com> References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA7103395@GRFEXC.intern.adiscon.com> Message-ID: <4AF8424D.30306@jackpot.uk.net> Rainer Gerhards wrote: > Jack, > > Quick update: I was able to run a (relatively quick) test on a > configuration that hopefully later becomes part of the testbench (I > am working towards that goal and thought doing a manual check at that > stage would not hurt). It is not based your config, and it is > relatively simple and straightforward. Still, it uses TLS, and uses > it in anon mode like you did. I used 4.4.2 on the server. I processed > around 500,000 message (not kept track of the actual number). I did > three such runs, all runing under the excellent valgrind[1] memory > debugger. > > For none of the runs, valgrind reported any memory leaks. While this > may not be an ultimate indication, valgrind is *very* effective in > finding leaks and based on what you wrote I would have expected a > small chunk of memory to be lost per message. OK (thanks). I have formed the impression that the problems are occurring after some period of running time; free memory decreases quite slowly, until some unknown event causes a rather rapid degeneration. That is: I suspect that any leak is not likely to be observable on a per-message basis, until this unknown event has occured. > > So the bottom line is that I currently cannot reproduce the bug. This > may change when I finally import your config. However, it would be > useful for me if you could run valgrind on rsyslogd in your > environment and let me know if valgrind reports any memory leaks. > Doing so considerably slows down rsyslogd, but given your load, I'd > expect that it would be acceptable. > > To run under valgrind is relatively simply. Valgrind is available as > a package on almost all distros. All you need to do is run valgrind > and specify your usual rsyslogd command line as the parameter. It is > recommended to do this in the foreground (see rsyslog troubleshooting > doc). > > So, for example, if you start rsyslogd usually by > > /sbin/rsyslogd -c4 -... So "valgrind /usr/sbin/rsyslogd -c4" results in rsyslogd backgrounding promptly, at which point valgrind prints its report (which shows no leaks - unsurprisingly). "valgrind /usr/sbin/rsyslogd -c4 -n" results in a hang. CTRL-C fails to kill the foreground task. "kill -9 " kills the task, but no valgrind report is produced. The same command without valgrind also results in a hung foreground task. If run under valgrind, memcheck-x86-li goes to 99% CPU on CTRL-C. I currently suspect problems with mySQL as the origin of these problems. I was this morning getting messages of the form "Lost connection to MySQL server at 'reading authorization packet'". I was also observing MySQL aborted clients and connects. I have increased the MySQL connect timeout, and can no longer reproduce these reports. For now, I assume that problem is fixed, but I can't yet say if the rsyslog hangs have stopped. I wonder if what was happening was that MySQL was "going away" in some sense, and that rsyslog was not reconnecting to it successfully, *and* not retrying? I noticed that although the ActionQueue for the mysql output module is not disk-assisted, the debug log records: action 4 queue: save on shutdown 1, max disk space allowed 0 So I've set $ActionQueueSaveOnShutdown off. However this hasn't changed the hang behaviour with -n. Since I can't get rsyslog to run in the foreground under valgrind, I am now running daemonized without valgrind (but with encryption); perhaps these changes have fixed the problem. I should know by late this evening - when the problem is observed, the server never lives for more than a few hours. -- Jack. From rgerhards at hq.adiscon.com Tue Nov 10 09:16:32 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 09:16:32 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls (valgrind log) References: <4AF8677B.40808@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033A3@GRFEXC.intern.adiscon.com> Thanks for the log file. Unfortunately, it is not a full run. I gues I needed to make some more things clear. The command line you need to use here $ valgrind /usr/sbin/rsyslogd -c4 -d >rsyslog.log 2>&1 Is the same that you usually use, except that -n must be specified. I unfortunately do no know what you (or better: your distro) usually uses, but you may see it in a process list. Before you start it that way, you must stop the regular syslogd and you must be sure to su to root. The -d switch, while generally useful, takes a lot of time and will generate an immense log. So it is best to not specify it for now. What I am after are valgrind violations (these are printed during the run) and the memory stats at the end of the run. The later will show any memory leaks. The log I received did not receive any message via TLS but two via UDP. It looks like a very short run. Thanks again for your help, I hope I have now explained better ;) Rainer > -----Original Message----- > From: Mr. Demeanour [mailto:mrdemeanour at jackpot.uk.net] > Sent: Monday, November 09, 2009 8:03 PM > To: Rainer Gerhards > Subject: Re: Rsyslog 4.4.2: server out-of-memory with gnutls > (valgrind log) > > Hi, > > I finally got a decent valgrind log using this command: > $ valgrind /usr/sbin/rsyslogd -c4 -d >rsyslog.log 2>&1 > > The log is attached. Free memory was declining steadily > throughout this run. > > Thanks, > -- > Jack. > From rgerhards at hq.adiscon.com Tue Nov 10 09:44:25 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 09:44:25 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103395@GRFEXC.intern.adiscon.com> <4AF8424D.30306@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033A8@GRFEXC.intern.adiscon.com> And finally a reply to "the meat" ;) > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Monday, November 09, 2009 5:25 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Rainer Gerhards wrote: > > Jack, > > > > Quick update: I was able to run a (relatively quick) test on a > > configuration that hopefully later becomes part of the testbench (I > > am working towards that goal and thought doing a manual > check at that > > stage would not hurt). It is not based your config, and it is > > relatively simple and straightforward. Still, it uses TLS, and uses > > it in anon mode like you did. I used 4.4.2 on the server. I > processed > > around 500,000 message (not kept track of the actual number). I did > > three such runs, all runing under the excellent valgrind[1] memory > > debugger. > > > > For none of the runs, valgrind reported any memory leaks. While this > > may not be an ultimate indication, valgrind is *very* effective in > > finding leaks and based on what you wrote I would have expected a > > small chunk of memory to be lost per message. > > OK (thanks). > > I have formed the impression that the problems are occurring > after some > period of running time; free memory decreases quite slowly, until some > unknown event causes a rather rapid degeneration. That is: I suspect > that any leak is not likely to be observable on a per-message basis, > until this unknown event has occured. This would explain what we currently see. It seems to be TLS-related, as you said. So it is unlikely to be a message at all, but rather a TLS event that causes the mem leak. With that said, I should probbably see if a connection abort can trigger one... > > So the bottom line is that I currently cannot reproduce the > bug. This > > may change when I finally import your config. However, it would be > > useful for me if you could run valgrind on rsyslogd in your > > environment and let me know if valgrind reports any memory leaks. > > Doing so considerably slows down rsyslogd, but given your load, I'd > > expect that it would be acceptable. > > > > To run under valgrind is relatively simply. Valgrind is available as > > a package on almost all distros. All you need to do is run valgrind > > and specify your usual rsyslogd command line as the parameter. It is > > recommended to do this in the foreground (see rsyslog > troubleshooting > > doc). > > > > So, for example, if you start rsyslogd usually by > > > > /sbin/rsyslogd -c4 -... > > So "valgrind /usr/sbin/rsyslogd -c4" results in rsyslogd backgrounding > promptly, at which point valgrind prints its report (which shows no > leaks - unsurprisingly). > > "valgrind /usr/sbin/rsyslogd -c4 -n" results in a hang. > CTRL-C fails to > kill the foreground task. That's intended behavior, ctrl-c is only enabled in debug mode (I took this over from sysklogd and never question it". > "kill -9 " kills the task, but no > valgrind report is produced. You must not use -9, which is untrappable. Just send the default SIGTERM: $ kill # no signal specified at all! >The same command without valgrind also > results in a hung foreground task. If run under valgrind, > memcheck-x86-li goes to 99% CPU on CTRL-C. > > I currently suspect problems with mySQL as the origin of > these problems. > I was this morning getting messages of the form > "Lost connection to MySQL server at 'reading authorization packet'". > I was also observing MySQL aborted clients and connects. I have > increased the MySQL connect timeout, and can no longer reproduce these > reports. For now, I assume that problem is fixed, but I can't > yet say if > the rsyslog hangs have stopped. > > I wonder if what was happening was that MySQL was "going away" in some > sense, and that rsyslog was not reconnecting to it successfully, *and* > not retrying? Hard to guess... > > I noticed that although the ActionQueue for the mysql output module is > not disk-assisted, the debug log records: > action 4 queue: save on shutdown 1, max disk space allowed 0 The debug output just spits out the values of the variables. Saveonshutdown is true by default, but if there is no disk queue, that won't help anything at all ;) In short: that's OK. > So I've set $ActionQueueSaveOnShutdown off. However this > hasn't changed > the hang behaviour with -n. > > Since I can't get rsyslog to run in the foreground under > valgrind, I am > now running daemonized without valgrind (but with encryption); perhaps > these changes have fixed the problem. I should know by late > this evening > - when the problem is observed, the server never lives for more than a > few hours. I hope this -and the other- mail will help straighten out the issue. Any logs you can send to my private mail address as you have already done ;) Rainer > -- > Jack. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 10 10:28:55 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 10:28:55 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103395@GRFEXC.intern.adiscon.com><4AF8424D.30306@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033A8@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033AB@GRFEXC.intern.adiscon.com> > This would explain what we currently see. It seems to be TLS-related, > as you > said. So it is unlikely to be a message at all, but rather a TLS event > that > causes the mem leak. With that said, I should probbably see if a > connection > abort can trigger one... Just for the records, the connection aborts I triggered did NOT result in leaked memory :( Rainer From rgerhards at hq.adiscon.com Tue Nov 10 10:53:16 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 10:53:16 +0100 Subject: [rsyslog] 4.4.2 leaks: another log References: <4AF93722.8040101@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com> excellent! I also see some violations, what is a good thing. But I notice I unfortunately missed one thing: by default, the violations do not include the loadbale module addresses. Would it be possible that you create a custom build of rsyslog with the "--enable-valgrind" configure option? That does not cause any overhead, but permits to see the function names on exit (and thus is really useful). Thanks, Rainer > -----Original Message----- > From: Mr. Demeanour [mailto:mrdemeanour at jackpot.uk.net] > Sent: Tuesday, November 10, 2009 10:49 AM > To: Rainer Gerhards > Subject: 4.4.2 leaks: another log > > Hi. > > Thanks for all your help. I didn't realise that -n suppressed SIGKILL. > I > also didn't realise that it was taking up to five minutes with -n and > valgrind, between the successful opening of a UDP listener on 514 and a > TCP listener appearing on 10514! That (and my misunderstanding of > signal > handling) is why I thought it had hung. > > So perhaps something is strange about my certificate. I'll see if I can > test it somehow. > > Anyway, here is a log apparently showing leakage; it represents about > 400 TCP messages over 30 minutes. The command was: > valgrind --leak-check=full /usr/sbin/rsyslogd -c4 -n >rsyslog.log 2>&1 > > By the time I killed it, memory usage had climbed from about 70M to > 100M. With a non-encrypting rsyslog (and without valgrind), usage is > stable at around 70M. Here's a "free" while running a non-encrypting > service: > > # free > total used free shared buffers > cached > Mem: 774972 732564 42408 0 22544 > 641436 > -/+ buffers/cache: 68584 706388 > Swap: 1951856 5640 1946216 > > The picture remains like that more-or-less indefinitely. > > Just before I killed the valgrind run corresponding to that log, it > looked like this: > # free > total used free shared buffers > cached > Mem: 774972 766292 8680 0 22016 > 643940 > -/+ buffers/cache: 100336 674636 > Swap: 1951856 5640 1946216 > > > > Now I know how to do this, I should be able to do it on demand. Thanks > again. > > -- > Jack. From rgerhards at hq.adiscon.com Tue Nov 10 11:09:41 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 11:09:41 +0100 Subject: [rsyslog] 4.4.2 leaks: another log References: <4AF93722.8040101@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> It just occurred to me that you may be seeing an anomaly in clib's malloc subsystem. I fought with this in early summer, but somehow have not mentioned it in the change log. The change itself was done on June, 22nd 2009. I have just checked, 4.4.2 does not have this code. I noticed that while there were some reports in the valgrind log, they were not large enough to explain what you saw. The libc malloc issue was discussed on the mailing list, and this here is a link to a later wrap-up type of message: http://lists.adiscon.net/pipermail/rsyslog/2009-June/002372.html So my suggestion is to move to v4-stable and see if the problem persists there (obviously, contrary to what I said, v5 would have fixed it in that case as well - so never trust a dev... ;)). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, November 10, 2009 10:53 AM > To: rsyslog-users > Subject: Re: [rsyslog] 4.4.2 leaks: another log > > excellent! I also see some violations, what is a good thing. But I > notice I > unfortunately missed one thing: by default, the violations do not > include the > loadbale module addresses. Would it be possible that you create a > custom > build of rsyslog with the "--enable-valgrind" configure option? That > does not > cause any overhead, but permits to see the function names on exit (and > thus > is really useful). > > Thanks, > Rainer > > > -----Original Message----- > > From: Mr. Demeanour [mailto:mrdemeanour at jackpot.uk.net] > > Sent: Tuesday, November 10, 2009 10:49 AM > > To: Rainer Gerhards > > Subject: 4.4.2 leaks: another log > > > > Hi. > > > > Thanks for all your help. I didn't realise that -n suppressed > SIGKILL. > > I > > also didn't realise that it was taking up to five minutes with -n and > > valgrind, between the successful opening of a UDP listener on 514 and > a > > TCP listener appearing on 10514! That (and my misunderstanding of > > signal > > handling) is why I thought it had hung. > > > > So perhaps something is strange about my certificate. I'll see if I > can > > test it somehow. > > > > Anyway, here is a log apparently showing leakage; it represents about > > 400 TCP messages over 30 minutes. The command was: > > valgrind --leak-check=full /usr/sbin/rsyslogd -c4 -n >rsyslog.log > 2>&1 > > > > By the time I killed it, memory usage had climbed from about 70M to > > 100M. With a non-encrypting rsyslog (and without valgrind), usage is > > stable at around 70M. Here's a "free" while running a non-encrypting > > service: > > > > # free > > total used free shared buffers > > cached > > Mem: 774972 732564 42408 0 22544 > > 641436 > > -/+ buffers/cache: 68584 706388 > > Swap: 1951856 5640 1946216 > > > > The picture remains like that more-or-less indefinitely. > > > > Just before I killed the valgrind run corresponding to that log, it > > looked like this: > > # free > > total used free shared buffers > > cached > > Mem: 774972 766292 8680 0 22016 > > 643940 > > -/+ buffers/cache: 100336 674636 > > Swap: 1951856 5640 1946216 > > > > > > > > Now I know how to do this, I should be able to do it on demand. > Thanks > > again. > > > > -- > > Jack. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mrdemeanour at jackpot.uk.net Tue Nov 10 11:29:41 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Tue, 10 Nov 2009 10:29:41 +0000 Subject: [rsyslog] 4.4.2 leaks: another log In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> References: <4AF93722.8040101@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> Message-ID: <4AF94095.2040205@jackpot.uk.net> Rainer Gerhards wrote: > It just occurred to me that you may be seeing an anomaly in clib's > malloc subsystem. I fought with this in early summer, but somehow > have not mentioned it in the change log. The change itself was done > on June, 22nd 2009. I have just checked, 4.4.2 does not have this > code. I noticed that while there were some reports in the valgrind > log, they were not large enough to explain what you saw. > > The libc malloc issue was discussed on the mailing list, and this > here is a link to a later wrap-up type of message: > > http://lists.adiscon.net/pipermail/rsyslog/2009-June/002372.html > > So my suggestion is to move to v4-stable and see if the problem > persists there (obviously, contrary to what I said, v5 would have > fixed it in that case as well - so never trust a dev... ;)). OK. I have to be somewhere else in half an hour, so I'll return to this later. -- Jack. From rgerhards at hq.adiscon.com Tue Nov 10 11:30:59 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 11:30:59 +0100 Subject: [rsyslog] 4.4.2 leaks: another log References: <4AF93722.8040101@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> <4AF94095.2040205@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033AE@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Tuesday, November 10, 2009 11:30 AM > To: rsyslog-users > Subject: Re: [rsyslog] 4.4.2 leaks: another log > > Rainer Gerhards wrote: > > It just occurred to me that you may be seeing an anomaly in clib's > > malloc subsystem. I fought with this in early summer, but somehow > > have not mentioned it in the change log. The change itself was done > > on June, 22nd 2009. I have just checked, 4.4.2 does not have this > > code. I noticed that while there were some reports in the valgrind > > log, they were not large enough to explain what you saw. > > > > The libc malloc issue was discussed on the mailing list, and this > > here is a link to a later wrap-up type of message: > > > > http://lists.adiscon.net/pipermail/rsyslog/2009-June/002372.html > > > > So my suggestion is to move to v4-stable and see if the problem > > persists there (obviously, contrary to what I said, v5 would have > > fixed it in that case as well - so never trust a dev... ;)). args, important type: suggest to move to v4-beta NOT -stable... > > OK. > > I have to be somewhere else in half an hour, so I'll return to this > later. > > -- > Jack. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 10 14:03:00 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 14:03:00 +0100 Subject: [rsyslog] On-Demand Debugging Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033B1@GRFEXC.intern.adiscon.com> Hi all, I worked today on the capability to obtain debug information from a running instance. Unfortunately, a small code change was needed to make it work well. If you cannot upgrade or apply the patch, I can create instructions to do it even without the patch, but that is a rather inconvenient way of doing thigs. The patch is here, it should apply to all recent v4 and v5 builds: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=2b5e15d549940c7ace9017ea f40d523e3741c9a2 I have also updated the documentation of the debug system and hope that it is much more usable this time. It specifically explains how to work with this new type of on-demand debugging. Details are online: http://www.rsyslog.com/doc-debug.html If that capability is useful, I could extend it in some ways. Most importantly, the current version does NOT permit to send debug logs to another remote instance. However, I tried to do as few changes as possible in an effort to help our current debugging efforts. Any new functionality will probably be v5-exclusive. I hope this is a useful new feature. So far, it is available via the git heads only, but I'll see that I release as soon as it makes sense (I don't like to release just for this mini-feature...). Rainer From szybalski at gmail.com Wed Nov 11 02:02:47 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Tue, 10 Nov 2009 19:02:47 -0600 Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> Message-ID: <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> Hello, I have a web application deployed in multiprocess and multi-thread scenario. ( 3 processes and 10 threads each). I want to log search query string from users to a file ?called todaysdate.log ...20091109.log I'm using a python app but I can't get rsyslog in debian system to catch and write my messages. Would you know how can I set this up? Here is a sample test case....in python. Save below to a file and run it. python testfile.py #----testfile.py----- from logging.handlers import SysLogHandler import logging # create logger logger = logging.getLogger("myapp") logger.setLevel(logging.DEBUG) ch = SysLogHandler('/dev/log') ch.setLevel(logging.DEBUG) # create formatter formatter = logging.Formatter("%(asctime)s - %(name)s-%(levelname)s - %(message)s") # add formatter to ch ch.setFormatter(formatter) logger.addHandler(ch) logger.info('This is a message') How can I setup rsyslog to filter "myapp" and save the messages to a file in /home/lucas/myapp/20091109.txt Thanks, Lucas From mrdemeanour at jackpot.uk.net Wed Nov 11 18:38:21 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Wed, 11 Nov 2009 17:38:21 +0000 Subject: [rsyslog] 4.4.2 leaks: another log In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71033AE@GRFEXC.intern.adiscon.com> References: <4AF93722.8040101@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> <4AF94095.2040205@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AE@GRFEXC.intern.adiscon.com> Message-ID: <4AFAF68D.9060203@jackpot.uk.net> Rainer Gerhards wrote: >>> >>> So my suggestion is to move to v4-stable and see if the problem >>> persists there (obviously, contrary to what I said, v5 would have >>> fixed it in that case as well - so never trust a dev... ;)). > > > args, important type: suggest to move to v4-beta NOT -stable... OK, so it looks like I've still got the problem, with v4 beta (4.5.6). I'll have another go with valgrind tomorrow. -- Jack. From szybalski at gmail.com Thu Nov 12 15:19:26 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Thu, 12 Nov 2009 08:19:26 -0600 Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> Message-ID: <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> Anybody know what the filter or a config line would be that would filter my "myapp" messages to a file in /home/lucas/20091112.txt? Ideas? Thanks, Lucas On Tue, Nov 10, 2009 at 7:02 PM, Lukasz Szybalski wrote: > Hello, > I have a web application deployed in multiprocess and multi-thread > scenario. ( 3 processes and 10 threads each). > > I want to log search query string from users to a file ?called > todaysdate.log ...20091109.log > > > I'm using a python app but I can't get rsyslog in debian system to > catch and write my messages. Would you know how can I set this up? > > > Here is a sample test case....in python. > Save below to a file and run it. > python testfile.py > > > #----testfile.py----- > from logging.handlers import SysLogHandler > > import logging > > > # create logger > logger = logging.getLogger("myapp") > logger.setLevel(logging.DEBUG) > ch = SysLogHandler('/dev/log') > ch.setLevel(logging.DEBUG) > # create formatter > formatter = logging.Formatter("%(asctime)s - %(name)s-%(levelname)s - > %(message)s") > # add formatter to ch > ch.setFormatter(formatter) > logger.addHandler(ch) > logger.info('This is a message') > > > How can I setup rsyslog to filter "myapp" and save the messages to a > file in /home/lucas/myapp/20091109.txt > > Thanks, > Lucas > -- Setup CalendarServer for your company. http://lucasmanual.com/mywiki/CalendarServer Automotive Recall Database - See if you vehicle has a recall http://lucasmanual.com/recall From rgerhards at hq.adiscon.com Thu Nov 12 15:23:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 12 Nov 2009 15:23:44 +0100 Subject: [rsyslog] multiprocess/multithread web app to rsyslog References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com><804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> If "myapp" is the tag, you can use this: :syslogtag, contains, "maypp" /home/lucas/20091112.txt There are better comparisons than "contains", but I don't know them out of my head. They are in the filter doc. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Lukasz Szybalski > Sent: Thursday, November 12, 2009 3:19 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] multiprocess/multithread web app to rsyslog > > Anybody know what the filter or a config line would be that would > filter my "myapp" messages to a file in /home/lucas/20091112.txt? > > Ideas? > Thanks, > Lucas > > > On Tue, Nov 10, 2009 at 7:02 PM, Lukasz Szybalski > wrote: > > Hello, > > I have a web application deployed in multiprocess and multi-thread > > scenario. ( 3 processes and 10 threads each). > > > > I want to log search query string from users to a file ?called > > todaysdate.log ...20091109.log > > > > > > I'm using a python app but I can't get rsyslog in debian system to > > catch and write my messages. Would you know how can I set this up? > > > > > > Here is a sample test case....in python. > > Save below to a file and run it. > > python testfile.py > > > > > > #----testfile.py----- > > from logging.handlers import SysLogHandler > > > > import logging > > > > > > # create logger > > logger = logging.getLogger("myapp") > > logger.setLevel(logging.DEBUG) > > ch = SysLogHandler('/dev/log') > > ch.setLevel(logging.DEBUG) > > # create formatter > > formatter = logging.Formatter("%(asctime)s - %(name)s-%(levelname)s - > > %(message)s") > > # add formatter to ch > > ch.setFormatter(formatter) > > logger.addHandler(ch) > > logger.info('This is a message') > > > > > > How can I setup rsyslog to filter "myapp" and save the messages to a > > file in /home/lucas/myapp/20091109.txt > > > > Thanks, > > Lucas > > > > > > -- > Setup CalendarServer for your company. > http://lucasmanual.com/mywiki/CalendarServer > Automotive Recall Database - See if you vehicle has a recall > http://lucasmanual.com/recall > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From szybalski at gmail.com Thu Nov 12 15:59:59 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Thu, 12 Nov 2009 08:59:59 -0600 Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> Message-ID: <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> On Thu, Nov 12, 2009 at 8:23 AM, Rainer Gerhards wrote: > If "myapp" is the tag, you can use this: > > :syslogtag, contains, "maypp" /home/lucas/20091112.txt > > There are better comparisons than "contains", but I don't know them out of my > head. They are in the filter doc. > Getting closer. I put a file in /etc/rsyslog.d/myapp.conf Inside I have: :syslogtag, contains, "myapp" /home/lucas/20091112.txt the rsyslog creates the file but no messages are logged in, they only appear in /var/log/messages Nov 12 08:45:32 localhost 2009-11-12 08:45:32,452 - myapp - INFO - This is a message Ideas why its not logging it in? Also, Can I make this config file like this? It doesn't seem to work...? :syslogtag, contains, "myapp" /home/lucas/$year$month$day.txt http://www.rsyslog.com/doc-property_replacer.html Thanks, Lucas > HTH > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Lukasz Szybalski >> Sent: Thursday, November 12, 2009 3:19 PM >> To: rsyslog at lists.adiscon.com >> Subject: Re: [rsyslog] multiprocess/multithread web app to rsyslog >> >> Anybody know what the filter or a config line would be that would >> filter my "myapp" messages to a file in /home/lucas/20091112.txt? >> >> Ideas? >> Thanks, >> Lucas >> >> >> On Tue, Nov 10, 2009 at 7:02 PM, Lukasz Szybalski >> wrote: >> > Hello, >> > I have a web application deployed in multiprocess and multi-thread >> > scenario. ( 3 processes and 10 threads each). >> > >> > I want to log search query string from users to a file ?called >> > todaysdate.log ...20091109.log >> > >> > >> > I'm using a python app but I can't get rsyslog in debian system to >> > catch and write my messages. Would you know how can I set this up? >> > >> > >> > Here is a sample test case....in python. >> > Save below to a file and run it. >> > python testfile.py >> > >> > >> > #----testfile.py----- >> > from logging.handlers import SysLogHandler >> > >> > import logging >> > >> > >> > # create logger >> > logger = logging.getLogger("myapp") >> > logger.setLevel(logging.DEBUG) >> > ch = SysLogHandler('/dev/log') >> > ch.setLevel(logging.DEBUG) >> > # create formatter >> > formatter = logging.Formatter("%(asctime)s - %(name)s-%(levelname)s - >> > %(message)s") >> > # add formatter to ch >> > ch.setFormatter(formatter) >> > logger.addHandler(ch) >> > logger.info('This is a message') >> > >> > >> > How can I setup rsyslog to filter "myapp" and save the messages to a >> > file in /home/lucas/myapp/20091109.txt >> > >> > Thanks, >> > Lucas >> > >> >> From rgerhards at hq.adiscon.com Thu Nov 12 16:33:34 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 12 Nov 2009 16:33:34 +0100 Subject: [rsyslog] multiprocess/multithread web app to rsyslog References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com><804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com><804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Lukasz Szybalski > Sent: Thursday, November 12, 2009 4:00 PM > To: rsyslog-users > Subject: Re: [rsyslog] multiprocess/multithread web app to rsyslog > > On Thu, Nov 12, 2009 at 8:23 AM, Rainer Gerhards > wrote: > > If "myapp" is the tag, you can use this: > > > > :syslogtag, contains, "maypp" /home/lucas/20091112.txt > > > > There are better comparisons than "contains", but I don't know them > out of my > > head. They are in the filter doc. > > > > Getting closer. > I put a file in /etc/rsyslog.d/myapp.conf > > Inside I have: > :syslogtag, contains, "myapp" /home/lucas/20091112.txt > > the rsyslog creates the file but no messages are logged in, they only > appear in /var/log/messages > > Nov 12 08:45:32 localhost 2009-11-12 08:45:32,452 - myapp - INFO - aha! this message is not really well-formed, please also see http://www.rsyslog.com/doc-syslog_parsing.html So the tag here is "2009-11-12". You can either use a custom parser, or if that would be sufficent, check msg instead of syslogtag. Rainer > This is a message > > > Ideas why its not logging it in? > > Also, > Can I make this config file like this? It doesn't seem to work...? > :syslogtag, contains, "myapp" /home/lucas/$year$month$day.txt > > > http://www.rsyslog.com/doc-property_replacer.html > > Thanks, > Lucas > > > > > > HTH > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Lukasz Szybalski > >> Sent: Thursday, November 12, 2009 3:19 PM > >> To: rsyslog at lists.adiscon.com > >> Subject: Re: [rsyslog] multiprocess/multithread web app to rsyslog > >> > >> Anybody know what the filter or a config line would be that would > >> filter my "myapp" messages to a file in /home/lucas/20091112.txt? > >> > >> Ideas? > >> Thanks, > >> Lucas > >> > >> > >> On Tue, Nov 10, 2009 at 7:02 PM, Lukasz Szybalski > > >> wrote: > >> > Hello, > >> > I have a web application deployed in multiprocess and multi-thread > >> > scenario. ( 3 processes and 10 threads each). > >> > > >> > I want to log search query string from users to a file ?called > >> > todaysdate.log ...20091109.log > >> > > >> > > >> > I'm using a python app but I can't get rsyslog in debian system to > >> > catch and write my messages. Would you know how can I set this up? > >> > > >> > > >> > Here is a sample test case....in python. > >> > Save below to a file and run it. > >> > python testfile.py > >> > > >> > > >> > #----testfile.py----- > >> > from logging.handlers import SysLogHandler > >> > > >> > import logging > >> > > >> > > >> > # create logger > >> > logger = logging.getLogger("myapp") > >> > logger.setLevel(logging.DEBUG) > >> > ch = SysLogHandler('/dev/log') > >> > ch.setLevel(logging.DEBUG) > >> > # create formatter > >> > formatter = logging.Formatter("%(asctime)s - %(name)s- > %(levelname)s - > >> > %(message)s") > >> > # add formatter to ch > >> > ch.setFormatter(formatter) > >> > logger.addHandler(ch) > >> > logger.info('This is a message') > >> > > >> > > >> > How can I setup rsyslog to filter "myapp" and save the messages to > a > >> > file in /home/lucas/myapp/20091109.txt > >> > > >> > Thanks, > >> > Lucas > >> > > >> > >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From szybalski at gmail.com Thu Nov 12 18:18:02 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Thu, 12 Nov 2009 11:18:02 -0600 Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> Message-ID: <804e5c70911120918s935f756j602d1c0ce7f0798b@mail.gmail.com> >> >> > >> >> > >> >> > Here is a sample test case....in python. >> >> > Save below to a file and run it. >> >> > python testfile.py >> >> > >> >> > >> >> > #----testfile.py----- >> >> > from logging.handlers import SysLogHandler >> >> > >> >> > import logging >> >> > >> >> > >> >> > # create logger >> >> > logger = logging.getLogger("myapp") >> >> > logger.setLevel(logging.DEBUG) >> >> > ch = SysLogHandler('/dev/log') >> >> > ch.setLevel(logging.DEBUG) >> >> > # create formatter >> >> > formatter = logging.Formatter("%(asctime)s - %(name)s- >> %(levelname)s - >> >> > %(message)s") >> >> > # add formatter to ch >> >> > ch.setFormatter(formatter) >> >> > logger.addHandler(ch) >> >> > logger.info('This is a message') >> >> > >> >> > >> >> > How can I setup rsyslog to filter "myapp" and save the messages to 1. So in my code I've defined a formatter to be: formatter = logging.Formatter("%(asctime)s - %(name)s- %(levelname)s - %(message)s") I don't care how it looks as long as it has a date, time, name of the app, level, and message. Is there a standard formatting I should be using? I don't want to create a custom solution if I can use standard one? 2. What about saving the file with todays date. I don't want to rotate the logs, but I want to have a file for each day saved in /home/lucas/$year$month$date.txt Above does not work, unless I'm missing some escape characters/parentheses? Thanks, Lucas From david at lang.hm Thu Nov 12 20:25:40 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 12 Nov 2009 11:25:40 -0800 (PST) Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <804e5c70911120918s935f756j602d1c0ce7f0798b@mail.gmail.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> <804e5c70911120918s935f756j602d1c0ce7f0798b@mail.gmail.com> Message-ID: On Thu, 12 Nov 2009, Lukasz Szybalski wrote: >>>>>> >>>>>> >>>>>> Here is a sample test case....in python. >>>>>> Save below to a file and run it. >>>>>> python testfile.py >>>>>> >>>>>> >>>>>> #----testfile.py----- >>>>>> from logging.handlers import SysLogHandler >>>>>> >>>>>> import logging >>>>>> >>>>>> >>>>>> # create logger >>>>>> logger = logging.getLogger("myapp") >>>>>> logger.setLevel(logging.DEBUG) >>>>>> ch = SysLogHandler('/dev/log') >>>>>> ch.setLevel(logging.DEBUG) >>>>>> # create formatter >>>>>> formatter = logging.Formatter("%(asctime)s - %(name)s- >>> %(levelname)s - >>>>>> %(message)s") >>>>>> # add formatter to ch >>>>>> ch.setFormatter(formatter) >>>>>> logger.addHandler(ch) >>>>>> logger.info('This is a message') >>>>>> >>>>>> >>>>>> How can I setup rsyslog to filter "myapp" and save the messages to > > 1. > So in my code I've defined a formatter to be: > formatter = logging.Formatter("%(asctime)s - %(name)s- %(levelname)s - > %(message)s") > > I don't care how it looks as long as it has a date, time, name of the > app, level, and message. Is there a standard formatting I should be > using? I don't want to create a custom solution if I can use standard > one? the format over the wire should be date server tag message if the message does not have this format, rsyslog will add a priority, date and server to the message in addition, the date you are setting is not formatted appropriately the date should be MMM DD HH:MM:SS I am not familiar enough with the logger package in python to tell you how to format it this way. however, based on the message that you sent earlier, if you set your format to be formatter = logging.Formatter("%(name)s %(message)s") it may be close enough to the correct thing to work. > 2. What about saving the file with todays date. I don't want to rotate > the logs, but I want to have a file for each day saved in > /home/lucas/$year$month$date.txt > Above does not work, unless I'm missing some escape characters/parentheses? look in the section on dynafiles. I have not used this feature, so I can't help directly. David Lang From szybalski at gmail.com Thu Nov 12 23:32:43 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Thu, 12 Nov 2009 16:32:43 -0600 Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> <804e5c70911120918s935f756j602d1c0ce7f0798b@mail.gmail.com> Message-ID: <804e5c70911121432y404d0a00h2f080590140addf@mail.gmail.com> On Thu, Nov 12, 2009 at 1:25 PM, wrote: > On Thu, 12 Nov 2009, Lukasz Szybalski wrote: > >>>>>>> >>>>>>> >>>>>>> Here is a sample test case....in python. >>>>>>> Save below to a file and run it. >>>>>>> python testfile.py >>>>>>> >>>>>>> >>>>>>> #----testfile.py----- >>>>>>> from logging.handlers import SysLogHandler >>>>>>> >>>>>>> import logging >>>>>>> >>>>>>> >>>>>>> # create logger >>>>>>> logger = logging.getLogger("myapp") >>>>>>> logger.setLevel(logging.DEBUG) >>>>>>> ch = SysLogHandler('/dev/log') >>>>>>> ch.setLevel(logging.DEBUG) >>>>>>> # create formatter >>>>>>> formatter = logging.Formatter("%(asctime)s - %(name)s- >>>> %(levelname)s - >>>>>>> %(message)s") >>>>>>> # add formatter to ch >>>>>>> ch.setFormatter(formatter) >>>>>>> logger.addHandler(ch) >>>>>>> logger.info('This is a message') >>>>>>> >>>>>>> >>>>>>> How can I setup rsyslog to filter "myapp" and save the messages to >> >> 1. >> So in my code I've defined a formatter to be: >> formatter = logging.Formatter("%(asctime)s - %(name)s- %(levelname)s - >> %(message)s") >> >> I don't care how it looks as long as it has a date, time, name of the >> app, level, and message. Is there a standard formatting I should be >> using? I don't want to create a custom solution if I can use standard >> one? > > the format over the wire should be > > date server tag message > > if the message does not have this format, rsyslog will add a priority, > date and server to the message > > in addition, the date you are setting is not formatted appropriately > > the date should be > > MMM DD HH:MM:SS > > I am not familiar enough with the logger package in python to tell you how > to format it this way. > > however, based on the message that you sent earlier, if you set your > format to be > > formatter = logging.Formatter("%(name)s %(message)s") This works.... nice... > > it may be close enough to the correct thing to work. > >> 2. What about saving the file with todays date. I don't want to rotate >> the logs, but I want to have a file for each day saved in >> /home/lucas/$year$month$date.txt >> Above does not work, unless I'm missing some escape characters/parentheses? > > look in the section on dynafiles. I have not used this feature, so I can't > help directly. Now I just have to find the proper syntax to get the file to save to :syslogtag, contains, "myapp" /home/lucas/20091112.txt to /home/lucas/$year$month$date.txt Lucas From david at lang.hm Fri Nov 13 00:02:08 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 12 Nov 2009 15:02:08 -0800 (PST) Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <804e5c70911121432y404d0a00h2f080590140addf@mail.gmail.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> <804e5c70911120918s935f756j602d1c0ce7f0798b@mail.gmail.com> <804e5c70911121432y404d0a00h2f080590140addf@mail.gmail.com> Message-ID: On Thu, 12 Nov 2009, Lukasz Szybalski wrote: > On Thu, Nov 12, 2009 at 1:25 PM, wrote: >> On Thu, 12 Nov 2009, Lukasz Szybalski wrote: >> >> >> it may be close enough to the correct thing to work. >> >>> 2. What about saving the file with todays date. I don't want to rotate >>> the logs, but I want to have a file for each day saved in >>> /home/lucas/$year$month$date.txt >>> Above does not work, unless I'm missing some escape characters/parentheses? >> >> look in the section on dynafiles. I have not used this feature, so I can't >> help directly. > > Now I just have to find the proper syntax to get the file to save to > :syslogtag, contains, "myapp" /home/lucas/20091112.txt > to > /home/lucas/$year$month$date.txt look at DynFile in http://www.rsyslog.com/doc-rsyslog_conf_actions.html David Lang From tbergfeld at hq.adiscon.com Fri Nov 13 12:51:09 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 13 Nov 2009 12:51:09 +0100 Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com> Hi all, Today, we released a new v5-beta branch. It bases on the last v5-devel, plus has some bug fixes and some enhancements. The enhancements provide a slightly increased performance. This release is a very important milestone. Based on feedback from the v5-devel branches, it is in pretty good shape. Our aim is to have a quick beta phase this time (taking the problems with the current v5-stable) into account. To do so, we need your cooperation. Please try out this new version and report back any issues you may experience. It would also be good to know if you use it and do not run into any issues. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article427.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-187.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From david at lang.hm Sat Nov 14 03:48:06 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 13 Nov 2009 18:48:06 -0800 (PST) Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com> Message-ID: I've got this installed in production and it's working well so far. monday morning (pacific time) should give it a good stress test. David Lang On Fri, 13 Nov 2009, Tom Bergfeld wrote: > Date: Fri, 13 Nov 2009 12:51:09 +0100 > From: Tom Bergfeld > Reply-To: rsyslog-users > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released > > Hi all, > > Today, we released a new v5-beta branch. It bases on the last v5-devel, plus > has some bug fixes and some enhancements. The enhancements provide a slightly > increased performance. This release is a very important milestone. Based on > feedback from the v5-devel branches, it is in pretty good shape. Our aim is > to have a quick beta phase this time (taking the problems with the current > v5-stable) into account. To do so, we need your cooperation. Please try out > this new version and report back any issues you may experience. It would also > be good to know if you use it and do not run into any issues. > > See Changelog for more details. > > ChangeLog: > > http://www.rsyslog.com/Article427.phtml > > Download: > > http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-187.phtml > > > As always, feedback is appreciated. > > Best regards, > Tom Bergfeld > -- > Support > ======= > > Improving rsyslog is costly, but you can help! We are looking for > organizations that find rsyslog useful and wish to contribute back. You can > contribute by reporting bugs, improve the software, or donate money or > equipment. > > Commercial support contracts for rsyslog are available, and they help finance > continued maintenance. Adiscon GmbH, a privately held German company, is > currently funding rsyslog development. We are always looking for interesting > development projects. For details on how to help, please see > http://www.rsyslog.com/doc-how2help.html . > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mrdemeanour at jackpot.uk.net Sat Nov 14 14:59:43 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Sat, 14 Nov 2009 13:59:43 +0000 Subject: [rsyslog] 4.4.2 leaks: another log In-Reply-To: <4AFAF68D.9060203@jackpot.uk.net> References: <4AF93722.8040101@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> <4AF94095.2040205@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AE@GRFEXC.intern.adiscon.com> <4AFAF68D.9060203@jackpot.uk.net> Message-ID: <4AFEB7CF.3060309@jackpot.uk.net> Mr. Demeanour wrote: > Rainer Gerhards wrote: >>>> So my suggestion is to move to v4-stable and see if the problem >>>> persists there (obviously, contrary to what I said, v5 would >>>> have fixed it in that case as well - so never trust a dev... >>>> ;)). >> >> args, important type: suggest to move to v4-beta NOT -stable... > > OK, so it looks like I've still got the problem, with v4 beta > (4.5.6). I'll have another go with valgrind tomorrow. I have been running a setup with 4.5.6 collecting messages by GnuTLS only, and logging to a file; and the problem has failed to materialise. Perhaps I didn't run it for long enough; but a few hours is normally sufficient for serious memory depletion to occur, and it hasn't so far. The other variables are that only one (LAN) client is logging to this instance - all previous runs have involved multiple clients, some logging via the internet; and that only one input and one output method is configured. Previous tests have had both imudp and imtcp configured, and have used both omfile and ommysql. I've now changed this from logging to a file to log via mySQL. If that remains stable after several hours, I'll reconfigure with dual input and output methods; then with multiple clients. Eventually I should be able to describe a method of replication. -- Jack. From rgerhards at hq.adiscon.com Mon Nov 16 15:00:20 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 16 Nov 2009 15:00:20 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> Hi all, I am working again on moving the DNS name resolution outside of the input thread of those sources where this is potentially time-consuming and affecting message acceptance rates. As it turned out, currently imudp seems to be the only case. While this is potentially easy to do, a problem is ACLs ($AllowedSender) which use system names rather than ip addresses. In order to check these ACLs, we need to do a DNS lookup. Especially in the case of UDP, such a lookup may actually case message loss and thus may be abused by an attacker to cause a certain degree of denial of service (what also points out that these types of ACLs are not really a good idea, even though requested by practice). In the light of this, I will now do something that sounds strange at first: I will always accept messages that require DNS lookups and enqueue these into the main queue and do the name resolution AND the final name-based ACL check only on the queue consumer part. Please note that it will be done BEFORE message content is parsed, so there is no chance that buffer overlow attacks can be carried out from non-authenticated hosts. The core idea is to move the lengthy, potentially message-loss causing code, away from the input thread. The only questionable effect I can currently see is that queue space is potentially taken up by messages which will immediately be discarded and should not be there in the first place. At the extreme end, that could lead to loss of valid messages. But on the other hand valid messages are more likely to be lost by the DNS name query overhead if I do the ACL check directly in the input thread. As such, I think my intended move is correct. Does anyone have an argument against the approach I am now taking? Thanks, Rainer From theinric at redhat.com Mon Nov 16 17:24:30 2009 From: theinric at redhat.com (Tomas Heinrich) Date: Mon, 16 Nov 2009 17:24:30 +0100 Subject: [rsyslog] support for arbitrary number of open files Message-ID: <4B017CBE.80904@redhat.com> Hi all, currently the total number of files (and tcp connections) that can be open simultaneously is limited by the select() system call. Ideally this would be changed to *poll(), but that can take some time. Until that happens, this patch[1] tries to remove the limitations of select() by enlarging the bit mask that is used for storing file descriptor information and redefining the macros that process it. This modification is inspired by Bind's use of select(). It is rather a workaround and may not be entirely portable. The actual changes to the code aren't big, but they are in many places so sufficient testing is needed. Allocating and freeing fd_sets in some frequently called functions may decrease performance. This can be dealt with but would require more pervasive changes. Thoughts? Thanks, Tomas [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited-select.patch From david at lang.hm Mon Nov 16 17:51:40 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 16 Nov 2009 08:51:40 -0800 (PST) Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 16 Nov 2009, Rainer Gerhards wrote: > Hi all, > > I am working again on moving the DNS name resolution outside of the input > thread of those sources where this is potentially time-consuming and > affecting message acceptance rates. As it turned out, currently imudp seems > to be the only case. > > While this is potentially easy to do, a problem is ACLs ($AllowedSender) > which use system names rather than ip addresses. In order to check these > ACLs, we need to do a DNS lookup. Especially in the case of UDP, such a > lookup may actually case message loss and thus may be abused by an attacker > to cause a certain degree of denial of service (what also points out that > these types of ACLs are not really a good idea, even though requested by > practice). > > In the light of this, I will now do something that sounds strange at first: I > will always accept messages that require DNS lookups and enqueue these into > the main queue and do the name resolution AND the final name-based ACL check > only on the queue consumer part. Please note that it will be done BEFORE > message content is parsed, so there is no chance that buffer overlow attacks > can be carried out from non-authenticated hosts. The core idea is to move the > lengthy, potentially message-loss causing code, away from the input thread. > The only questionable effect I can currently see is that queue space is > potentially taken up by messages which will immediately be discarded and > should not be there in the first place. At the extreme end, that could lead > to loss of valid messages. But on the other hand valid messages are more > likely to be lost by the DNS name query overhead if I do the ACL check > directly in the input thread. > > As such, I think my intended move is correct. Does anyone have an argument > against the approach I am now taking? personally I don't think that this sort of filtering belongs in rsyslog, it can be done at the OS level (with things like iptables), or rsyslog could use the tcpwrappers library. both cases would filter (by IP) prior to it hitting rsyslog in the first place. in addition, with UDP the source IP can be forged easily (rsyslog now contains this capability), so as a security measure it's questionable anyway. I agree that fewer messages will probably be lost by accepting them and checking later than by pausing to do the check initially. David Lang From rgerhards at hq.adiscon.com Mon Nov 16 17:54:28 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 16 Nov 2009 17:54:28 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> > personally I don't think that this sort of filtering belongs in > rsyslog, > it can be done at the OS level (with things like iptables), or rsyslog > could use the tcpwrappers library. both cases would filter (by IP) > prior > to it hitting rsyslog in the first place. > > in addition, with UDP the source IP can be forged easily (rsyslog now > contains this capability), so as a security measure it's questionable > anyway. I agree, but many people seem to use this functionality, and it was introduced some years ago by request. I do not feel comfortable with the idea of removing support for it. The newer protocols do not support ACLs in any case. Are there some more voices in regard to removing that functionality? Would make the implementation (and probably the throughput) a bit simpler/faster. > I agree that fewer messages will probably be lost by accepting them and > checking later than by pausing to do the check initially. Thanks for voicing this. Rainer From rgerhards at hq.adiscon.com Mon Nov 16 18:08:32 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 16 Nov 2009 18:08:32 +0100 Subject: [rsyslog] support for arbitrary number of open files References: <4B017CBE.80904@redhat.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103412@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > Sent: Monday, November 16, 2009 5:25 PM > To: rsyslog-users > Subject: [rsyslog] support for arbitrary number of open files > > Hi all, > > currently the total number of files (and tcp connections) that can be > open simultaneously is limited by the select() system call. Ideally > this > would be changed to *poll(), but that can take some time. For imudp, this change already happened (in the current devel). The others are on my radar. TCP is probably the most problematic, I need to redo the driver level. I wonder if I should split out gssapi while doing so, because that would simplify a lot of the calling conventions (at the price of some code duplication). For real-world scenarios, I think tcp is probably the only real issue, it is hard to believe that e.g. imuxsock will have an issue with that ;) > Until that happens, this patch[1] tries to remove the limitations of > select() by enlarging the bit mask that is used for storing file > descriptor information and redefining the macros that process it. > This modification is inspired by Bind's use of select(). > It is rather a workaround and may not be entirely portable. > > The actual changes to the code aren't big, but they are in many places > so sufficient testing is needed. Allocating and freeing fd_sets in > some > frequently called functions may decrease performance. This can be dealt > with but would require more pervasive changes. I need to have a close look at the patch. I consider it useful, but I agree that it must be very carefully checked. I've had a quick look and my understanding is that it is enabled by conditional compilation. That would ease many of my concerns, as we could disable it by default and only those that actually need it are at risk. > > Thoughts? I'd also appreciate feedback from other list members. Thanks, Rainer > > Thanks, > Tomas > > [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited- > select.patch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From aoz.syn at gmail.com Mon Nov 16 18:45:57 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 16 Nov 2009 10:45:57 -0700 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> Message-ID: <4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com> On Mon, Nov 16, 2009 at 09:54, Rainer Gerhards wrote: > I agree, but many people seem to use this functionality, and it was > introduced some years ago by request. I do not feel comfortable with the idea > of removing support for it. The newer protocols do not support ACLs in any > case. > > Are there some more voices in regard to removing that functionality? Would > make the implementation (and probably the throughput) a bit simpler/faster. I agree with david's assessment that "security" by this type of ACL is minimally effective. However, the functionality is occasionally useful for situations where management of the software is easier than management of the firewall (typically for business/operational constraints). I'd love to see it reimplemented as a modular part of the ephemeral "middle layer" along with other filtering and modification functionality. I know RanierScript is supposed to fill a lot of that void, but until it's implemented and proven performant my wishes still lie with a filter layer API. >> I agree that fewer messages will probably be lost by accepting them and >> checking later than by pausing to do the check initially. Ditto - although conceptually pure, front-end checking is probably too expensive for such a performance-critical component as the receiver. From rgerhards at hq.adiscon.com Mon Nov 16 20:56:38 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 16 Nov 2009 20:56:38 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> <4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of RB > Sent: Monday, November 16, 2009 6:46 PM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback request: ACLs, imudp and > accepting messages > > On Mon, Nov 16, 2009 at 09:54, Rainer Gerhards > wrote: > > I agree, but many people seem to use this functionality, and it was > > introduced some years ago by request. I do not feel > comfortable with the idea > > of removing support for it. The newer protocols do not > support ACLs in any > > case. > > > > Are there some more voices in regard to removing that > functionality? Would > > make the implementation (and probably the throughput) a bit > simpler/faster. > > I agree with david's assessment that "security" by this type of ACL is > minimally effective. >From a security POV, it may be more effective to remove that option, at least for UDP (people than know it is not secure, but neither is a similar firewall setting). But I fear all those broken configs, then without any protection at all. Maybe a thing for a new v5 default, but even then... > However, the functionality is occasionally > useful for situations where management of the software is easier than > management of the firewall (typically for business/operational > constraints). I'd love to see it reimplemented as a modular part of > the ephemeral "middle layer" along with other filtering and > modification functionality. I fail to see the interface you envision. Could you elaborate a bit? That would be most useful. I guess that a lot of this can now be done by the custom parsers, it may just need to be documented as a valid use case. > I know RanierScript is supposed to fill a > lot of that void, but until it's implemented and proven performant My thinking is moving a bit away from this "once size fits all" approach, primarily for performance reasons. It is quite expensive to start up a full scripting engine (with its VM), so native code loadable modules require more work to be done upfront, but are much faster. Rainer > my > wishes still lie with a filter layer API. > > >> I agree that fewer messages will probably be lost by > accepting them and > >> checking later than by pausing to do the check initially. > > Ditto - although conceptually pure, front-end checking is probably too > expensive for such a performance-critical component as the receiver. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From aoz.syn at gmail.com Mon Nov 16 21:28:54 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 16 Nov 2009 13:28:54 -0700 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> <4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com> Message-ID: <4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> On Mon, Nov 16, 2009 at 12:56, Rainer Gerhards wrote: > I fail to see the interface you envision. Could you elaborate a bit? ?That > would be most useful. Certainly. Some time ago, I expressed interest in writing a 'running checksum' module but had run into issues with a lack of state-maintenance in the output modules that helped my lack of time drop the issue. At the time, you'd hinted at a 'filter' module API that would (I presumed) fit somewhere between input and output and eventually address that use case. Such a layer could perform message modification, rate limiting, ACL enforcement, and so on. Perhaps this has already been addressed, since I've not kept as close of tabs on the development head as I'd like. Ideally, I think it would be a very interesting step to be able to 'stack' syslog modules to define the path a given set of messages take through the collector. Something akin to the Linux iptables concept, where a basic high-speed framework is laid out with a well-defined order and an API that enables administrators to elect how specific run-time modules interact within the stack. That's less of a request and more of a pipe dream, and is definitely influenced by the half network security hat I wear. From rgerhards at hq.adiscon.com Mon Nov 16 21:52:15 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 16 Nov 2009 21:52:15 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com><4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com> <4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103415@GRFEXC.intern.adiscon.com> Let me think about this, but it really sounds like custom parser modules could be moved into that direction with relative ease (at least for part of the functionality). Some info on them: http://www.rsyslog.com/doc-messageparser.html Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of RB > Sent: Monday, November 16, 2009 9:29 PM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback request: ACLs, imudp and > accepting messages > > On Mon, Nov 16, 2009 at 12:56, Rainer Gerhards > wrote: > > I fail to see the interface you envision. Could you > elaborate a bit? ?That > > would be most useful. > > Certainly. Some time ago, I expressed interest in writing a 'running > checksum' module but had run into issues with a lack of > state-maintenance in the output modules that helped my lack of time > drop the issue. At the time, you'd hinted at a 'filter' module API > that would (I presumed) fit somewhere between input and output and > eventually address that use case. Such a layer could perform message > modification, rate limiting, ACL enforcement, and so on. Perhaps this > has already been addressed, since I've not kept as close of tabs on > the development head as I'd like. > > Ideally, I think it would be a very interesting step to be able to > 'stack' syslog modules to define the path a given set of messages take > through the collector. Something akin to the Linux iptables concept, > where a basic high-speed framework is laid out with a well-defined > order and an API that enables administrators to elect how specific > run-time modules interact within the stack. That's less of a request > and more of a pipe dream, and is definitely influenced by the half > network security hat I wear. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Tue Nov 17 03:58:14 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 16 Nov 2009 18:58:14 -0800 (PST) Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages In-Reply-To: <4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> <4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com> <4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> Message-ID: On Mon, 16 Nov 2009, RB wrote: > On Mon, Nov 16, 2009 at 12:56, Rainer Gerhards wrote: >> I fail to see the interface you envision. Could you elaborate a bit? ?That >> would be most useful. > > Certainly. Some time ago, I expressed interest in writing a 'running > checksum' module but had run into issues with a lack of > state-maintenance in the output modules that helped my lack of time > drop the issue. At the time, you'd hinted at a 'filter' module API > that would (I presumed) fit somewhere between input and output and > eventually address that use case. Such a layer could perform message > modification, rate limiting, ACL enforcement, and so on. Perhaps this > has already been addressed, since I've not kept as close of tabs on > the development head as I'd like. > > Ideally, I think it would be a very interesting step to be able to > 'stack' syslog modules to define the path a given set of messages take > through the collector. Something akin to the Linux iptables concept, > where a basic high-speed framework is laid out with a well-defined > order and an API that enables administrators to elect how specific > run-time modules interact within the stack. That's less of a request > and more of a pipe dream, and is definitely influenced by the half > network security hat I wear. hmm, the recent ability to have secondary queues would allow this to fit in very nicely as a filter for what to do when moving messages between the queues. David Lang From philr at jaspers.co.nz Tue Nov 17 04:10:13 2009 From: philr at jaspers.co.nz (Phil Reilly) Date: Tue, 17 Nov 2009 16:10:13 +1300 Subject: [rsyslog] Alerting rules via a database In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> References: <4AF5DAA7.5010001@jaspers.co.nz> <4AF6816A.6050901@jaspers.co.nz> <9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> Message-ID: <4B021415.7080906@jaspers.co.nz> Any luck with the template? Or should I just roll my own. Cheers, Phil Rainer Gerhards wrote: > So what you are actually looking for is a system that can work with > dynamically changable alert definitions? As David said, there is no such > thing currently, but the best road to approach is is to write a custom output > plugin, that you pass each message to. That plugin can even decide if > messages should be discarded and not further processed. I envisioned such a > plugin, but had not yet time to write, for a similar use case. > > If you intend to write one AND contribute it to the project, I can help you > get started with the interface, would even be willing to create you a custom > skeleton that you can fill in your logic ;) > > HTH > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phil Reilly >> Sent: Sunday, November 08, 2009 9:30 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Alerting rules via a database >> >> david at lang.hm wrote: >> >>> On Sun, 8 Nov 2009, Phil Reilly wrote: >>> >>> >>> >>>> I attempting to allow for flexible rule matches on Syslogs >>>> >> from a web >> >>>> front end (rather than entires into the rsyslog config files) >>>> >>>> I want to get regexp filters from a db to alert upon >>>> >> messages. Not sure >> >>>> the best way to achieve this. I've so far though of. >>>> >>>> * Outputting to a pipe and runing it via an alerting script. >>>> * Having file watch the messages. >>>> * Recieving the messages then passing them to rsyslog (yuck) >>>> >>>> Can the rule engine allow for match rules outside of the config? is >>>> there an elegant way of doing this? >>>> >>>> >>> rsyslog doesn't give you this ability, but it's not really the best >>> approach to use for alerting anyway. >>> >>> what are you trying to achieve by having the alert definitions in a >>> database? there are several tools out there to do alerting >>> >> (SEC, Simple >> >>> Event Correlator) is one of the leading ones, but I'm not >>> >> aware of any of >> >>> them that use a database for their rulesets. >>> >>> I'm also scratching my head trying to figure out what the >>> >> advantage of >> >>> doing so would be. >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> Thanks David, >> >> We have a networked environment. We also have a web page that >> allows you >> to configure regexp to match certain syslog messages. These >> patterns are >> compiled and kept in a table. The current syslog process we >> use listens >> for udp. When it gets a syslog message, we examine the >> patterns (which >> are re-read upon addition or change) and pass them to an alertering >> process before writing the logs to disk. The existing system >> works well, >> but we now want to scale it over a few machines and I'm >> examining what >> syslog products out there cater for alerting. >> >> So a database will make configuring alerts far more dynamic than >> statically entering them into a config file. It will also allow for >> grouped views so different groups have the ability to have >> custom alerts >> based upon their own interpretation of syslog messages. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 17 07:28:25 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 07:28:25 +0100 Subject: [rsyslog] Alerting rules via a database Message-ID: <000901ca674f$38fd00a1$100013ac@intern.adiscon.com> Sorry, i have to admit it slipped my mind. Will create one this morning. rainer ----- Urspr?ngliche Nachricht ----- Von: "Phil Reilly" An: "rsyslog-users" Gesendet: 17.11.09 04:10 Betreff: Re: [rsyslog] Alerting rules via a database Any luck with the template? Or should I just roll my own. Cheers, Phil Rainer Gerhards wrote: > So what you are actually looking for is a system that can work with > dynamically changable alert definitions? As David said, there is no such > thing currently, but the best road to approach is is to write a custom output > plugin, that you pass each message to. That plugin can even decide if > messages should be discarded and not further processed. I envisioned such a > plugin, but had not yet time to write, for a similar use case. > > If you intend to write one AND contribute it to the project, I can help you > get started with the interface, would even be willing to create you a custom > skeleton that you can fill in your logic ;) > > HTH > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phil Reilly >> Sent: Sunday, November 08, 2009 9:30 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Alerting rules via a database >> >> david at lang.hm wrote: >> >>> On Sun, 8 Nov 2009, Phil Reilly wrote: >>> >>> >>> >>>> I attempting to allow for flexible rule matches on Syslogs >>>> >> from a web >> >>>> front end (rather than entires into the rsyslog config files) >>>> >>>> I want to get regexp filters from a db to alert upon >>>> >> messages. Not sure >> >>>> the best way to achieve this. I've so far though of. >>>> >>>> * Outputting to a pipe and runing it via an alerting script. >>>> * Having file watch the messages. >>>> * Recieving the messages then passing them to rsyslog (yuck) >>>> >>>> Can the rule engine allow for match rules outside of the config? is >>>> there an elegant way of doing this? >>>> >>>> >>> rsyslog doesn't give you this ability, but it's not really the best >>> approach to use for alerting anyway. >>> >>> what are you trying to achieve by having the alert definitions in a >>> database? there are several tools out there to do alerting >>> >> (SEC, Simple >> >>> Event Correlator) is one of the leading ones, but I'm not >>> >> aware of any of >> >>> them that use a database for their rulesets. >>> >>> I'm also scratching my head trying to figure out what the >>> >> advantage of >> >>> doing so would be. >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> Thanks David, >> >> We have a networked environment. We also have a web page that >> allows you >> to configure regexp to match certain syslog messages. These >> patterns are >> compiled and kept in a table. The current syslog process we >> use listens >> for udp. When it gets a syslog message, we examine the >> patterns (which >> are re-read upon addition or change) and pass them to an alertering >> process before writing the logs to disk. The existing system >> works well, >> but we now want to scale it over a few machines and I'm >> examining what >> syslog products out there cater for alerting. >> >> So a database will make configuring alerts far more dynamic than >> statically entering them into a config file. It will also allow for >> grouped views so different groups have the ability to have >> custom alerts >> based upon their own interpretation of syslog messages. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 17 08:15:22 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 08:15:22 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com><4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com><4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103417@GRFEXC.intern.adiscon.com> > > Ideally, I think it would be a very interesting step to be able to > > 'stack' syslog modules to define the path a given set of messages > take > > through the collector. Something akin to the Linux iptables concept, > > where a basic high-speed framework is laid out with a well-defined > > order and an API that enables administrators to elect how specific > > run-time modules interact within the stack. That's less of a request > > and more of a pipe dream, and is definitely influenced by the half > > network security hat I wear. > > hmm, the recent ability to have secondary queues would allow this to > fit > in very nicely as a filter for what to do when moving messages between > the > queues. I have thought more about this over the night. I think most of the capabilities are already present, just need to be a bit "re-labled". However, what is missing is functionality to use loadable modules as a filter condition. However, especially for complex filters, this sounds very valuable, both from a performance as well as from a capability POV. It also looks like it is not hard to add. I will definitely look into that direction. Thanks all for bringing up these thoughts, any further comments are of course appreciated as well ;) Rainer From rgerhards at hq.adiscon.com Tue Nov 17 08:49:28 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 08:49:28 +0100 Subject: [rsyslog] Alerting rules via a database References: <4AF5DAA7.5010001@jaspers.co.nz><4AF6816A.6050901@jaspers.co.nz><9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> <4B021415.7080906@jaspers.co.nz> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103418@GRFEXC.intern.adiscon.com> It is now present in the master branch. It resides in ./plugins/omdbalerting to enable it, use $ ./configure --enable-omdbalerting Obviously, the code now needs to be populated. I suggest that you create a simple sample with fixed constants, submit it to me, and then we can talk about how to obtain parameters from the config file. I hope this is useful. If you have any question (I guess you will have), please ask. But be sure to review module-template.h before beginning with any work. Looking at omtemplate.c is also a good idea. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Phil Reilly > Sent: Tuesday, November 17, 2009 4:10 AM > To: rsyslog-users > Subject: Re: [rsyslog] Alerting rules via a database > > Any luck with the template? > > Or should I just roll my own. > > Cheers, > > Phil > > Rainer Gerhards wrote: > > So what you are actually looking for is a system that can work with > > dynamically changable alert definitions? As David said, there is no > such > > thing currently, but the best road to approach is is to write a > custom output > > plugin, that you pass each message to. That plugin can even decide if > > messages should be discarded and not further processed. I envisioned > such a > > plugin, but had not yet time to write, for a similar use case. > > > > If you intend to write one AND contribute it to the project, I can > help you > > get started with the interface, would even be willing to create you a > custom > > skeleton that you can fill in your logic ;) > > > > HTH > > Rainer > > > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phil Reilly > >> Sent: Sunday, November 08, 2009 9:30 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Alerting rules via a database > >> > >> david at lang.hm wrote: > >> > >>> On Sun, 8 Nov 2009, Phil Reilly wrote: > >>> > >>> > >>> > >>>> I attempting to allow for flexible rule matches on Syslogs > >>>> > >> from a web > >> > >>>> front end (rather than entires into the rsyslog config files) > >>>> > >>>> I want to get regexp filters from a db to alert upon > >>>> > >> messages. Not sure > >> > >>>> the best way to achieve this. I've so far though of. > >>>> > >>>> * Outputting to a pipe and runing it via an alerting script. > >>>> * Having file watch the messages. > >>>> * Recieving the messages then passing them to rsyslog (yuck) > >>>> > >>>> Can the rule engine allow for match rules outside of the config? > is > >>>> there an elegant way of doing this? > >>>> > >>>> > >>> rsyslog doesn't give you this ability, but it's not really the best > >>> approach to use for alerting anyway. > >>> > >>> what are you trying to achieve by having the alert definitions in a > >>> database? there are several tools out there to do alerting > >>> > >> (SEC, Simple > >> > >>> Event Correlator) is one of the leading ones, but I'm not > >>> > >> aware of any of > >> > >>> them that use a database for their rulesets. > >>> > >>> I'm also scratching my head trying to figure out what the > >>> > >> advantage of > >> > >>> doing so would be. > >>> > >>> David Lang > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >>> > >> Thanks David, > >> > >> We have a networked environment. We also have a web page that > >> allows you > >> to configure regexp to match certain syslog messages. These > >> patterns are > >> compiled and kept in a table. The current syslog process we > >> use listens > >> for udp. When it gets a syslog message, we examine the > >> patterns (which > >> are re-read upon addition or change) and pass them to an alertering > >> process before writing the logs to disk. The existing system > >> works well, > >> but we now want to scale it over a few machines and I'm > >> examining what > >> syslog products out there cater for alerting. > >> > >> So a database will make configuring alerts far more dynamic than > >> statically entering them into a config file. It will also allow for > >> grouped views so different groups have the ability to have > >> custom alerts > >> based upon their own interpretation of syslog messages. > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 17 10:00:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 10:00:44 +0100 Subject: [rsyslog] support for arbitrary number of open files References: <4B017CBE.80904@redhat.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710341A@GRFEXC.intern.adiscon.com> Tomas, I am currently working on integration of this patch. What puzzles me is the call to "howmany()". I don't find any doc on it, nor an implementation inside the patch. Could you elaborate what it does and where it stems from? Also, I think there is a segfault in gss-misc, because the glbl interface is never aquired (the will result in a NULL-pointer dereference). I also need to change the glbl interface definitions, FdSetSize must always be present, else the interface is no longer well-defined. I will post the completely integrated patch when I am done. But feedback on howmany() would be most appreciated, because I currently do not know exactly how to handle it. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > Sent: Monday, November 16, 2009 5:25 PM > To: rsyslog-users > Subject: [rsyslog] support for arbitrary number of open files > > Hi all, > > currently the total number of files (and tcp connections) that can be > open simultaneously is limited by the select() system call. Ideally > this > would be changed to *poll(), but that can take some time. > > Until that happens, this patch[1] tries to remove the limitations of > select() by enlarging the bit mask that is used for storing file > descriptor information and redefining the macros that process it. > This modification is inspired by Bind's use of select(). > It is rather a workaround and may not be entirely portable. > > The actual changes to the code aren't big, but they are in many places > so sufficient testing is needed. Allocating and freeing fd_sets in > some > frequently called functions may decrease performance. This can be dealt > with but would require more pervasive changes. > > Thoughts? > > Thanks, > Tomas > > [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited- > select.patch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 17 10:49:16 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 10:49:16 +0100 Subject: [rsyslog] support for arbitrary number of open files References: <4B017CBE.80904@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA710341A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710341B@GRFEXC.intern.adiscon.com> OK, I have finally integrated the patch, I could work around my missing knowledge of howmany() [but still would appreciate some more info on it]. As of rsyslog policy, new features are never integrated into stable or beta builds. As such, it is available in the v4-devel and master branches. Note that I made a number of changes, you will probably like to re-check for your use case. I ran the testbench successfully in both modes and did some manual tests with the new functionality disabled. The v5-branch does NOT include the patch for imudp. As you said, it is a (good ;)) work-around and imudp already supports epoll(), so there is no need for this workaround. I plan to gradually remove that feature in v5 a I work towards supporting epoll in all inputs. The master branch will probably be released tomorrow, v4-devel as need arises (but it is available from git right now). Thanks again, I think this is a very useful addition. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, November 17, 2009 10:01 AM > To: rsyslog-users > Subject: Re: [rsyslog] support for arbitrary number of open files > > Tomas, > > I am currently working on integration of this patch. What puzzles me is > the > call to "howmany()". I don't find any doc on it, nor an implementation > inside > the patch. Could you elaborate what it does and where it stems from? > > Also, I think there is a segfault in gss-misc, because the glbl > interface is > never aquired (the will result in a NULL-pointer dereference). I also > need to > change the glbl interface definitions, FdSetSize must always be > present, else > the interface is no longer well-defined. I will post the completely > integrated patch when I am done. > > But feedback on howmany() would be most appreciated, because I > currently do > not know exactly how to handle it. > > Thanks, > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > > Sent: Monday, November 16, 2009 5:25 PM > > To: rsyslog-users > > Subject: [rsyslog] support for arbitrary number of open files > > > > Hi all, > > > > currently the total number of files (and tcp connections) that can be > > open simultaneously is limited by the select() system call. Ideally > > this > > would be changed to *poll(), but that can take some time. > > > > Until that happens, this patch[1] tries to remove the limitations of > > select() by enlarging the bit mask that is used for storing file > > descriptor information and redefining the macros that process it. > > This modification is inspired by Bind's use of select(). > > It is rather a workaround and may not be entirely portable. > > > > The actual changes to the code aren't big, but they are in many > places > > so sufficient testing is needed. Allocating and freeing fd_sets in > > some > > frequently called functions may decrease performance. This can be > dealt > > with but would require more pervasive changes. > > > > Thoughts? > > > > Thanks, > > Tomas > > > > [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited- > > select.patch > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 17 11:29:57 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 11:29:57 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com><4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com><4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103417@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710341E@GRFEXC.intern.adiscon.com> I have added some doc about what modules can do today: http://www.rsyslog.com/doc-rsyslog_conf_modules.html Note that message modification is possible, it was just not spelled out. The actual code also currently creates the false impression it is not possible. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, November 17, 2009 8:15 AM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback request: ACLs, imudp and accepting > messages > > > > Ideally, I think it would be a very interesting step to be able to > > > 'stack' syslog modules to define the path a given set of messages > > take > > > through the collector. Something akin to the Linux iptables > concept, > > > where a basic high-speed framework is laid out with a well-defined > > > order and an API that enables administrators to elect how specific > > > run-time modules interact within the stack. That's less of a > request > > > and more of a pipe dream, and is definitely influenced by the half > > > network security hat I wear. > > > > hmm, the recent ability to have secondary queues would allow this to > > fit > > in very nicely as a filter for what to do when moving messages > between > > the > > queues. > > I have thought more about this over the night. I think most of the > capabilities are already present, just need to be a bit "re-labled". > However, > what is missing is functionality to use loadable modules as a filter > condition. However, especially for complex filters, this sounds very > valuable, both from a performance as well as from a capability POV. It > also > looks like it is not hard to add. > > I will definitely look into that direction. Thanks all for bringing up > these > thoughts, any further comments are of course appreciated as well ;) > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Nov 18 04:13:06 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 17 Nov 2009 19:13:06 -0800 (PST) Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 13 Nov 2009, david at lang.hm wrote: > I've got this installed in production and it's working well so far. monday > morning (pacific time) should give it a good stress test. it's survived since friday without a problem on a couple of dozen machines (all my perimiter relay boxes plus my core logging server) looking good. David Lang > David Lang > > On Fri, 13 Nov 2009, Tom Bergfeld wrote: > >> Date: Fri, 13 Nov 2009 12:51:09 +0100 >> From: Tom Bergfeld >> Reply-To: rsyslog-users >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released >> >> Hi all, >> >> Today, we released a new v5-beta branch. It bases on the last v5-devel, >> plus >> has some bug fixes and some enhancements. The enhancements provide a >> slightly >> increased performance. This release is a very important milestone. Based on >> feedback from the v5-devel branches, it is in pretty good shape. Our aim is >> to have a quick beta phase this time (taking the problems with the current >> v5-stable) into account. To do so, we need your cooperation. Please try out >> this new version and report back any issues you may experience. It would >> also >> be good to know if you use it and do not run into any issues. >> >> See Changelog for more details. >> >> ChangeLog: >> >> http://www.rsyslog.com/Article427.phtml >> >> Download: >> >> http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-187.phtml >> >> >> As always, feedback is appreciated. >> >> Best regards, >> Tom Bergfeld >> -- >> Support >> ======= >> >> Improving rsyslog is costly, but you can help! We are looking for >> organizations that find rsyslog useful and wish to contribute back. You >> can >> contribute by reporting bugs, improve the software, or donate money or >> equipment. >> >> Commercial support contracts for rsyslog are available, and they help >> finance >> continued maintenance. Adiscon GmbH, a privately held German company, is >> currently funding rsyslog development. We are always looking for >> interesting >> development projects. For details on how to help, please see >> http://www.rsyslog.com/doc-how2help.html . >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > From rgerhards at hq.adiscon.com Wed Nov 18 09:23:24 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 18 Nov 2009 09:23:24 +0100 Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released References: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710342D@GRFEXC.intern.adiscon.com> This is excellent news, thanks for sharing it. Did anybody else also try it? Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, November 18, 2009 4:13 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 5.3.5 (v5-beta) released > > On Fri, 13 Nov 2009, david at lang.hm wrote: > > > I've got this installed in production and it's working well so far. > monday > > morning (pacific time) should give it a good stress test. > > it's survived since friday without a problem on a couple of dozen > machines > (all my perimiter relay boxes plus my core logging server) > > looking good. > > David Lang > > > David Lang > > > > On Fri, 13 Nov 2009, Tom Bergfeld wrote: > > > >> Date: Fri, 13 Nov 2009 12:51:09 +0100 > >> From: Tom Bergfeld > >> Reply-To: rsyslog-users > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released > >> > >> Hi all, > >> > >> Today, we released a new v5-beta branch. It bases on the last v5- > devel, > >> plus > >> has some bug fixes and some enhancements. The enhancements provide a > >> slightly > >> increased performance. This release is a very important milestone. > Based on > >> feedback from the v5-devel branches, it is in pretty good shape. Our > aim is > >> to have a quick beta phase this time (taking the problems with the > current > >> v5-stable) into account. To do so, we need your cooperation. Please > try out > >> this new version and report back any issues you may experience. It > would > >> also > >> be good to know if you use it and do not run into any issues. > >> > >> See Changelog for more details. > >> > >> ChangeLog: > >> > >> http://www.rsyslog.com/Article427.phtml > >> > >> Download: > >> > >> http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid- > 187.phtml > >> > >> > >> As always, feedback is appreciated. > >> > >> Best regards, > >> Tom Bergfeld > >> -- > >> Support > >> ======= > >> > >> Improving rsyslog is costly, but you can help! We are looking for > >> organizations that find rsyslog useful and wish to contribute back. > You > >> can > >> contribute by reporting bugs, improve the software, or donate money > or > >> equipment. > >> > >> Commercial support contracts for rsyslog are available, and they > help > >> finance > >> continued maintenance. Adiscon GmbH, a privately held German > company, is > >> currently funding rsyslog development. We are always looking for > >> interesting > >> development projects. For details on how to help, please see > >> http://www.rsyslog.com/doc-how2help.html . > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From martinmie at PartyGaming.com Wed Nov 18 10:52:21 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Wed, 18 Nov 2009 10:52:21 +0100 Subject: [rsyslog] rsyslog in uninterruptable sleep In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032D3@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA0892EE19@GIBSVWIN004X.partygaming.local> <9B6E2A8877C38245BFB15CC491A11DA71032D3@GRFEXC.intern.adiscon.com> Message-ID: <0B1B9163138571439EAF804646F3F6AA08B398B6@GIBSVWIN004X.partygaming.local> Hi Rainer, Since some weeks ago we run rsyslog 4.4.2 and unfortunately yesterday I stumbled on rsyslog with "D"-state process again... so I guess that the fix is somewhere else. Can you please confirm which rsyslog version provides the solution for this problem? Thanks, Martin > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: 27 October 2009 16:02 > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog in uninterruptable sleep > > I think we just recently had this on the mailing list or bug tracker and I > made a fix for it. I guess the fix is included in 4.4.2. > > HTH > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Martin Mielke > > Sent: Tuesday, October 27, 2009 11:57 AM > > To: rsyslog-users > > Subject: [rsyslog] rsyslog in uninterruptable sleep > > > > > > Hi, > > > > I just did some small changes to the rsyslog.conf and wanted to restart > > the process: > > > > # /etc/init.d/rsyslog restart > > Shutting down system logger: [FAILED] > > Starting system logger: Already running. > > [FAILED] > > > > # ps -elf | grep rsyslog > > 5 D root 2037 1 1 76 0 - 12040 sync_b Oct18 ? > > 03:32:10 rsyslogd -c 4 -x -f /etc/rsyslog.conf -i /var/run/syslogd.pid > > > > > > This is rsyslog 4.2.0 running on RHEL5 and it's not the first time it > > happens. > > > > > > Any ideas on why this might happened and how to solve this w/o > > rebooting > > the system?? > > > > > > Thanks, > > Martin > > > > > > This email and any attachments are confidential, and may be legally > > privileged and protected by copyright. If you are not the intended > > recipient dissemination or copying of this email is prohibited. If you > > have received this in error, please notify the sender by replying by > > email and then delete the email completely from your system. > > > > Any views or opinions are solely those of the sender. This > > communication is not intended to form a binding contract unless > > expressly indicated to the contrary and properly authorised. Any > > actions taken on the basis of this email are at the recipient's own > > risk. > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > yslog > http://www.rsyslog.com From tbergfeld at hq.adiscon.com Wed Nov 18 11:05:28 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 18 Nov 2009 11:05:28 +0100 Subject: [rsyslog] rsyslog 4.5.7 (v4-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103435@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.5.7, a member of the v4-beta branch. Most importantly, a so-called "On Demand Debug" mode was added, in which debug output can be generated only after the process has started, but not right from the beginning. This is assumed to be useful for hard-to-find bugs. Also improved the doc on the debug system. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article429.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-188.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From tbergfeld at hq.adiscon.com Wed Nov 18 11:14:30 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 18 Nov 2009 11:14:30 +0100 Subject: [rsyslog] rsyslog 5.5.0 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103437@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.5.0, a member of the devel branch. The 5.5.0 version begins a new development branch. The primary new functionality in that release is moving away reverse DNS resolution from the udp receiver's input thread to backend processing, what should reduce UDP message loss. ChangeLog: http://www.rsyslog.com/Article431.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-189.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From theinric at redhat.com Wed Nov 18 14:18:48 2009 From: theinric at redhat.com (Tomas Heinrich) Date: Wed, 18 Nov 2009 14:18:48 +0100 Subject: [rsyslog] support for arbitrary number of open files In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710341B@GRFEXC.intern.adiscon.com> References: <4B017CBE.80904@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA710341A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710341B@GRFEXC.intern.adiscon.com> Message-ID: <4B03F438.5030303@redhat.com> Rainer, thanks for integrating it and thanks for fixing the things I've overlooked. I've skimmed through the commit logs and the changes look good, I'll run some more tests with the new sources. Regarding howmany(), it is defined here in my environment: /usr/include/sys/param.h: # define howmany(x, y) (((x) + ((y) - 1)) / (y)) It computes the number of 'y's needed to store 'x'. Tomas On 11/17/2009 10:49 AM, Rainer Gerhards wrote: > OK, I have finally integrated the patch, I could work around my missing > knowledge of howmany() [but still would appreciate some more info on it]. > > As of rsyslog policy, new features are never integrated into stable or beta > builds. As such, it is available in the v4-devel and master branches. Note > that I made a number of changes, you will probably like to re-check for your > use case. I ran the testbench successfully in both modes and did some manual > tests with the new functionality disabled. > > The v5-branch does NOT include the patch for imudp. As you said, it is a > (good ;)) work-around and imudp already supports epoll(), so there is no need > for this workaround. I plan to gradually remove that feature in v5 a I work > towards supporting epoll in all inputs. > > The master branch will probably be released tomorrow, v4-devel as need arises > (but it is available from git right now). > > Thanks again, I think this is a very useful addition. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Tuesday, November 17, 2009 10:01 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] support for arbitrary number of open files >> >> Tomas, >> >> I am currently working on integration of this patch. What puzzles me is >> the >> call to "howmany()". I don't find any doc on it, nor an implementation >> inside >> the patch. Could you elaborate what it does and where it stems from? >> >> Also, I think there is a segfault in gss-misc, because the glbl >> interface is >> never aquired (the will result in a NULL-pointer dereference). I also >> need to >> change the glbl interface definitions, FdSetSize must always be >> present, else >> the interface is no longer well-defined. I will post the completely >> integrated patch when I am done. >> >> But feedback on howmany() would be most appreciated, because I >> currently do >> not know exactly how to handle it. >> >> Thanks, >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich >>> Sent: Monday, November 16, 2009 5:25 PM >>> To: rsyslog-users >>> Subject: [rsyslog] support for arbitrary number of open files >>> >>> Hi all, >>> >>> currently the total number of files (and tcp connections) that can be >>> open simultaneously is limited by the select() system call. Ideally >>> this >>> would be changed to *poll(), but that can take some time. >>> >>> Until that happens, this patch[1] tries to remove the limitations of >>> select() by enlarging the bit mask that is used for storing file >>> descriptor information and redefining the macros that process it. >>> This modification is inspired by Bind's use of select(). >>> It is rather a workaround and may not be entirely portable. >>> >>> The actual changes to the code aren't big, but they are in many >> places >>> so sufficient testing is needed. Allocating and freeing fd_sets in >>> some >>> frequently called functions may decrease performance. This can be >> dealt >>> with but would require more pervasive changes. >>> >>> Thoughts? >>> >>> Thanks, >>> Tomas >>> >>> [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited- >>> select.patch >>> From rgerhards at hq.adiscon.com Wed Nov 18 14:35:33 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 18 Nov 2009 14:35:33 +0100 Subject: [rsyslog] support for arbitrary number of open files References: <4B017CBE.80904@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA710341A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710341B@GRFEXC.intern.adiscon.com> <4B03F438.5030303@redhat.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710343E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > Sent: Wednesday, November 18, 2009 2:19 PM > To: rsyslog-users > Subject: Re: [rsyslog] support for arbitrary number of open files > > Rainer, > > thanks for integrating it and thanks for fixing the things I've > overlooked. > > I've skimmed through the commit logs and the changes look good, I'll > run > some more tests with the new sources. excellent - pls let me know if you see some issues. As a side-note, I've also used this as a reminder to finally look at epoll() for the plain tcp driver, probably the most prominent case, also performance wise. I hope I am able continue work on it, so we should have it in the not so distant future (in v5). > > Regarding howmany(), it is defined here in my environment: > /usr/include/sys/param.h: > # define howmany(x, y) (((x) + ((y) - 1)) / (y)) > It computes the number of 'y's needed to store 'x'. > Ahhh! I should have done a grep ;) But somehow did not expect it there... Thanks, Rainer > Tomas > > On 11/17/2009 10:49 AM, Rainer Gerhards wrote: > > OK, I have finally integrated the patch, I could work around my > missing > > knowledge of howmany() [but still would appreciate some more info on > it]. > > > > As of rsyslog policy, new features are never integrated into stable > or beta > > builds. As such, it is available in the v4-devel and master branches. > Note > > that I made a number of changes, you will probably like to re-check > for your > > use case. I ran the testbench successfully in both modes and did some > manual > > tests with the new functionality disabled. > > > > The v5-branch does NOT include the patch for imudp. As you said, it > is a > > (good ;)) work-around and imudp already supports epoll(), so there is > no need > > for this workaround. I plan to gradually remove that feature in v5 a > I work > > towards supporting epoll in all inputs. > > > > The master branch will probably be released tomorrow, v4-devel as > need arises > > (but it is available from git right now). > > > > Thanks again, I think this is a very useful addition. > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >> Sent: Tuesday, November 17, 2009 10:01 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] support for arbitrary number of open files > >> > >> Tomas, > >> > >> I am currently working on integration of this patch. What puzzles me > is > >> the > >> call to "howmany()". I don't find any doc on it, nor an > implementation > >> inside > >> the patch. Could you elaborate what it does and where it stems from? > >> > >> Also, I think there is a segfault in gss-misc, because the glbl > >> interface is > >> never aquired (the will result in a NULL-pointer dereference). I > also > >> need to > >> change the glbl interface definitions, FdSetSize must always be > >> present, else > >> the interface is no longer well-defined. I will post the completely > >> integrated patch when I am done. > >> > >> But feedback on howmany() would be most appreciated, because I > >> currently do > >> not know exactly how to handle it. > >> > >> Thanks, > >> Rainer > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > >>> Sent: Monday, November 16, 2009 5:25 PM > >>> To: rsyslog-users > >>> Subject: [rsyslog] support for arbitrary number of open files > >>> > >>> Hi all, > >>> > >>> currently the total number of files (and tcp connections) that can > be > >>> open simultaneously is limited by the select() system call. Ideally > >>> this > >>> would be changed to *poll(), but that can take some time. > >>> > >>> Until that happens, this patch[1] tries to remove the limitations > of > >>> select() by enlarging the bit mask that is used for storing file > >>> descriptor information and redefining the macros that process it. > >>> This modification is inspired by Bind's use of select(). > >>> It is rather a workaround and may not be entirely portable. > >>> > >>> The actual changes to the code aren't big, but they are in many > >> places > >>> so sufficient testing is needed. Allocating and freeing fd_sets in > >>> some > >>> frequently called functions may decrease performance. This can be > >> dealt > >>> with but would require more pervasive changes. > >>> > >>> Thoughts? > >>> > >>> Thanks, > >>> Tomas > >>> > >>> [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited- > >>> select.patch > >>> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Nov 19 10:58:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 19 Nov 2009 10:58:53 +0100 Subject: [rsyslog] clarification on priorities for rsyslog work Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710344B@GRFEXC.intern.adiscon.com> Hi all, as we currently have a lot of requests (both bugs and features), I would like to make *my* decision process a bit more transparent. So I have blogged about how I usually assign priorities: http://blog.gerhards.net/2009/11/priorities-for-rsyslog-work.html I hope this is useful. Rainer From mrdemeanour at jackpot.uk.net Thu Nov 19 19:19:02 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Thu, 19 Nov 2009 18:19:02 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <4AF2F9CD.9000402@jackpot.uk.net> References: <4AF2F9CD.9000402@jackpot.uk.net> Message-ID: <4B058C16.7010008@jackpot.uk.net> Mr. Demeanour wrote: > Hi, > > I'm running a central rsyslog server with a couple of remote WAN > (internet) clients and several remote LAN clients. Traffic is low - > of the order of 10,000 messages per day. Internet clients communicate > with the server using gnutls. LAN clients are currently using UDP. > The server writes client logs to mysql, and also writes messages of > local origin to disk. Further to this: I have been running 4.5.6 for about a week now, *without* gnutls enabled. No leaks. This evening I re-enabled gnutls, and almost immediately noted excessive memory usage, *and* 99% cpu. It seems that the high CPU usage occurs with hosts outside my local network; it may be that there is some misconfiguration of NAT that is behind that problem. I note that leaks are possible with the versions of gnutls shipping with Debian: http://permalink.gmane.org/gmane.network.gnutls.general/1465 That document describes a leak that would be expected to arise during connection setup, but not per message. I guess a dubious connection (e.g. resulting from misconfigured NAT) might result in repeated setup attempts, and so in leaks *and* cpu spiking. -- Jack. From mrdemeanour at jackpot.uk.net Thu Nov 19 19:50:16 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Thu, 19 Nov 2009 18:50:16 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <4B058C16.7010008@jackpot.uk.net> References: <4AF2F9CD.9000402@jackpot.uk.net> <4B058C16.7010008@jackpot.uk.net> Message-ID: <4B059368.2070701@jackpot.uk.net> Mr. Demeanour wrote: > > It seems that the high CPU usage occurs with hosts outside my local > network; it may be that there is some misconfiguration of NAT that is > behind that problem. Hmmm. What I meant is that the problem was only observed with gnutls *clients* outside my local network; the rsyslog server is inside. Sorry for being unclear. -- Jack. From rgerhards at hq.adiscon.com Thu Nov 19 21:30:22 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 19 Nov 2009 21:30:22 +0100 Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing References: <003001ca6092$792833d0$6b789b70$@com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> Hi, Sorry, I seem to have overlooked that message. But the mailing list processor has stripped the attachment. Could you please re-send to my personal mail address. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Jonathan Bond-Caron > Sent: Sunday, November 08, 2009 5:43 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing > > Hi all, > > > > Attached is a patch for 3 things: > > > > a) Check that TCP connection is still alive when using TLS > > > > b) Improve TAG parsing so that it ends when it sees a > tab or escape > control character. > > > > c) Added configuration directive: escapecontrolcharactertab > > In rsyslog.conf, you can set: > > $escapeControlCharactersOnReceive on > > $escapeControlCharacterTab off > > > > This avoids having a tab being escaped since it is after a viewable > character J > > I'd prefer this being the default behavior but I've left > $escapeControlCharacterTab on as default in case people > expect tabs to be > escaped. > > > > This is my first GIT patch, hope it works well ;) > > > > From david at lang.hm Thu Nov 19 21:37:32 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 19 Nov 2009 12:37:32 -0800 (PST) Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> References: <003001ca6092$792833d0$6b789b70$@com> <9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> Message-ID: for what it's worth, I think that b and c are very useful when dealing with strange data sources, and a is probably a bugfix. David Lang On Thu, 19 Nov 2009, Rainer Gerhards wrote: > > Sorry, I seem to have overlooked that message. But the mailing list processor > has stripped the attachment. Could you please re-send to my personal mail > address. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >> Jonathan Bond-Caron >> Sent: Sunday, November 08, 2009 5:43 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing >> >> Hi all, >> >> >> >> Attached is a patch for 3 things: >> >> >> >> a) Check that TCP connection is still alive when using TLS >> >> >> >> b) Improve TAG parsing so that it ends when it sees a >> tab or escape >> control character. >> >> >> >> c) Added configuration directive: escapecontrolcharactertab >> >> In rsyslog.conf, you can set: >> >> $escapeControlCharactersOnReceive on >> >> $escapeControlCharacterTab off >> >> >> >> This avoids having a tab being escaped since it is after a viewable >> character J >> >> I'd prefer this being the default behavior but I've left >> $escapeControlCharacterTab on as default in case people >> expect tabs to be >> escaped. >> >> >> >> This is my first GIT patch, hope it works well ;) >> >> >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Nov 20 11:03:59 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 20 Nov 2009 11:03:59 +0100 Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing References: <003001ca6092$792833d0$6b789b70$@com><9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710346E@GRFEXC.intern.adiscon.com> I, too, think this is useful. Will look at it as soon as I get the patch. The b) and c) parts may be a bit harder to integrate if they are not for the newest devel branch, as I rewrote the parser part of rsyslog. But in any case, it is worth trying to merge it in. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, November 19, 2009 9:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] [PATCH] TLS connection check + syslog TAG > parsing > > for what it's worth, I think that b and c are very useful when dealing > with strange data sources, and a is probably a bugfix. > > David Lang > > On Thu, 19 Nov 2009, Rainer Gerhards wrote: > > > > > Sorry, I seem to have overlooked that message. But the mailing list > processor > > has stripped the attachment. Could you please re-send to my personal > mail > > address. > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >> Jonathan Bond-Caron > >> Sent: Sunday, November 08, 2009 5:43 PM > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing > >> > >> Hi all, > >> > >> > >> > >> Attached is a patch for 3 things: > >> > >> > >> > >> a) Check that TCP connection is still alive when using TLS > >> > >> > >> > >> b) Improve TAG parsing so that it ends when it sees a > >> tab or escape > >> control character. > >> > >> > >> > >> c) Added configuration directive: escapecontrolcharactertab > >> > >> In rsyslog.conf, you can set: > >> > >> $escapeControlCharactersOnReceive on > >> > >> $escapeControlCharacterTab off > >> > >> > >> > >> This avoids having a tab being escaped since it is after a viewable > >> character J > >> > >> I'd prefer this being the default behavior but I've left > >> $escapeControlCharacterTab on as default in case people > >> expect tabs to be > >> escaped. > >> > >> > >> > >> This is my first GIT patch, hope it works well ;) > >> > >> > >> > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mrdemeanour at jackpot.uk.net Fri Nov 20 17:10:24 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 20 Nov 2009 16:10:24 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <4B058C16.7010008@jackpot.uk.net> References: <4AF2F9CD.9000402@jackpot.uk.net> <4B058C16.7010008@jackpot.uk.net> Message-ID: <4B06BF70.6070900@jackpot.uk.net> Mr. Demeanour wrote: > Mr. Demeanour wrote: >> Hi, >> >> I'm running a central rsyslog server with a couple of remote WAN >> (internet) clients and several remote LAN clients. Traffic is low - >> of the order of 10,000 messages per day. Internet clients >> communicate with the server using gnutls. LAN clients are currently >> using UDP. The server writes client logs to mysql, and also writes >> messages of local origin to disk. > > Further to this: > > I have been running 4.5.6 for about a week now, *without* gnutls > enabled. No leaks. > > This evening I re-enabled gnutls, and almost immediately noted > excessive memory usage, *and* 99% cpu. > > It seems that the high CPU usage occurs with hosts outside my local > network; it may be that there is some misconfiguration of NAT that is > behind that problem. Not NAT. It seems that I had set up the server certificate with an incorrect CN. I guess the client was trying repeatedly to make a connection that was doomed to fail every time. That would explain the CPU spike. If there is also a memory leak in the gnutls server code concerning connection setup, that would explain the memory consumption also. Perhaps rsyslog should give up trying to connect to a remote server, or at least back off, if the error it encounters is of a kind that most likely requires human intervention? Such would generally be the case if a certificate is invalid. -- Jack. From mrdemeanour at jackpot.uk.net Fri Nov 20 17:11:37 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 20 Nov 2009 16:11:37 +0000 Subject: [rsyslog] Possible bug: REs in templates Message-ID: <4B06BFB9.8080100@jackpot.uk.net> The following template produces one character fewer than expected. %HOSTNAME:R,ERE,1,FIELD,0:([^\.]+)--end% If, for example, the HOSTNAME string is 192.168.1.1, then the output is "19". -- Jack. From rgerhards at hq.adiscon.com Sun Nov 22 12:42:20 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 22 Nov 2009 12:42:20 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net> <4B058C16.7010008@jackpot.uk.net> <4B06BF70.6070900@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710348E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Friday, November 20, 2009 5:10 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Mr. Demeanour wrote: > > Mr. Demeanour wrote: > >> Hi, > >> > >> I'm running a central rsyslog server with a couple of remote WAN > >> (internet) clients and several remote LAN clients. Traffic is low - > >> of the order of 10,000 messages per day. Internet clients > >> communicate with the server using gnutls. LAN clients are currently > >> using UDP. The server writes client logs to mysql, and also writes > >> messages of local origin to disk. > > > > Further to this: > > > > I have been running 4.5.6 for about a week now, *without* gnutls > > enabled. No leaks. > > > > This evening I re-enabled gnutls, and almost immediately noted > > excessive memory usage, *and* 99% cpu. > > > > It seems that the high CPU usage occurs with hosts outside my local > > network; it may be that there is some misconfiguration of > NAT that is > > behind that problem. > > Not NAT. It seems that I had set up the server certificate with an > incorrect CN. > > I guess the client was trying repeatedly to make a connection that was > doomed to fail every time. That would explain the CPU spike. > If there is > also a memory leak in the gnutls server code concerning connection > setup, that would explain the memory consumption also. > > Perhaps rsyslog should give up trying to connect to a remote > server, or > at least back off, if the error it encounters is of a kind that most > likely requires human intervention? Such would generally be > the case if > a certificate is invalid. The situation is a bit complex in such cases. There, a shortcoming in RFC5425 "works together" with a shortcoming of GnuTLS. The end result is that I am unable to detect a problem occurs. This happens: RFC5425 does not specifiy app-level acknowledgements. Consequently, there is no way to convey an error condition back to the client. GnuTLS does not permit to check the certificate *BEFORE* accepting a connection. Consequently, the connection must be accepted, then the certificate checked and then terminated if the cert if invalid. However, to the client this looks like a successfully established connection (beause it was connected) that just quickly went down (but the client does not know an error state as there is no app-layer vehicle to provide it back). This also disables the usual rsyslog logic to pause failed connection attempts - because the connection succeeds in the first place. The only solution I currently see is either not to use RFC5425 or use something else than GnuTLS (for example, openssl permits to check the certificate BEFORE accepting the connection). An other approach is to modify GnuTLS so that it provides an ability to do the check before accepting the certificate. I looked into that option, but it requires far too much effort to me. So the right thing to do would probably write an additional TLS driver for e.g. openssl. While this is actually on the todo list, I have to admit that I do not have time to do so at the moment, nor do I see a spot in the near future where I could tackle that beast. I still have the hope that we will receive a contributed driver for NSS, which would most probably also solve this issue. Or wait for GnuTLS to evolve... I guess you get the picture ;) Rainer From xkubina at fi.muni.cz Mon Nov 23 16:56:14 2009 From: xkubina at fi.muni.cz (Tomas Kubina) Date: Mon, 23 Nov 2009 16:56:14 +0100 Subject: [rsyslog] imgssapi configuration Message-ID: <4B0AB09E.9030005@fi.muni.cz> Hi, I am trying to set up rsyslog server with GSSAPI auth. I have used the doc that is here http://www.rsyslog.com/doc-gssapi.html but without successful end. The issue is that when I want to start rsyslog server it hangs-up during starting procedure. Here are some details: I have version 5.2.0 and this is server's conf file: === $ModLoad immark # provides --MARK-- message capability $ModLoad imuxsock # provides support for local system logging (e.g. via logger) $ModLoad imklog # kernel logging (formerly provided by rklogd) #Global directives $ActionFileDefaultTemplate RSYSLOG_FileFormat <(rules for storing syslog messages)> $ModLoad imgssapi $InputGSSServerPermitPlainTCP off $InputGSSServerRun 10514 === client config is simple: === $ModLoad omgssapi *.info : omgssapi:server.example.com:10514 === Here is a part of debug output: ============================ 9766.638881000:2b51af4b1ac0: cfline: '$ModLoad imgssapi' 9766.638893000:2b51af4b1ac0: Requested to load module 'imgssapi' 9766.638905000:2b51af4b1ac0: loading module '/usr/local/lib/rsyslog/imgssapi.so' 9766.640185000:2b51af4b1ac0: caller requested object 'tcps_sess', not found (iRet -3003) 9766.640207000:2b51af4b1ac0: Requested to load module 'lmtcpsrv' 9766.640219000:2b51af4b1ac0: loading module '/usr/local/lib/rsyslog/lmtcpsrv.so' 9766.640292000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 80: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[1/2b51af4b1ac0] 9766.640309000:2b51af4b1ac0: caller requested object 'netstrm', not found (iRet -3003) 9766.640322000:2b51af4b1ac0: Requested to load module 'lmnetstrms' 9766.640333000:2b51af4b1ac0: loading module '/usr/local/lib/rsyslog/lmnetstrms.so' 9766.640394000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 80: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[1/2b51af4b1ac0] 9766.640409000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640424000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 80: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[1/2b51af4b1ac0] 9766.640436000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640450000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 80: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[1/2b51af4b1ac0] 9766.640461000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640477000:2b51af4b1ac0: module of type 2 being loaded. 9766.640488000:2b51af4b1ac0: entry point 'isCompatibleWithFeature' not present in module 9766.640500000:2b51af4b1ac0: source file tcps_sess.c requested reference for module 'lmnetstrms', reference count now 1 9766.640512000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640528000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640541000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640553000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640573000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640587000:2b51af4b1ac0: source file tcpsrv.c requested reference for module 'lmnet', reference count now 4 9766.640599000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640611000:2b51af4b1ac0: source file tcpsrv.c requested reference for module 'lmnetstrms', reference count now 2 9766.640624000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640637000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640651000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640665000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640678000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640691000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640705000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640717000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640737000:2b51af4b1ac0: module of type 2 being loaded. 9766.640748000:2b51af4b1ac0: entry point 'isCompatibleWithFeature' not present in module 9766.640759000:2b51af4b1ac0: source file imgssapi.c requested reference for module 'lmtcpsrv', reference count now 1 9766.640771000:2b51af4b1ac0: source file imgssapi.c requested reference for module 'lmtcpsrv', reference count now 2 9766.640784000:2b51af4b1ac0: caller requested object 'gssutil', not found (iRet -3003) 9766.640794000:2b51af4b1ac0: Requested to load module 'lmgssutil' 9766.640805000:2b51af4b1ac0: loading module '/usr/local/lib/rsyslog/lmgssutil.so' 9766.640880000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 100: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[1/2b51af4b1ac0] 9766.640897000:2b51af4b1ac0: module of type 2 being loaded. 9766.640908000:2b51af4b1ac0: entry point 'isCompatibleWithFeature' not present in module 9766.640919000:2b51af4b1ac0: source file imgssapi.c requested reference for module 'lmgssutil', reference count now 1 9766.640933000:2b51af4b1ac0: source file imgssapi.c requested reference for module 'lmnetstrms', reference count now 3 9766.640946000:2b51af4b1ac0: source file imgssapi.c requested reference for module 'lmnet', reference count now 5 9766.640969000:2b51af4b1ac0: module of type 0 being loaded. 9766.640985000:2b51af4b1ac0: cfline: '$InputGSSServerPermitPlainTCP off' 9766.641000000:2b51af4b1ac0: cfline: '$InputGSSServerRun 10514' 9766.641027000:2b51af4b1ac0: Signal 11 (SIGSEGV) occured, execution must be terminated. 9766.641147000:2b51af4b1ac0: 9766.641175000:2b51af4b1ac0: Recorded Call Order for Thread '2b51af4b1ac0': 9766.641186000:2b51af4b1ac0: 0: syslogd.c:3097:realMain: 9766.641196000:2b51af4b1ac0: 1: syslogd.c:2749:mainThread: 9766.641206000:2b51af4b1ac0: 2: syslogd.c:2185:init: 9766.641219000:2b51af4b1ac0: 3: conf.c:410:processConfFile: 9766.641229000:2b51af4b1ac0: 4: conf.c:1179:cfline: 9766.641238000:2b51af4b1ac0: 5: conf.c:359:cfsysline: 9766.641249000:2b51af4b1ac0: 6: cfsysline.c:902:processCfSysLineCommand: 9766.641259000:2b51af4b1ac0: 7: cfsysline.c:673:cslchCallHdlr: 9766.641269000:2b51af4b1ac0: 8: cfsysline.c:501:doGetWord: 9766.641281000:2b51af4b1ac0: 9: imgssapi.c:310:addGSSListener: 9766.641292000:2b51af4b1ac0: 10: tcpsrv.c:135:configureTCPListen: 9766.641301000:2b51af4b1ac0: 11: tcpsrv.c:102:addNewLstnPort: 9766.641311000:2b51af4b1ac0: maximum number of nested calls for this thread: 26. 9766.641324000:2b51af4b1ac0: NOTE: not all calls may have been recorded, code does not currently guarantee that! 9766.641333000:2b51af4b1ac0: Mutex log for all known mutex operations: 9766.641343000:2b51af4b1ac0: If the call trace is empty, you may want to ./configure --enable-rtinst 9766.641353000:2b51af4b1ac0: To submit bug reports, visit http://www.rsyslog.com/bugs 9766.641363000:2b51af4b1ac0: To submit bug reports, visit http://www.rsyslog.com/bugs Aborted === I am a newbie, any idea? Thanks, Tomas Kubina From josesan311 at yahoo.com Thu Nov 26 05:23:11 2009 From: josesan311 at yahoo.com (Jose Sanchez) Date: Wed, 25 Nov 2009 20:23:11 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog Message-ID: <948839.16588.qm@web35605.mail.mud.yahoo.com> Hello, I've been using classic syslog for centralizing apache access logs from one server to a remote syslog server, the thing is syslog adds some nasty tags before the lines in the access logs and I cant get them off, ie: "Nov 25 21:25:37 server1 logger:" I would like to know if rsyslog has the option to filter this kind of stuff, I just want to have the logs sent to the syslog server exactly like I was saving them in a local access.log file. Thanks in advance. From rgerhards at hq.adiscon.com Thu Nov 26 09:11:01 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 26 Nov 2009 09:11:01 +0100 Subject: [rsyslog] filter logger tags from syslog References: <948839.16588.qm@web35605.mail.mud.yahoo.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034AC@GRFEXC.intern.adiscon.com> I don't have the specifics at hand, but you can surely do that. I would even assume that if you just write the %msg% property to file, the infomation should look good. If not, you need to fiddle a bit with the property replacer. The basic idea is to use $template line,"%msg%\n" *.* /path/to/log;line HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jose Sanchez > Sent: Thursday, November 26, 2009 5:23 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] filter logger tags from syslog > > Hello, > > I've been using classic syslog for centralizing apache access > logs from one server to a remote syslog server, the thing is > syslog adds some nasty tags before the lines in the access > logs and I cant get them off, ie: > > "Nov 25 21:25:37 server1 logger:" > > I would like to know if rsyslog has the option to filter this > kind of stuff, I just want to have the logs sent to the > syslog server exactly like I was saving them in a local > access.log file. > > Thanks in advance. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu Nov 26 09:21:44 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 26 Nov 2009 00:21:44 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <948839.16588.qm@web35605.mail.mud.yahoo.com> References: <948839.16588.qm@web35605.mail.mud.yahoo.com> Message-ID: On Wed, 25 Nov 2009, Jose Sanchez wrote: > Hello, > > I've been using classic syslog for centralizing apache access logs from one server to a remote syslog server, the thing is syslog adds some nasty tags before the lines in the access logs and I cant get them off, ie: > > "Nov 25 21:25:37 server1 logger:" > > I would like to know if rsyslog has the option to filter this kind of stuff, I just want to have the logs sent to the syslog server exactly like I was saving them in a local access.log file. > > Thanks in advance. 'logger:' is added by the logger program that apache is using to send the logs to syslog. a properly formatted syslog message will include a timestamp and what server it came from (note that the apache logs do _not_ tell you what virtual server the log comes from, it usually uses a different file for each log, so when you mix them into syslog you won't be able to tell them apart) so you have three basic options 1. let logger do it's default thing and then use a formatting command to strip off the 'syslogie' parts to get back to the apache default in the file 2. leave the 'syslogie' parts in when you write it to a file and have your analysis tool strip them out 3. reformat the apache log message so that you put useful information in the 'syslogie' parts of the message. you can move the timestamp to the beginning (you can do this with or without the timezone, the format obviously differs slightly) you can put the name of the virtual host in the server field you can replace 'logger:' with something like apache[80]: or apache[443]: I am going to be setting up something along the lines of #3 in the next few weeks. I figure I will also want to tinker with other things in the log message. there are items that apache can log, but does not log by default (I believe that how long it took to process the request is one of these), and also since syslog defaults to limiting log messages to 1-2K (depending on your impementation), there are some fields that I would want to move late in the message so that if they get very long they don't cause other fields to be lost due to truncation (URL and referrer fields can be several K long by themselves) David Lang From vanderleeden at logicunited.com Thu Nov 26 13:57:54 2009 From: vanderleeden at logicunited.com (Rudolf van der Leeden) Date: Thu, 26 Nov 2009 13:57:54 +0100 Subject: [rsyslog] Indefinite number of retries with ompgsql Message-ID: Hi, I'm currently testing ompgsql and running into a problem with the number of pgsql retries if the INSERT statement is wrong. My rsyslog.conf setting: $ModLoad ompgsql # load the output driver for PostgreSQL $ModLoad imudp # network reception $MaxMessageSize 8192 # default 2048; use 32768 to support large message sizes for IHE $UDPServerRun 514 # start a udp server at port 514 $WorkDirectory /var/spool/rsyslog/work # default location for work (spool) files $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName dbq # set file name, also enables disk mode $ActionResumeRetryCount 0 $template applogFormat,"INSERT INTO applog (message, host, priority, priotext, tag, logtime) VALUES \ (E'%msg%', '%hostname%', '%syslogpriority%', '%syslogpriority-text %', '%programname%', '%timegenerated:::date-pgsql%')",stdsql if $programname contains 'production' then :ompgsql:loghost,syslog,syslog,syslog;applogFormat I changed $ActionResumeRetryCount from -1 to 1 and then to 0, but always the same result: indefinite retries of rsyslogd to get the INSERT done. This setup is basically running OK. Only if a binary 0 (\000) is contained in the message Postgres returns: ERROR: invalid byte sequence for encoding "UTF8": 0x00 What am I doing wrong? Any hints are welcome. Thanks for your help, Rudolf. Rudolf VanderLeeden Scoreloop GmbH vanderleeden at scoreloop.com From rgerhards at hq.adiscon.com Thu Nov 26 17:47:26 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 26 Nov 2009 17:47:26 +0100 Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710346E@GRFEXC.intern.adiscon.com> References: <003001ca6092$792833d0$6b789b70$@com> <9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710346E@GRFEXC.intern.adiscon.com> Message-ID: <1259254046.16407.1.camel@rgf11> Hi all, I have now integrated the TLS part of the patch into all relevant versions. It is available in public git now. I will try to look into the parsing part tomorrow. Thanks again to Jonathan for sharing it! Rainer On Fri, 2009-11-20 at 11:03 +0100, Rainer Gerhards wrote: > I, too, think this is useful. Will look at it as soon as I get the patch. The > b) and c) parts may be a bit harder to integrate if they are not for the > newest devel branch, as I rewrote the parser part of rsyslog. But in any > case, it is worth trying to merge it in. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Thursday, November 19, 2009 9:38 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] [PATCH] TLS connection check + syslog TAG > > parsing > > > > for what it's worth, I think that b and c are very useful when dealing > > with strange data sources, and a is probably a bugfix. > > > > David Lang > > > > On Thu, 19 Nov 2009, Rainer Gerhards wrote: > > > > > > > > Sorry, I seem to have overlooked that message. But the mailing list > > processor > > > has stripped the attachment. Could you please re-send to my personal > > mail > > > address. > > > > > > Rainer > > > > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com > > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > >> Jonathan Bond-Caron > > >> Sent: Sunday, November 08, 2009 5:43 PM > > >> To: rsyslog at lists.adiscon.com > > >> Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing > > >> > > >> Hi all, > > >> > > >> > > >> > > >> Attached is a patch for 3 things: > > >> > > >> > > >> > > >> a) Check that TCP connection is still alive when using TLS > > >> > > >> > > >> > > >> b) Improve TAG parsing so that it ends when it sees a > > >> tab or escape > > >> control character. > > >> > > >> > > >> > > >> c) Added configuration directive: escapecontrolcharactertab > > >> > > >> In rsyslog.conf, you can set: > > >> > > >> $escapeControlCharactersOnReceive on > > >> > > >> $escapeControlCharacterTab off > > >> > > >> > > >> > > >> This avoids having a tab being escaped since it is after a viewable > > >> character J > > >> > > >> I'd prefer this being the default behavior but I've left > > >> $escapeControlCharacterTab on as default in case people > > >> expect tabs to be > > >> escaped. > > >> > > >> > > >> > > >> This is my first GIT patch, hope it works well ;) > > >> > > >> > > >> > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From josesan311 at yahoo.com Fri Nov 27 01:25:22 2009 From: josesan311 at yahoo.com (Jose Sanchez) Date: Thu, 26 Nov 2009 16:25:22 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: Message-ID: <886471.94508.qm@web35606.mail.mud.yahoo.com> Hello, I appreciate all the responses. Im not sure how can I can acconplish options 1) and 2) automatically. For option 3) the thing is I need "combined" log type so I cannot reform this. Im trying to centralize an access_log file from one server to the rsyslog server and I need to completely remove the tags I mentioned on my previous post. I have also tried using a perl script mentioned at the botton of this email, but it salso arriving with a tag, "apache_syslog:" as showed below, "apache_syslog: XXX.XXX.XXX.XXX - - [26/Nov/2009:18:23:02 -0600] \"GET /.." Basically, this log will be parsed by awstats which is pretty much stricted with the log format so that's why I need a clean log sent from the apache server to the rsyslog server. Thank you very much for all the help. Below is the Perl script: #!/usr/local/bin/perl # script: apache-access-logger use Sys::Syslog; $SERVER_NAME = shift || ''; $PRIORITY = 'info'; $FACILITY = 'local1'; Sys::Syslog::setlogsock('unix'); openlog ($SERVER_NAME,'ndelay', $FACILITY); while (<>) { chomp; syslog($PRIORITY,$_); } closelog; --- On Thu, 11/26/09, david at lang.hm wrote: > From: david at lang.hm > Subject: Re: [rsyslog] filter logger tags from syslog > To: "rsyslog-users" > Date: Thursday, November 26, 2009, 2:21 AM > On Wed, 25 Nov 2009, Jose Sanchez > wrote: > > > Hello, > > > > I've been using classic syslog for centralizing apache > access logs from one server to a remote syslog server, the > thing is syslog adds some nasty tags before the lines in the > access logs and I cant get them off, ie: > > > > "Nov 25 21:25:37 server1 logger:" > > > > I would like to know if rsyslog has the option to > filter this kind of stuff, I just want to have the logs sent > to the syslog server exactly like I was saving them in a > local access.log file. > > > > Thanks in advance. > > 'logger:' is added by the logger program that apache is > using to send the > logs to syslog. > > a properly formatted syslog message will include a > timestamp and what > server it came from (note that the apache logs do _not_ > tell you what > virtual server the log comes from, it usually uses a > different file for > each log, so when you mix them into syslog you won't be > able to tell them > apart) > > so you have three basic options > > 1. let logger do it's default thing and then use a > formatting command to > strip off the 'syslogie' parts to get back to the apache > default in the > file > > 2. leave the 'syslogie' parts in when you write it to a > file and have your > analysis tool strip them out > > 3. reformat the apache log message so that you put useful > information in > the 'syslogie' parts of the message. > > you can move the timestamp to the beginning (you can do > this with or > without the timezone, the format obviously differs > slightly) > > you can put the name of the virtual host in the server > field > > you can replace 'logger:' with something like apache[80]: > or apache[443]: > > I am going to be setting up something along the lines of #3 > in the next > few weeks. I figure I will also want to tinker with other > things in the > log message. there are items that apache can log, but does > not log by > default (I believe that how long it took to process the > request is one of > these), and also since syslog defaults to limiting log > messages to 1-2K > (depending on your impementation), there are some fields > that I would want > to move late in the message so that if they get very long > they don't cause > other fields to be lost due to truncation (URL and referrer > fields can be > several K long by themselves) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From josesan311 at yahoo.com Fri Nov 27 01:26:24 2009 From: josesan311 at yahoo.com (Jose Sanchez) Date: Thu, 26 Nov 2009 16:26:24 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71034AC@GRFEXC.intern.adiscon.com> Message-ID: <840944.77123.qm@web35608.mail.mud.yahoo.com> Hello, Will this work if Im using rsyslog just at the server side? (client is using syslog) Many thanks. --- On Thu, 11/26/09, Rainer Gerhards wrote: > From: Rainer Gerhards > Subject: Re: [rsyslog] filter logger tags from syslog > To: "rsyslog-users" > Date: Thursday, November 26, 2009, 2:11 AM > I don't have the specifics at hand, > but you can surely do that. I would even > assume that if you just write the %msg% property to file, > the infomation > should look good. If not, you need to fiddle a bit with the > property > replacer. > > The basic idea is to use > > $template line,"%msg%\n" > *.* /path/to/log;line > > HTH > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com > > > [mailto:rsyslog-bounces at lists.adiscon.com] > On Behalf Of Jose Sanchez > > Sent: Thursday, November 26, 2009 5:23 AM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] filter logger tags from syslog > > > > Hello, > > > > I've been using classic syslog for centralizing apache > access > > logs from one server to a remote syslog server, the > thing is > > syslog adds some nasty tags before the lines in the > access > > logs and I cant get them off, ie: > > > > "Nov 25 21:25:37 server1 logger:" > > > > I would like to know if rsyslog has the option to > filter this > > kind of stuff, I just want to have the logs sent to > the > > syslog server exactly like I was saving them in a > local > > access.log file. > > > > Thanks in advance. > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Nov 27 01:30:52 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 26 Nov 2009 16:30:52 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <840944.77123.qm@web35608.mail.mud.yahoo.com> References: <840944.77123.qm@web35608.mail.mud.yahoo.com> Message-ID: On Thu, 26 Nov 2009, Jose Sanchez wrote: > Will this work if Im using rsyslog just at the server side? (client is using syslog) yes, although depending on exactly which syslog you may need to tinker with the template. try what Rainer suggested below and see what you get out. David Lang > Many thanks. > > --- On Thu, 11/26/09, Rainer Gerhards wrote: > >> From: Rainer Gerhards >> Subject: Re: [rsyslog] filter logger tags from syslog >> To: "rsyslog-users" >> Date: Thursday, November 26, 2009, 2:11 AM >> I don't have the specifics at hand, >> but you can surely do that. I would even >> assume that if you just write the %msg% property to file, >> the infomation >> should look good. If not, you need to fiddle a bit with the >> property >> replacer. >> >> The basic idea is to use >> >> $template line,"%msg%\n" >> *.* /path/to/log;line >> >> HTH >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com >> >>> [mailto:rsyslog-bounces at lists.adiscon.com] >> On Behalf Of Jose Sanchez >>> Sent: Thursday, November 26, 2009 5:23 AM >>> To: rsyslog at lists.adiscon.com >>> Subject: [rsyslog] filter logger tags from syslog >>> >>> Hello, >>> >>> I've been using classic syslog for centralizing apache >> access >>> logs from one server to a remote syslog server, the >> thing is >>> syslog adds some nasty tags before the lines in the >> access >>> logs and I cant get them off, ie: >>> >>> "Nov 25 21:25:37 server1 logger:" >>> >>> I would like to know if rsyslog has the option to >> filter this >>> kind of stuff, I just want to have the logs sent to >> the >>> syslog server exactly like I was saving them in a >> local >>> access.log file. >>> >>> Thanks in advance. >>> >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Nov 27 01:38:03 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 26 Nov 2009 16:38:03 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <886471.94508.qm@web35606.mail.mud.yahoo.com> References: <886471.94508.qm@web35606.mail.mud.yahoo.com> Message-ID: On Thu, 26 Nov 2009, Jose Sanchez wrote: > Hello, > > I appreciate all the responses. > Im not sure how can I can acconplish options 1) and 2) automatically. > For option 3) the thing is I need "combined" log type so I cannot reform this. > > Im trying to centralize an access_log file from one server to the rsyslog server and I need to completely remove the tags I mentioned on my previous post. > I have also tried using a perl script mentioned at the botton of this email, but it salso arriving with a tag, "apache_syslog:" as showed below, > > "apache_syslog: XXX.XXX.XXX.XXX - - [26/Nov/2009:18:23:02 -0600] \"GET /.." > > Basically, this log will be parsed by awstats which is pretty much stricted with the log format so that's why I need a clean log sent from the apache server to the rsyslog server. don't forget that you need to filter these messages into a seperate file, otherwise you will have your apache combined log messages mixed with other syslog messages (which will really confuse awstats) option 1 is what Rainer suggested option 2 is to run the log through another step before awstats runs, something along the lines of cut -c 16- file |cut -f 3- -d ' ' |awstats the first cut removes the timestamp (always 15 characters, but with a variable number of spaces in it), the second cut removes the servername and the syslog tag ('logger:' in your first example) David Lang > Thank you very much for all the help. > > Below is the Perl script: > > #!/usr/local/bin/perl > # script: apache-access-logger > > use Sys::Syslog; > $SERVER_NAME = shift || ''; > > $PRIORITY = 'info'; > $FACILITY = 'local1'; > > Sys::Syslog::setlogsock('unix'); > openlog ($SERVER_NAME,'ndelay', $FACILITY); > > while (<>) { > chomp; > syslog($PRIORITY,$_); > } > closelog; > > --- On Thu, 11/26/09, david at lang.hm wrote: > >> From: david at lang.hm >> Subject: Re: [rsyslog] filter logger tags from syslog >> To: "rsyslog-users" >> Date: Thursday, November 26, 2009, 2:21 AM >> On Wed, 25 Nov 2009, Jose Sanchez >> wrote: >> >>> Hello, >>> >>> I've been using classic syslog for centralizing apache >> access logs from one server to a remote syslog server, the >> thing is syslog adds some nasty tags before the lines in the >> access logs and I cant get them off, ie: >>> >>> "Nov 25 21:25:37 server1 logger:" >>> >>> I would like to know if rsyslog has the option to >> filter this kind of stuff, I just want to have the logs sent >> to the syslog server exactly like I was saving them in a >> local access.log file. >>> >>> Thanks in advance. >> >> 'logger:' is added by the logger program that apache is >> using to send the >> logs to syslog. >> >> a properly formatted syslog message will include a >> timestamp and what >> server it came from (note that the apache logs do _not_ >> tell you what >> virtual server the log comes from, it usually uses a >> different file for >> each log, so when you mix them into syslog you won't be >> able to tell them >> apart) >> >> so you have three basic options >> >> 1. let logger do it's default thing and then use a >> formatting command to >> strip off the 'syslogie' parts to get back to the apache >> default in the >> file >> >> 2. leave the 'syslogie' parts in when you write it to a >> file and have your >> analysis tool strip them out >> >> 3. reformat the apache log message so that you put useful >> information in >> the 'syslogie' parts of the message. >> >> you can move the timestamp to the beginning (you can do >> this with or >> without the timezone, the format obviously differs >> slightly) >> >> you can put the name of the virtual host in the server >> field >> >> you can replace 'logger:' with something like apache[80]: >> or apache[443]: >> >> I am going to be setting up something along the lines of #3 >> in the next >> few weeks. I figure I will also want to tinker with other >> things in the >> log message. there are items that apache can log, but does >> not log by >> default (I believe that how long it took to process the >> request is one of >> these), and also since syslog defaults to limiting log >> messages to 1-2K >> (depending on your impementation), there are some fields >> that I would want >> to move late in the message so that if they get very long >> they don't cause >> other fields to be lost due to truncation (URL and referrer >> fields can be >> several K long by themselves) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From josesan311 at yahoo.com Fri Nov 27 05:13:00 2009 From: josesan311 at yahoo.com (Jose Sanchez) Date: Thu, 26 Nov 2009 20:13:00 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: Message-ID: <925071.2214.qm@web35604.mail.mud.yahoo.com> Hello, Thanks again for the great response. It's actually working! rsyslog is removing the "logger:" thing and all the nasty stuff from it automatically, how come? Is it because we are not adding any tag in the template? Im still not understanding how rsyslog removes the logger thing. Ok, Im now getting the proper output and like David said Im now getting issues with filtering the apache logs with all the rsyslog messages. I've tried to use the following filter but for some reason is not working and Im not 100% if this is the best solution to use, This is how I had set it up, $template line,"%msg%\n" if $msg contains 'GET' then /var/log/apache.test.log;line *.* /var/log/test.log;line Not sure if Im on the right path, any help will be appreciated. I have also tried the "if" sentence without specifying the template name. Many Thanks. --- On Thu, 11/26/09, david at lang.hm wrote: > From: david at lang.hm > Subject: Re: [rsyslog] filter logger tags from syslog > To: "rsyslog-users" > Date: Thursday, November 26, 2009, 6:38 PM > On Thu, 26 Nov 2009, Jose Sanchez > wrote: > > > Hello, > > > > I appreciate all the responses. > > Im not sure how can I can acconplish options 1) and 2) > automatically. > > For option 3) the thing is I need "combined" log type > so I cannot reform this. > > > > Im trying to centralize an access_log file from one > server to the rsyslog server and I need to completely remove > the tags I mentioned on my previous post. > > I have also tried using a perl script mentioned at the > botton of this email, but it salso arriving with a tag, > "apache_syslog:" as showed below, > > > > "apache_syslog: XXX.XXX.XXX.XXX - - > [26/Nov/2009:18:23:02 -0600] \"GET /.." > > > > Basically, this log will be parsed by awstats which is > pretty much stricted with the log format so that's why I > need a clean log sent from the apache server to the rsyslog > server. > > don't forget that you need to filter these messages into a > seperate file, > otherwise you will have your apache combined log messages > mixed with other > syslog messages (which will really confuse awstats) > > option 1 is what Rainer suggested > > option 2 is to run the log through another step before > awstats runs, > something along the lines of > > cut -c 16- file |cut -f 3- -d ' ' |awstats > > the first cut removes the timestamp (always 15 characters, > but with a > variable number of spaces in it), the second cut removes > the servername > and the syslog tag ('logger:' in your first example) > > David Lang > > > Thank you very much for all the help. > > > > Below is the Perl script: > > > > #!/usr/local/bin/perl > > # script: apache-access-logger > > > > use Sys::Syslog; > > $SERVER_NAME = shift || ''; > > > > $PRIORITY = 'info'; > > $FACILITY = 'local1'; > > > > Sys::Syslog::setlogsock('unix'); > > openlog ($SERVER_NAME,'ndelay', $FACILITY); > > > > while (<>) { > >? chomp; > >? syslog($PRIORITY,$_); > > } > > closelog; > > > > --- On Thu, 11/26/09, david at lang.hm > wrote: > > > >> From: david at lang.hm > >> Subject: Re: [rsyslog] filter logger tags from > syslog > >> To: "rsyslog-users" > >> Date: Thursday, November 26, 2009, 2:21 AM > >> On Wed, 25 Nov 2009, Jose Sanchez > >> wrote: > >> > >>> Hello, > >>> > >>> I've been using classic syslog for > centralizing apache > >> access logs from one server to a remote syslog > server, the > >> thing is syslog adds some nasty tags before the > lines in the > >> access logs and I cant get them off, ie: > >>> > >>> "Nov 25 21:25:37 server1 logger:" > >>> > >>> I would like to know if rsyslog has the option > to > >> filter this kind of stuff, I just want to have the > logs sent > >> to the syslog server exactly like I was saving > them in a > >> local access.log file. > >>> > >>> Thanks in advance. > >> > >> 'logger:' is added by the logger program that > apache is > >> using to send the > >> logs to syslog. > >> > >> a properly formatted syslog message will include > a > >> timestamp and what > >> server it came from (note that the apache logs do > _not_ > >> tell you what > >> virtual server the log comes from, it usually uses > a > >> different file for > >> each log, so when you mix them into syslog you > won't be > >> able to tell them > >> apart) > >> > >> so you have three basic options > >> > >> 1. let logger do it's default thing and then use > a > >> formatting command to > >> strip off the 'syslogie' parts to get back to the > apache > >> default in the > >> file > >> > >> 2. leave the 'syslogie' parts in when you write it > to a > >> file and have your > >> analysis tool strip them out > >> > >> 3. reformat the apache log message so that you put > useful > >> information in > >> the 'syslogie' parts of the message. > >> > >> you can move the timestamp to the beginning (you > can do > >> this with or > >> without the timezone, the format obviously > differs > >> slightly) > >> > >> you can put the name of the virtual host in the > server > >> field > >> > >> you can replace 'logger:' with something like > apache[80]: > >> or apache[443]: > >> > >> I am going to be setting up something along the > lines of #3 > >> in the next > >> few weeks. I figure I will also want to tinker > with other > >> things in the > >> log message. there are items that apache can log, > but does > >> not log by > >> default (I believe that how long it took to > process the > >> request is one of > >> these), and also since syslog defaults to limiting > log > >> messages to 1-2K > >> (depending on your impementation), there are some > fields > >> that I would want > >> to move late in the message so that if they get > very long > >> they don't cause > >> other fields to be lost due to truncation (URL and > referrer > >> fields can be > >> several K long by themselves) > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Nov 27 09:41:13 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 27 Nov 2009 09:41:13 +0100 Subject: [rsyslog] filter logger tags from syslog References: <925071.2214.qm@web35604.mail.mud.yahoo.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034C3@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Jose Sanchez > Sent: Friday, November 27, 2009 5:13 AM > To: rsyslog-users > Subject: Re: [rsyslog] filter logger tags from syslog > > Hello, > > Thanks again for the great response. > It's actually working! rsyslog is removing the "logger:" thing and all > the nasty stuff from it automatically, how come? you really need to read through the property doc. I know the doc is not great, but with some persistence you'll find anything you need (and doc contributions are always very welcome!). Understanding properties is key to getting complex configurations right. > Is it because we are > not adding any tag in the template? Yes, because the template specifies which properties are used (and how). > Im still not understanding how > rsyslog removes the logger thing. > > Ok, Im now getting the proper output and like David said Im now getting > issues with filtering the apache logs with all the rsyslog messages. > I've tried to use the following filter but for some reason is not > working and Im not 100% if this is the best solution to use, > > This is how I had set it up, > > $template line,"%msg%\n" > if $msg contains 'GET' then /var/log/apache.test.log;line > *.* /var/log/test.log;line If you want to split the messages, you need to discard the one that you just wrote: if $msg contains 'GET' then /var/log/apache.test.log;line & ~ Helpful read is http://www.rsyslog.com/doc-queues_analogy.html HTH Rainer > > Not sure if Im on the right path, any help will be appreciated. > I have also tried the "if" sentence without specifying the template > name. > > Many Thanks. > > --- On Thu, 11/26/09, david at lang.hm wrote: > > > From: david at lang.hm > > Subject: Re: [rsyslog] filter logger tags from syslog > > To: "rsyslog-users" > > Date: Thursday, November 26, 2009, 6:38 PM > > On Thu, 26 Nov 2009, Jose Sanchez > > wrote: > > > > > Hello, > > > > > > I appreciate all the responses. > > > Im not sure how can I can acconplish options 1) and 2) > > automatically. > > > For option 3) the thing is I need "combined" log type > > so I cannot reform this. > > > > > > Im trying to centralize an access_log file from one > > server to the rsyslog server and I need to completely remove > > the tags I mentioned on my previous post. > > > I have also tried using a perl script mentioned at the > > botton of this email, but it salso arriving with a tag, > > "apache_syslog:" as showed below, > > > > > > "apache_syslog: XXX.XXX.XXX.XXX - - > > [26/Nov/2009:18:23:02 -0600] \"GET /.." > > > > > > Basically, this log will be parsed by awstats which is > > pretty much stricted with the log format so that's why I > > need a clean log sent from the apache server to the rsyslog > > server. > > > > don't forget that you need to filter these messages into a > > seperate file, > > otherwise you will have your apache combined log messages > > mixed with other > > syslog messages (which will really confuse awstats) > > > > option 1 is what Rainer suggested > > > > option 2 is to run the log through another step before > > awstats runs, > > something along the lines of > > > > cut -c 16- file |cut -f 3- -d ' ' |awstats > > > > the first cut removes the timestamp (always 15 characters, > > but with a > > variable number of spaces in it), the second cut removes > > the servername > > and the syslog tag ('logger:' in your first example) > > > > David Lang > > > > > Thank you very much for all the help. > > > > > > Below is the Perl script: > > > > > > #!/usr/local/bin/perl > > > # script: apache-access-logger > > > > > > use Sys::Syslog; > > > $SERVER_NAME = shift || ''; > > > > > > $PRIORITY = 'info'; > > > $FACILITY = 'local1'; > > > > > > Sys::Syslog::setlogsock('unix'); > > > openlog ($SERVER_NAME,'ndelay', $FACILITY); > > > > > > while (<>) { > > >? chomp; > > >? syslog($PRIORITY,$_); > > > } > > > closelog; > > > > > > --- On Thu, 11/26/09, david at lang.hm > > wrote: > > > > > >> From: david at lang.hm > > >> Subject: Re: [rsyslog] filter logger tags from > > syslog > > >> To: "rsyslog-users" > > >> Date: Thursday, November 26, 2009, 2:21 AM > > >> On Wed, 25 Nov 2009, Jose Sanchez > > >> wrote: > > >> > > >>> Hello, > > >>> > > >>> I've been using classic syslog for > > centralizing apache > > >> access logs from one server to a remote syslog > > server, the > > >> thing is syslog adds some nasty tags before the > > lines in the > > >> access logs and I cant get them off, ie: > > >>> > > >>> "Nov 25 21:25:37 server1 logger:" > > >>> > > >>> I would like to know if rsyslog has the option > > to > > >> filter this kind of stuff, I just want to have the > > logs sent > > >> to the syslog server exactly like I was saving > > them in a > > >> local access.log file. > > >>> > > >>> Thanks in advance. > > >> > > >> 'logger:' is added by the logger program that > > apache is > > >> using to send the > > >> logs to syslog. > > >> > > >> a properly formatted syslog message will include > > a > > >> timestamp and what > > >> server it came from (note that the apache logs do > > _not_ > > >> tell you what > > >> virtual server the log comes from, it usually uses > > a > > >> different file for > > >> each log, so when you mix them into syslog you > > won't be > > >> able to tell them > > >> apart) > > >> > > >> so you have three basic options > > >> > > >> 1. let logger do it's default thing and then use > > a > > >> formatting command to > > >> strip off the 'syslogie' parts to get back to the > > apache > > >> default in the > > >> file > > >> > > >> 2. leave the 'syslogie' parts in when you write it > > to a > > >> file and have your > > >> analysis tool strip them out > > >> > > >> 3. reformat the apache log message so that you put > > useful > > >> information in > > >> the 'syslogie' parts of the message. > > >> > > >> you can move the timestamp to the beginning (you > > can do > > >> this with or > > >> without the timezone, the format obviously > > differs > > >> slightly) > > >> > > >> you can put the name of the virtual host in the > > server > > >> field > > >> > > >> you can replace 'logger:' with something like > > apache[80]: > > >> or apache[443]: > > >> > > >> I am going to be setting up something along the > > lines of #3 > > >> in the next > > >> few weeks. I figure I will also want to tinker > > with other > > >> things in the > > >> log message. there are items that apache can log, > > but does > > >> not log by > > >> default (I believe that how long it took to > > process the > > >> request is one of > > >> these), and also since syslog defaults to limiting > > log > > >> messages to 1-2K > > >> (depending on your impementation), there are some > > fields > > >> that I would want > > >> to move late in the message so that if they get > > very long > > >> they don't cause > > >> other fields to be lost due to truncation (URL and > > referrer > > >> fields can be > > >> several K long by themselves) > > >> > > >> David Lang > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Nov 27 09:49:35 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 27 Nov 2009 00:49:35 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <925071.2214.qm@web35604.mail.mud.yahoo.com> References: <925071.2214.qm@web35604.mail.mud.yahoo.com> Message-ID: On Thu, 26 Nov 2009, Jose Sanchez wrote: > Hello, > > Thanks again for the great response. > It's actually working! rsyslog is removing the "logger:" thing and all > the nasty stuff from it automatically, how come? Is it because we are > not adding any tag in the template? Im still not understanding how > rsyslog removes the logger thing. > > Ok, Im now getting the proper output and like David said Im now getting > issues with filtering the apache logs with all the rsyslog messages. > I've tried to use the following filter but for some reason is not > working and Im not 100% if this is the best solution to use, > > This is how I had set it up, > > $template line,"%msg%\n" > if $msg contains 'GET' then /var/log/apache.test.log;line > *.* /var/log/test.log;line > > Not sure if Im on the right path, any help will be appreciated. > I have also tried the "if" sentence without specifying the template name. when rsyslog receives a message it parses it. the message over the wire hasn't changed (still has the timestamp, servername, logger: etc), but rsyslog puts those parts into the seperate variables and puts what is left of the message into the %msg% variable. so when you change the template from the default of %timestamp% %hostname% %syslogtag%%msg% to just %msg% the log file has just the part you care about. now for the filtering. you could do :%programname, isequal, "logger" /var/log/apache.test.log;line (as I understand it, this format is a bit more efficiant for rsyslog than the equivalent of if $programname eq "logger" then /var/log/apache.test.log;line ) I would actually suggest that you use the perl script that you posted, and filter for programname equal to "apache_syslog", filtering on just 'logger' means that you can't use logger for anything else. you don't want to just filter for 'GET' as there are a bunch of log files that won't have GET in them David Lang > Many Thanks. > > --- On Thu, 11/26/09, david at lang.hm wrote: > >> From: david at lang.hm >> Subject: Re: [rsyslog] filter logger tags from syslog >> To: "rsyslog-users" >> Date: Thursday, November 26, 2009, 6:38 PM >> On Thu, 26 Nov 2009, Jose Sanchez >> wrote: >> >>> Hello, >>> >>> I appreciate all the responses. >>> Im not sure how can I can acconplish options 1) and 2) >> automatically. >>> For option 3) the thing is I need "combined" log type >> so I cannot reform this. >>> >>> Im trying to centralize an access_log file from one >> server to the rsyslog server and I need to completely remove >> the tags I mentioned on my previous post. >>> I have also tried using a perl script mentioned at the >> botton of this email, but it salso arriving with a tag, >> "apache_syslog:" as showed below, >>> >>> "apache_syslog: XXX.XXX.XXX.XXX - - >> [26/Nov/2009:18:23:02 -0600] \"GET /.." >>> >>> Basically, this log will be parsed by awstats which is >> pretty much stricted with the log format so that's why I >> need a clean log sent from the apache server to the rsyslog >> server. >> >> don't forget that you need to filter these messages into a >> seperate file, >> otherwise you will have your apache combined log messages >> mixed with other >> syslog messages (which will really confuse awstats) >> >> option 1 is what Rainer suggested >> >> option 2 is to run the log through another step before >> awstats runs, >> something along the lines of >> >> cut -c 16- file |cut -f 3- -d ' ' |awstats >> >> the first cut removes the timestamp (always 15 characters, >> but with a >> variable number of spaces in it), the second cut removes >> the servername >> and the syslog tag ('logger:' in your first example) >> >> David Lang >> >>> Thank you very much for all the help. >>> >>> Below is the Perl script: >>> >>> #!/usr/local/bin/perl >>> # script: apache-access-logger >>> >>> use Sys::Syslog; >>> $SERVER_NAME = shift || ''; >>> >>> $PRIORITY = 'info'; >>> $FACILITY = 'local1'; >>> >>> Sys::Syslog::setlogsock('unix'); >>> openlog ($SERVER_NAME,'ndelay', $FACILITY); >>> >>> while (<>) { >>> ? chomp; >>> ? syslog($PRIORITY,$_); >>> } >>> closelog; >>> >>> --- On Thu, 11/26/09, david at lang.hm >> wrote: >>> >>>> From: david at lang.hm >>>> Subject: Re: [rsyslog] filter logger tags from >> syslog >>>> To: "rsyslog-users" >>>> Date: Thursday, November 26, 2009, 2:21 AM >>>> On Wed, 25 Nov 2009, Jose Sanchez >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> I've been using classic syslog for >> centralizing apache >>>> access logs from one server to a remote syslog >> server, the >>>> thing is syslog adds some nasty tags before the >> lines in the >>>> access logs and I cant get them off, ie: >>>>> >>>>> "Nov 25 21:25:37 server1 logger:" >>>>> >>>>> I would like to know if rsyslog has the option >> to >>>> filter this kind of stuff, I just want to have the >> logs sent >>>> to the syslog server exactly like I was saving >> them in a >>>> local access.log file. >>>>> >>>>> Thanks in advance. >>>> >>>> 'logger:' is added by the logger program that >> apache is >>>> using to send the >>>> logs to syslog. >>>> >>>> a properly formatted syslog message will include >> a >>>> timestamp and what >>>> server it came from (note that the apache logs do >> _not_ >>>> tell you what >>>> virtual server the log comes from, it usually uses >> a >>>> different file for >>>> each log, so when you mix them into syslog you >> won't be >>>> able to tell them >>>> apart) >>>> >>>> so you have three basic options >>>> >>>> 1. let logger do it's default thing and then use >> a >>>> formatting command to >>>> strip off the 'syslogie' parts to get back to the >> apache >>>> default in the >>>> file >>>> >>>> 2. leave the 'syslogie' parts in when you write it >> to a >>>> file and have your >>>> analysis tool strip them out >>>> >>>> 3. reformat the apache log message so that you put >> useful >>>> information in >>>> the 'syslogie' parts of the message. >>>> >>>> you can move the timestamp to the beginning (you >> can do >>>> this with or >>>> without the timezone, the format obviously >> differs >>>> slightly) >>>> >>>> you can put the name of the virtual host in the >> server >>>> field >>>> >>>> you can replace 'logger:' with something like >> apache[80]: >>>> or apache[443]: >>>> >>>> I am going to be setting up something along the >> lines of #3 >>>> in the next >>>> few weeks. I figure I will also want to tinker >> with other >>>> things in the >>>> log message. there are items that apache can log, >> but does >>>> not log by >>>> default (I believe that how long it took to >> process the >>>> request is one of >>>> these), and also since syslog defaults to limiting >> log >>>> messages to 1-2K >>>> (depending on your impementation), there are some >> fields >>>> that I would want >>>> to move late in the message so that if they get >> very long >>>> they don't cause >>>> other fields to be lost due to truncation (URL and >> referrer >>>> fields can be >>>> several K long by themselves) >>>> >>>> David Lang >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Nov 27 11:46:03 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 27 Nov 2009 11:46:03 +0100 Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710346E@GRFEXC.intern.adiscon.com> References: <003001ca6092$792833d0$6b789b70$@com> <9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710346E@GRFEXC.intern.adiscon.com> Message-ID: <1259318763.15106.1.camel@rgf11> I have now implemented the complete patch. For v5, I needed to rewrite it, but that wasn't much of an issue once I knew what was required ;) v5 now even has an automatted test for this functionality. Thanks again, Rainer On Fri, 2009-11-20 at 11:03 +0100, Rainer Gerhards wrote: > I, too, think this is useful. Will look at it as soon as I get the patch. The > b) and c) parts may be a bit harder to integrate if they are not for the > newest devel branch, as I rewrote the parser part of rsyslog. But in any > case, it is worth trying to merge it in. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Thursday, November 19, 2009 9:38 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] [PATCH] TLS connection check + syslog TAG > > parsing > > > > for what it's worth, I think that b and c are very useful when dealing > > with strange data sources, and a is probably a bugfix. > > > > David Lang > > > > On Thu, 19 Nov 2009, Rainer Gerhards wrote: > > > > > > > > Sorry, I seem to have overlooked that message. But the mailing list > > processor > > > has stripped the attachment. Could you please re-send to my personal > > mail > > > address. > > > > > > Rainer > > > > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com > > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > >> Jonathan Bond-Caron > > >> Sent: Sunday, November 08, 2009 5:43 PM > > >> To: rsyslog at lists.adiscon.com > > >> Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing > > >> > > >> Hi all, > > >> > > >> > > >> > > >> Attached is a patch for 3 things: > > >> > > >> > > >> > > >> a) Check that TCP connection is still alive when using TLS > > >> > > >> > > >> > > >> b) Improve TAG parsing so that it ends when it sees a > > >> tab or escape > > >> control character. > > >> > > >> > > >> > > >> c) Added configuration directive: escapecontrolcharactertab > > >> > > >> In rsyslog.conf, you can set: > > >> > > >> $escapeControlCharactersOnReceive on > > >> > > >> $escapeControlCharacterTab off > > >> > > >> > > >> > > >> This avoids having a tab being escaped since it is after a viewable > > >> character J > > >> > > >> I'd prefer this being the default behavior but I've left > > >> $escapeControlCharacterTab on as default in case people > > >> expect tabs to be > > >> escaped. > > >> > > >> > > >> > > >> This is my first GIT patch, hope it works well ;) > > >> > > >> > > >> > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From tbergfeld at hq.adiscon.com Fri Nov 27 12:08:17 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 27 Nov 2009 12:08:17 +0100 Subject: [rsyslog] rsyslog 5.5.1 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034CC@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.5.1, a member of the v5-devel branch. The primary new feature is that the epoll() interface is now supported for plain tcp syslog. This results in better performance and also removes the upper bound of connections in select() in a portable way. The select() interface is still supported for platforms that do not provide epoll(). There are also some important fixes. It is a recommended update for all users of the v5-beta. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article433.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-190.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From josesan311 at yahoo.com Sat Nov 28 01:42:02 2009 From: josesan311 at yahoo.com (Jose Sanchez) Date: Fri, 27 Nov 2009 16:42:02 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: Message-ID: <472628.6567.qm@web35606.mail.mud.yahoo.com> Hello David and Reiner, First I would like to thank you for all the help offered, I was able to setup almost everything because of you guys. I had some issues today, though. I found that rsyslog was removing the "logger" properly but it was adding an extra empty space not sure why so I had to cut if off (by watching how to do it on video tutorial first!) by modifying the template that David gave me, I currently have it like this, $template line,"%msg:2:1000%\n" The thing here is Im not sure if this is a reliable solution, I couldnt find if there is any setting that will tell rsyslog to simply remove the empty space or to get everything until the last letter so I configured a very long (1000) number in case rsyslog cuts some part of the text. Not sure if there is any negative impact on doing it this way, if there is any other better way, please let me know. Regarding David's reply, > I would actually suggest that you use the perl script that > you posted, and > filter for programname equal to "apache_syslog", filtering > on just > 'logger' means that you can't use logger for anything > else. I ended using logger because with logger I can specify tags when sending the logs and since I have multiple vhosts and two webserver nodes I could identify from which webserver is coming the logs from, ie: :programname, isequal, "node1.www.domain.com" /var/log/httpd/node1/www.domain.com.log;line :programname, isequal, "node1.blog.domain.com" /var/log/httpd/node1/blog.domain.com.log;line :programname, isequal, "node2.www.domain.com" /var/log/httpd/node2/www.domain.com.log;line :programname, isequal, "node2.blog.domain.com" /var/log/httpd/node2/blog.domain.com.log;line I spent like 3 hours adding those lines due the amount of vhosts I have anyway this created every log and started delivering every vhost for each node on their specific log file just fine. Same thing as above Im still not really sure if this is the most realiable way of doing this with rsyslog. I hope this is the right path for doing such a big thing. Thank you again for everything. --- On Fri, 11/27/09, david at lang.hm wrote: > From: david at lang.hm > Subject: Re: [rsyslog] filter logger tags from syslog > To: "rsyslog-users" > Date: Friday, November 27, 2009, 2:49 AM > On Thu, 26 Nov 2009, Jose Sanchez > wrote: > > > Hello, > > > > Thanks again for the great response. > > It's actually working! rsyslog is removing the > "logger:" thing and all > > the nasty stuff from it automatically, how come? Is it > because we are > > not adding any tag in the template? Im still not > understanding how > > rsyslog removes the logger thing. > > > > Ok, Im now getting the proper output and like David > said Im now getting > > issues with filtering the apache logs with all the > rsyslog messages. > > > I've tried to use the following filter but for some > reason is not > > working and Im not 100% if this is the best solution > to use, > > > > This is how I had set it up, > > > > $template line,"%msg%\n" > > if $msg contains 'GET' then > /var/log/apache.test.log;line > > *.* /var/log/test.log;line > > > > Not sure if Im on the right path, any help will be > appreciated. > > I have also tried the "if" sentence without specifying > the template name. > > > when rsyslog receives a message it parses it. the message > over the wire > hasn't changed (still has the timestamp, servername, > logger: etc), but > rsyslog puts those parts into the seperate variables and > puts what is left > of the message into the %msg% variable. > > so when you change the template from the default of > %timestamp% %hostname% %syslogtag%%msg% > to just > %msg% > > the log file has just the part you care about. > > now for the filtering. > > you could do > :%programname, isequal, "logger" > /var/log/apache.test.log;line > > (as I understand it, this format is a bit more efficiant > for rsyslog than > the equivalent of > if $programname eq "logger" then > /var/log/apache.test.log;line > ) > > I would actually suggest that you use the perl script that > you posted, and > filter for programname equal to "apache_syslog", filtering > on just > 'logger' means that you can't use logger for anything > else. > > you don't want to just filter for 'GET' as there are a > bunch of log files > that won't have GET in them > > David Lang > > > > Many Thanks. > > > > --- On Thu, 11/26/09, david at lang.hm > wrote: > > > >> From: david at lang.hm > >> Subject: Re: [rsyslog] filter logger tags from > syslog > >> To: "rsyslog-users" > >> Date: Thursday, November 26, 2009, 6:38 PM > >> On Thu, 26 Nov 2009, Jose Sanchez > >> wrote: > >> > >>> Hello, > >>> > >>> I appreciate all the responses. > >>> Im not sure how can I can acconplish options > 1) and 2) > >> automatically. > >>> For option 3) the thing is I need "combined" > log type > >> so I cannot reform this. > >>> > >>> Im trying to centralize an access_log file > from one > >> server to the rsyslog server and I need to > completely remove > >> the tags I mentioned on my previous post. > >>> I have also tried using a perl script > mentioned at the > >> botton of this email, but it salso arriving with a > tag, > >> "apache_syslog:" as showed below, > >>> > >>> "apache_syslog: XXX.XXX.XXX.XXX - - > >> [26/Nov/2009:18:23:02 -0600] \"GET /.." > >>> > >>> Basically, this log will be parsed by awstats > which is > >> pretty much stricted with the log format so that's > why I > >> need a clean log sent from the apache server to > the rsyslog > >> server. > >> > >> don't forget that you need to filter these > messages into a > >> seperate file, > >> otherwise you will have your apache combined log > messages > >> mixed with other > >> syslog messages (which will really confuse > awstats) > >> > >> option 1 is what Rainer suggested > >> > >> option 2 is to run the log through another step > before > >> awstats runs, > >> something along the lines of > >> > >> cut -c 16- file |cut -f 3- -d ' ' |awstats > >> > >> the first cut removes the timestamp (always 15 > characters, > >> but with a > >> variable number of spaces in it), the second cut > removes > >> the servername > >> and the syslog tag ('logger:' in your first > example) > >> > >> David Lang > >> > >>> Thank you very much for all the help. > >>> > >>> Below is the Perl script: > >>> > >>> #!/usr/local/bin/perl > >>> # script: apache-access-logger > >>> > >>> use Sys::Syslog; > >>> $SERVER_NAME = shift || ''; > >>> > >>> $PRIORITY = 'info'; > >>> $FACILITY = 'local1'; > >>> > >>> Sys::Syslog::setlogsock('unix'); > >>> openlog ($SERVER_NAME,'ndelay', $FACILITY); > >>> > >>> while (<>) { > >>> ? chomp; > >>> ? syslog($PRIORITY,$_); > >>> } > >>> closelog; > >>> > >>> --- On Thu, 11/26/09, david at lang.hm > >> wrote: > >>> > >>>> From: david at lang.hm > >>>> Subject: Re: [rsyslog] filter logger tags > from > >> syslog > >>>> To: "rsyslog-users" > >>>> Date: Thursday, November 26, 2009, 2:21 > AM > >>>> On Wed, 25 Nov 2009, Jose Sanchez > >>>> wrote: > >>>> > >>>>> Hello, > >>>>> > >>>>> I've been using classic syslog for > >> centralizing apache > >>>> access logs from one server to a remote > syslog > >> server, the > >>>> thing is syslog adds some nasty tags > before the > >> lines in the > >>>> access logs and I cant get them off, ie: > >>>>> > >>>>> "Nov 25 21:25:37 server1 logger:" > >>>>> > >>>>> I would like to know if rsyslog has > the option > >> to > >>>> filter this kind of stuff, I just want to > have the > >> logs sent > >>>> to the syslog server exactly like I was > saving > >> them in a > >>>> local access.log file. > >>>>> > >>>>> Thanks in advance. > >>>> > >>>> 'logger:' is added by the logger program > that > >> apache is > >>>> using to send the > >>>> logs to syslog. > >>>> > >>>> a properly formatted syslog message will > include > >> a > >>>> timestamp and what > >>>> server it came from (note that the apache > logs do > >> _not_ > >>>> tell you what > >>>> virtual server the log comes from, it > usually uses > >> a > >>>> different file for > >>>> each log, so when you mix them into syslog > you > >> won't be > >>>> able to tell them > >>>> apart) > >>>> > >>>> so you have three basic options > >>>> > >>>> 1. let logger do it's default thing and > then use > >> a > >>>> formatting command to > >>>> strip off the 'syslogie' parts to get back > to the > >> apache > >>>> default in the > >>>> file > >>>> > >>>> 2. leave the 'syslogie' parts in when you > write it > >> to a > >>>> file and have your > >>>> analysis tool strip them out > >>>> > >>>> 3. reformat the apache log message so that > you put > >> useful > >>>> information in > >>>> the 'syslogie' parts of the message. > >>>> > >>>> you can move the timestamp to the > beginning (you > >> can do > >>>> this with or > >>>> without the timezone, the format > obviously > >> differs > >>>> slightly) > >>>> > >>>> you can put the name of the virtual host > in the > >> server > >>>> field > >>>> > >>>> you can replace 'logger:' with something > like > >> apache[80]: > >>>> or apache[443]: > >>>> > >>>> I am going to be setting up something > along the > >> lines of #3 > >>>> in the next > >>>> few weeks. I figure I will also want to > tinker > >> with other > >>>> things in the > >>>> log message. there are items that apache > can log, > >> but does > >>>> not log by > >>>> default (I believe that how long it took > to > >> process the > >>>> request is one of > >>>> these), and also since syslog defaults to > limiting > >> log > >>>> messages to 1-2K > >>>> (depending on your impementation), there > are some > >> fields > >>>> that I would want > >>>> to move late in the message so that if > they get > >> very long > >>>> they don't cause > >>>> other fields to be lost due to truncation > (URL and > >> referrer > >>>> fields can be > >>>> several K long by themselves) > >>>> > >>>> David Lang > >>>> > _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> > _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > -----Inline Attachment Follows----- > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From ryan.b.lynch at gmail.com Sat Nov 28 05:45:42 2009 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Fri, 27 Nov 2009 23:45:42 -0500 Subject: [rsyslog] Auto-creating directories, when using templates and the property replacer. Message-ID: <115906d10911272045v237eada0n96b204fcaf1fa6d4@mail.gmail.com> I recently migrated a few Fedora and CentOS hosts from Rsyslog 4.x to 5.x. Several of my systems use output file path templates like this, with multiple dynamically-named directory components: `template DefaultOutputPath,"/var/log/rsyslog/%hostname:::secpath-replace%/%syslogfacility-text:::secpath-replace%/%programname:::secpath-replace%/%$year%/%$month%/%$day%/%$hour%/%$minute%/_.log"` Under 4.x, Rsyslog will automatically create any directories that don't already exist, as needed. But under 5.x, it just fails silently if any of the directories in the path don't exist. Is there a configuration setting that governs this in 5.x, or some way to get back the old automatic directory creation behavior? -Ryan Ryan B. Lynch ryan.b.lynch at gmail.com From rgerhards at hq.adiscon.com Sat Nov 28 09:52:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 28 Nov 2009 09:52:12 +0100 Subject: [rsyslog] Auto-creating directories, when using templates and the property replacer. Message-ID: <000101ca7008$230eb68e$100013ac@intern.adiscon.com> That sounds like a bug, will look into it hopefully early next week. Thanks for reporting! Oh: which version do you use? 5.2.0? Then you should try the recent beta first... rainer ----- Urspr?ngliche Nachricht ----- Von: "Ryan Lynch" An: "rsyslog at lists.adiscon.com" Gesendet: 28.11.09 05:46 Betreff: [rsyslog] Auto-creating directories,when using templates and the property replacer. I recently migrated a few Fedora and CentOS hosts from Rsyslog 4.x to 5.x. Several of my systems use output file path templates like this, with multiple dynamically-named directory components: `template DefaultOutputPath,"/var/log/rsyslog/%hostname:::secpath-replace%/%syslogfacility-text:::secpath-replace%/%programname:::secpath-replace%/%$year%/%$month%/%$day%/%$hour%/%$minute%/_.log"` Under 4.x, Rsyslog will automatically create any directories that don't already exist, as needed. But under 5.x, it just fails silently if any of the directories in the path don't exist. Is there a configuration setting that governs this in 5.x, or some way to get back the old automatic directory creation behavior? -Ryan Ryan B. Lynch ryan.b.lynch at gmail.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Sat Nov 28 10:43:03 2009 From: david at lang.hm (david at lang.hm) Date: Sat, 28 Nov 2009 01:43:03 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <472628.6567.qm@web35606.mail.mud.yahoo.com> References: <472628.6567.qm@web35606.mail.mud.yahoo.com> Message-ID: On Fri, 27 Nov 2009, Jose Sanchez wrote: > Hello David and Reiner, > > First I would like to thank you for all the help offered, I was able to setup almost everything because of you guys. > > I had some issues today, though. I found that rsyslog was removing the "logger" properly but it was adding an extra empty space not sure why so I had to cut if off (by watching how to do it on video tutorial first!) by modifying the template that David gave me, I currently have it like this, > > $template line,"%msg:2:1000%\n" > > The thing here is Im not sure if this is a reliable solution, I couldnt find if there is any setting that will tell rsyslog to simply remove the empty space or to get everything until the last letter so I configured a very long (1000) number in case rsyslog cuts some part of the text. Not sure if there is any negative impact on doing it this way, if there is any other better way, please let me know. if you use '$' instead of '1000' it will go to the end of the message, no matter how long it is (1000 is not long enough for some messages) I think that what you are doing is probably the best way to deal with this space. David Lang From ryan.b.lynch at gmail.com Sat Nov 28 11:08:17 2009 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Sat, 28 Nov 2009 05:08:17 -0500 Subject: [rsyslog] Auto-creating directories, when using templates and the property replacer. In-Reply-To: <000101ca7008$230eb68e$100013ac@intern.adiscon.com> References: <000101ca7008$230eb68e$100013ac@intern.adiscon.com> Message-ID: <115906d10911280208m3ee48754i1c755ebaee67a1d2@mail.gmail.com> On Sat, Nov 28, 2009 at 03:52, Rainer Gerhards wrote: > Oh: which version do you use? 5.2.0? Then you should try the recent beta first... I'm using Rsyslog v5.5.1, and Librelp v.0.1.3. -Ryan From rgerhards at hq.adiscon.com Mon Nov 30 11:38:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 30 Nov 2009 11:38:29 +0100 Subject: [rsyslog] Auto-creating directories, when using templates and the property replacer. References: <000101ca7008$230eb68e$100013ac@intern.adiscon.com> <115906d10911280208m3ee48754i1c755ebaee67a1d2@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034DB@GRFEXC.intern.adiscon.com> Thanks, I was able to reproduced the problem, will now look into it. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > Sent: Saturday, November 28, 2009 11:08 AM > To: rsyslog-users > Subject: Re: [rsyslog] Auto-creating directories,when using templates > and the property replacer. > > On Sat, Nov 28, 2009 at 03:52, Rainer Gerhards > wrote: > > Oh: which version do you use? 5.2.0? Then you should try the recent > beta first... > > I'm using Rsyslog v5.5.1, and Librelp v.0.1.3. > > -Ryan > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Nov 30 12:08:09 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 30 Nov 2009 12:08:09 +0100 Subject: [rsyslog] Auto-creating directories, when using templates and the property replacer. References: <000101ca7008$230eb68e$100013ac@intern.adiscon.com><115906d10911280208m3ee48754i1c755ebaee67a1d2@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71034DB@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034DC@GRFEXC.intern.adiscon.com> Actually, it looks like it was just an improperly initialized variable, so that the default for creating directories was most often "on" (as documented), but could randomly be "off" as well. This bug was hidden all the way back to v3 (I didn't bother to look at v2...). Simple solution, now merged into all branches. The patch for is http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=7b40604e9ae8a0948f17eafd 4299eeb7fb3356c2 Or probably better just pull the newest git version. I would appreciate if you let me know if it works for you. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Monday, November 30, 2009 11:38 AM > To: rsyslog-users > Subject: Re: [rsyslog] Auto-creating directories,when using templates > and the property replacer. > > Thanks, I was able to reproduced the problem, will now look into it. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > > Sent: Saturday, November 28, 2009 11:08 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] Auto-creating directories,when using templates > > and the property replacer. > > > > On Sat, Nov 28, 2009 at 03:52, Rainer Gerhards > > wrote: > > > Oh: which version do you use? 5.2.0? Then you should try the recent > > beta first... > > > > I'm using Rsyslog v5.5.1, and Librelp v.0.1.3. > > > > -Ryan > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From xkubina at fi.muni.cz Mon Nov 30 14:55:55 2009 From: xkubina at fi.muni.cz (Tomas Kubina) Date: Mon, 30 Nov 2009 14:55:55 +0100 Subject: [rsyslog] TLS/GSSAPI client message lost Message-ID: <4B13CEEB.4040504@fi.muni.cz> Hi, I want to use TLS or GSS for message delivering to central rsyslog server. The problem is that the first message logged after server's shutdown is lost, but when I use plain TCP this issue doesn't happen. Is it a feature or mistake in my config? This is config for client: ################# #### MODULES #### ################# $ModLoad imuxsock # provides support for local system logging $ModLoad imklog # provides kernel logging support (previously done by rklogd) $ModLoad immark # provides --MARK-- message capability ########################### #### GLOBAL DIRECTIVES #### ########################### # # Use traditional timestamp format. # To enable high precision timestamps, comment out the following line. # $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat ############### #### RULES #### ############### # # First some standard log files. Log by facility. # auth,authpriv.* /var/log/auth.log *.*;auth,authpriv.none -/var/log/syslog #cron.* /var/log/cron.log daemon.* -/var/log/daemon.log kern.* -/var/log/kern.log lpr.* -/var/log/lpr.log mail.* -/var/log/mail.log user.* -/var/log/user.log # # Logging for the mail system. Split it up so that # it is easy to write scripts to parse these files. # mail.info -/var/log/mail.info mail.warn -/var/log/mail.warn mail.err /var/log/mail.err # # Logging for INN news system. # news.crit /var/log/news/news.crit news.err /var/log/news/news.err news.notice -/var/log/news/news.notice # # Some "catch-all" log files. # *.=debug;\ auth,authpriv.none;\ news.none;mail.none -/var/log/debug *.=info;*.=notice;*.=warn;\ auth,authpriv.none;\ cron,daemon.none;\ mail,news.none -/var/log/messages # # Emergencies are sent to everybody logged in. # *.emerg * # # I like to have messages displayed on the console, but only on a virtual # console I usually leave idle. # #daemon,mail.*;\ # news.=crit;news.=err;news.=notice;\ # *.=debug;*.=info;\ # *.=notice;*.=warn /dev/tty8 # The named pipe /dev/xconsole is for the `xconsole' utility. To use it, # you must invoke `xconsole' with the `-file' option: # # $ xconsole -file /dev/xconsole [...] # # NOTE: adjust the list below, or you'll go crazy if you have a reasonably # busy site.. # daemon.*;mail.*;\ news.err;\ *.=debug;*.=info;\ *.=notice;*.=warn |/dev/xconsole # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. $WorkDirectory /var/tmp/rsyslog/spool # where to place spool files $ActionQueueFileName uniqName # unique name prefix for spool files $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) $ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously $ActionResumeRetryCount -1 # infinite retries if host is down #$DefaultNetstreamDriver gtls # use gtls netstream driver # certificate files - just CA for a client #$DefaultNetstreamDriverCAFile /root/ils/cacert.pem #$DefaultNetstreamDriverCertFile /root/ils/bobatko_cert.pem #$DefaultNetstreamDriverKeyFile /root/ils/bobatko_cert.pem # set up the action #$ActionSendStreamDriverMode 1 # require TLS for the connection #$ActionSendStreamDriverAuthMode anon # server is NOT authenticated #local7.info @@example.com:10514 $ModLoad omgssapi local7.info : omgssapi:example.com:10514 and the server side: $ModLoad immark # provides --MARK-- message capability $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imklog # kernel logging (formerly provided by rklogd) # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none -/var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* -/var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit -/var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) #$ModLoad imtcp # TCP listener # make gtls driver the default #$DefaultNetstreamDriver gtls # certificate files #$DefaultNetstreamDriverCAFile /root/rsyslog/cacert.pem #$DefaultNetstreamDriverCertFile /root/rsyslog/rsyslog_cert.pem #$DefaultNetstreamDriverKeyFile /root/rsyslog/rsyslog_key.pem #$InputTCPServerStreamDriverAuthMode x509/name #$InputTCPServerStreamDriverPermittedPeer bobatko #$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode #$InputTCPServerRun 10514 # start up listener at port 10514 $ModLoad imgssapi $InputGSSServerRun 10514 Thank you for the answer, Regards, Tomas Kubina From rgerhards at hq.adiscon.com Mon Nov 30 15:06:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 30 Nov 2009 15:06:53 +0100 Subject: [rsyslog] TLS/GSSAPI client message lost References: <4B13CEEB.4040504@fi.muni.cz> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034DF@GRFEXC.intern.adiscon.com> I unfortunately do not have enough insight into GSSAPI to provide a real answer. My guess is that this happens for the same reason it can happen with any ack-less transport. In the plain tcp driver, I have done quite some work to try to limit loss potential. I have also some ideas of how to further try to prevent message loss, but you cannot totally aovid it. some background: http://blog.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tomas Kubina > Sent: Monday, November 30, 2009 2:56 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] TLS/GSSAPI client message lost > > Hi, > > I want to use TLS or GSS for message delivering to central rsyslog > server. > The problem is that the first message logged after server's shutdown is > lost, > but when I use plain TCP this issue doesn't happen. Is it a feature or > mistake > in my config? > > This is config for client: > > ################# > #### MODULES #### > ################# > > $ModLoad imuxsock # provides support for local system logging > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > $ModLoad immark # provides --MARK-- message capability > > > ########################### > #### GLOBAL DIRECTIVES #### > ########################### > > # > # Use traditional timestamp format. > # To enable high precision timestamps, comment out the following line. > # > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > ############### > #### RULES #### > ############### > > # > # First some standard log files. Log by facility. > # > auth,authpriv.* /var/log/auth.log > *.*;auth,authpriv.none -/var/log/syslog > #cron.* /var/log/cron.log > daemon.* -/var/log/daemon.log > kern.* -/var/log/kern.log > lpr.* -/var/log/lpr.log > mail.* -/var/log/mail.log > user.* -/var/log/user.log > > # > # Logging for the mail system. Split it up so that > # it is easy to write scripts to parse these files. > # > mail.info -/var/log/mail.info > mail.warn -/var/log/mail.warn > mail.err /var/log/mail.err > > # > # Logging for INN news system. > # > news.crit /var/log/news/news.crit > news.err /var/log/news/news.err > news.notice -/var/log/news/news.notice > > # > # Some "catch-all" log files. > # > *.=debug;\ > auth,authpriv.none;\ > news.none;mail.none -/var/log/debug > *.=info;*.=notice;*.=warn;\ > auth,authpriv.none;\ > cron,daemon.none;\ > mail,news.none -/var/log/messages > > # > # Emergencies are sent to everybody logged in. > # > *.emerg * > > # > # I like to have messages displayed on the console, but only on a > virtual > # console I usually leave idle. > # > #daemon,mail.*;\ > # news.=crit;news.=err;news.=notice;\ > # *.=debug;*.=info;\ > # *.=notice;*.=warn /dev/tty8 > > # The named pipe /dev/xconsole is for the `xconsole' utility. To use > it, > # you must invoke `xconsole' with the `-file' option: > # > # $ xconsole -file /dev/xconsole [...] > # > # NOTE: adjust the list below, or you'll go crazy if you have a > reasonably > # busy site.. > # > daemon.*;mail.*;\ > news.err;\ > *.=debug;*.=info;\ > *.=notice;*.=warn |/dev/xconsole > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/tmp/rsyslog/spool # where to place spool files > $ActionQueueFileName uniqName # unique name prefix for spool files > $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) > $ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > $ActionResumeRetryCount -1 # infinite retries if host is down > > > #$DefaultNetstreamDriver gtls # use gtls netstream driver > > # certificate files - just CA for a client > #$DefaultNetstreamDriverCAFile /root/ils/cacert.pem > #$DefaultNetstreamDriverCertFile /root/ils/bobatko_cert.pem > #$DefaultNetstreamDriverKeyFile /root/ils/bobatko_cert.pem > > # set up the action > #$ActionSendStreamDriverMode 1 # require TLS for the connection > #$ActionSendStreamDriverAuthMode anon # server is NOT authenticated > > #local7.info @@example.com:10514 > > > $ModLoad omgssapi > local7.info : omgssapi:example.com:10514 > > > and the server side: > > $ModLoad immark # provides --MARK-- message capability > $ModLoad imuxsock # provides support for local system logging (e.g. via > logger command) > $ModLoad imklog # kernel logging (formerly provided by rklogd) > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none -/var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* -/var/log/maillog > > > # Log cron stuff > cron.* -/var/log/cron > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit -/var/log/spooler > > # Save boot messages also to boot.log > local7.* /var/log/boot.log > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > > #$ModLoad imtcp # TCP listener > > # make gtls driver the default > #$DefaultNetstreamDriver gtls > > # certificate files > #$DefaultNetstreamDriverCAFile /root/rsyslog/cacert.pem > #$DefaultNetstreamDriverCertFile /root/rsyslog/rsyslog_cert.pem > #$DefaultNetstreamDriverKeyFile /root/rsyslog/rsyslog_key.pem > > #$InputTCPServerStreamDriverAuthMode x509/name > #$InputTCPServerStreamDriverPermittedPeer bobatko > #$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode > > #$InputTCPServerRun 10514 # start up listener at port 10514 > > $ModLoad imgssapi > $InputGSSServerRun 10514 > > Thank you for the answer, > > Regards, > Tomas Kubina > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From iamsayan at gmail.com Mon Nov 30 17:20:16 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Mon, 30 Nov 2009 11:20:16 -0500 Subject: [rsyslog] using arbitrary facility id Message-ID: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com> Hello All, Is it possible to use a facility id other than local0-local7? I was using a facility id of 50 in some of the messages , and I had written a selector line in my rsyslog file as well to log messages with facility id of 50 into a seperate file. However, I see that sometimes the messages are being written into all the files(/var/log/messages, boot.log,/var/log/secure etc) instead of the one I specified in the rsyslog.conf. If I restart rsyslog the problem goes away. I am using rsyslog version 4.2.0. Here is the selector line in my config if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and $syslogseverity <= '6' then $log_rotation_50 where $log_rotation_50 is an outchannel which is configured to rotate the file when it reaches the size of 2MB. Regards, Sayan From rgerhards at hq.adiscon.com Mon Nov 30 17:40:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 30 Nov 2009 17:40:17 +0100 Subject: [rsyslog] using arbitrary facility id References: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com> You can use facilities other than local0..local7, but you cannot use a facility with a numerical value greater than 23, because the relevant standards do not permit this (see RFC5424, Table 1). It may be that rsyslog does not properly prevent this, maybe it uses modulo 24 in this case. Will check that. But it is invalid in any case (not only with rsyslog but with any syslogd). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Monday, November 30, 2009 5:20 PM > To: rsyslog-users > Subject: [rsyslog] using arbitrary facility id > > Hello All, > Is it possible to use a facility id other than local0-local7? > I was using a facility id of 50 in some of the messages , and I had > written > a selector line in my rsyslog file as well to log messages with > facility id > of 50 into a seperate file. > However, I see that sometimes the messages are being written into all > the > files(/var/log/messages, boot.log,/var/log/secure etc) instead of the > one I > specified in the rsyslog.conf. If I restart rsyslog the problem goes > away. > I am using rsyslog version 4.2.0. > > Here is the selector line in my config > > > if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and > $syslogseverity <= '6' then $log_rotation_50 > > where $log_rotation_50 is an outchannel which is configured to rotate > the > file when it reaches the size of 2MB. > Regards, > Sayan > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From iamsayan at gmail.com Mon Nov 30 17:51:40 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Mon, 30 Nov 2009 11:51:40 -0500 Subject: [rsyslog] using arbitrary facility id In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com> References: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540911300851r1f0bab68ldf7f995ed0a094d8@mail.gmail.com> Hello Rainer, Thanks... Yes I agree what I am doing is invalid. I was just trying it out, my idea was if I can use numbers greater than 23, then I can maybe use it as my own customized logging system, and I would use numbers greater than 23 for my application and use different ids (>23) for different kind of application level log messages. I would try to look into the code as well, is this something you thing may be useful in general? Regards, Sayan On Mon, Nov 30, 2009 at 11:40 AM, Rainer Gerhards wrote: > You can use facilities other than local0..local7, but you cannot use a > facility with a numerical value greater than 23, because the relevant > standards do not permit this (see RFC5424, Table 1). > > It may be that rsyslog does not properly prevent this, maybe it uses modulo > 24 in this case. Will check that. But it is invalid in any case (not only > with rsyslog but with any syslogd). > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > Sent: Monday, November 30, 2009 5:20 PM > > To: rsyslog-users > > Subject: [rsyslog] using arbitrary facility id > > > > Hello All, > > Is it possible to use a facility id other than local0-local7? > > I was using a facility id of 50 in some of the messages , and I had > > written > > a selector line in my rsyslog file as well to log messages with > > facility id > > of 50 into a seperate file. > > However, I see that sometimes the messages are being written into all > > the > > files(/var/log/messages, boot.log,/var/log/secure etc) instead of the > > one I > > specified in the rsyslog.conf. If I restart rsyslog the problem goes > > away. > > I am using rsyslog version 4.2.0. > > > > Here is the selector line in my config > > > > > > if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and > > $syslogseverity <= '6' then $log_rotation_50 > > > > where $log_rotation_50 is an outchannel which is configured to rotate > > the > > file when it reaches the size of 2MB. > > Regards, > > Sayan > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iamsayan at gmail.com Mon Nov 30 17:58:46 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Mon, 30 Nov 2009 11:58:46 -0500 Subject: [rsyslog] using arbitrary facility id In-Reply-To: <87a1ae540911300851r1f0bab68ldf7f995ed0a094d8@mail.gmail.com> References: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com> <87a1ae540911300851r1f0bab68ldf7f995ed0a094d8@mail.gmail.com> Message-ID: <87a1ae540911300858v1d4cda71k787a53158e162b7d@mail.gmail.com> BTW, just out of interest, why is this number restricted to 23 in the rfc? also each facility other than local0-7 seems to be rigidly defined. Just wanted to know why is it so ..shouldn't they have left scope for extension on this? On Mon, Nov 30, 2009 at 11:51 AM, Sayan Chowdhury wrote: > Hello Rainer, > Thanks... Yes I agree what I am doing is invalid. > I was just trying it out, my idea was if I can use numbers greater than > 23, then I can maybe use it as my own customized logging system, and I would > use numbers greater than 23 for my application and use different ids (>23) > for different kind of application level log messages. I would try to look > into the code as well, is this something you thing may be useful in general? > Regards, > Sayan > > > > On Mon, Nov 30, 2009 at 11:40 AM, Rainer Gerhards < > rgerhards at hq.adiscon.com> wrote: > >> You can use facilities other than local0..local7, but you cannot use a >> facility with a numerical value greater than 23, because the relevant >> standards do not permit this (see RFC5424, Table 1). >> >> It may be that rsyslog does not properly prevent this, maybe it uses >> modulo >> 24 in this case. Will check that. But it is invalid in any case (not only >> with rsyslog but with any syslogd). >> >> Rainer >> >> > -----Original Message----- >> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury >> > Sent: Monday, November 30, 2009 5:20 PM >> > To: rsyslog-users >> > Subject: [rsyslog] using arbitrary facility id >> > >> > Hello All, >> > Is it possible to use a facility id other than local0-local7? >> > I was using a facility id of 50 in some of the messages , and I had >> > written >> > a selector line in my rsyslog file as well to log messages with >> > facility id >> > of 50 into a seperate file. >> > However, I see that sometimes the messages are being written into all >> > the >> > files(/var/log/messages, boot.log,/var/log/secure etc) instead of the >> > one I >> > specified in the rsyslog.conf. If I restart rsyslog the problem goes >> > away. >> > I am using rsyslog version 4.2.0. >> > >> > Here is the selector line in my config >> > >> > >> > if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and >> > $syslogseverity <= '6' then $log_rotation_50 >> > >> > where $log_rotation_50 is an outchannel which is configured to rotate >> > the >> > file when it reaches the size of 2MB. >> > Regards, >> > Sayan >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > From jbondc at openmv.com Mon Nov 30 21:49:24 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Mon, 30 Nov 2009 15:49:24 -0500 Subject: [rsyslog] TLS/GSSAPI client message lost In-Reply-To: <4B13CEEB.4040504@fi.muni.cz> References: <4B13CEEB.4040504@fi.muni.cz> Message-ID: <006d01ca71fe$9a625d50$cf2717f0$@com> On Mon Nov 30 08:55 AM, Tomas Kubina wrote: > Hi, > > I want to use TLS or GSS for message delivering to central rsyslog > server. > The problem is that the first message logged after server's shutdown > is lost, but when I use plain TCP this issue doesn't happen. Is it a > feature or mistake in my config? > I think your problem will be fixed in the latest rsyslog: http://www.rsyslog.com/Article433.phtml From epiphani at gmail.com Mon Nov 2 23:51:40 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Mon, 2 Nov 2009 17:51:40 -0500 Subject: [rsyslog] Queuing bug (4.5.5) Message-ID: Greetings all, I appear to have run into an issue with 4.5.5 where queues are not being flushed in a timely manner. In this specific case, I have data from last wednesday that is being written to disk in small chunks today since last wednesday. Unfortunately I cannot be too specific in details, but here is what I can see: Using omfile+gzip, there appears to have been a decent burst in traffic sometime last wednesday. The rsyslog instance has grown to 1.8GB of memory and stayed there for a while. I have both main message and action queues defined using fixedarray, and I see no on-disk queues (appears to be entirely in memory). I've got templates writing out to filenames using an hourly timestamp (filenames like: [token]-[host]-YYYYMMDD-HH.txt.gz) In some of those files, all of them less than 5k in size, there are between 5 and 15 lines of content, all of them from last wednesday, and within a few seconds of each other. It's almost like there was a significant queue built up, the hour rolled over, and only the first block of lines were pulled from the queue. Then the hour rolled over again, and another block of lines were pulled from the queue. Then the next hour, then another 5-15 lines. Is it possible that one of the queues still has a good chunk of data built up, and is flushing it out very slowly? It hasn't been consistantly at the top of the hour either, and not every hour. But the log lines themselves are sequentially written out, and usually the lines are within a few seconds of each other. For example: syslog-myhost-20091102-18.txt.gz: 3 lines, 2 with TS Oct 21 18:46:34 and one 18:46:35 syslog-myhost-20091102-19.txt.gz: 17 lines, 3 Oct 21 18:46:35, 14 lines Oct 21 18:46:36 syslog-myhost-20091102-20.txt.gz: 12 lines, 8 Oct 21 18:46:36, 4 lines Oct 21 18:46:37 Thoughts? -Aaron From rgerhards at hq.adiscon.com Tue Nov 3 07:46:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 07:46:29 +0100 Subject: [rsyslog] Queuing bug (4.5.5) References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> mhhh... This is obviously not intended behavior. There are some rate-limiting features that can deliberately slow down a queue, but I guess you have not configured these. So there either is a bug that activates them at some point during processing (e.g. an invalid memory access could do that) or there is some other bug that causes the dequeue to happen very slowly. In any case, it is hard to guess. Given the volume of the messages, a debug log probably does not help. We could gain a little insight in at least the queue sizes that rsyslog sees via imdiag plus the (very rudamentary) v5 debug front-end (it doesn't require the engine to be v5!). I would need to spend at least a little work on the front-end to enable remote access, but that's not really a problem. One other thing is that I am still holding a v4-beta with additional fixes as I didn't want to put these in the middle of some other debugging. However, these fixes address potential memory problems, so the most appropriate course of action would be to give that version at least a try. It needs to be released in the next days in any case. I have uploaded that (pre-4.5.6) version so that you can give it a try if you like: http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz I think it would good if you could try it at least once, because I think any other troubleshooting will boil down to looking at the fixes this version contains. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Monday, November 02, 2009 11:52 PM > To: rsyslog-users > Subject: [rsyslog] Queuing bug (4.5.5) > > Greetings all, > > I appear to have run into an issue with 4.5.5 where queues are not > being flushed in a timely manner. In this specific case, I have data > from last wednesday that is being written to disk in small chunks > today since last wednesday. Unfortunately I cannot be too specific in > details, but here is what I can see: > > Using omfile+gzip, there appears to have been a decent burst in > traffic sometime last wednesday. The rsyslog instance has grown to > 1.8GB of memory and stayed there for a while. I have both main > message and action queues defined using fixedarray, and I see no > on-disk queues (appears to be entirely in memory). > > I've got templates writing out to filenames using an hourly timestamp > (filenames like: [token]-[host]-YYYYMMDD-HH.txt.gz) In some of those > files, all of them less than 5k in size, there are between 5 and 15 > lines of content, all of them from last wednesday, and within a few > seconds of each other. It's almost like there was a significant queue > built up, the hour rolled over, and only the first block of lines were > pulled from the queue. Then the hour rolled over again, and another > block of lines were pulled from the queue. Then the next hour, then > another 5-15 lines. > > Is it possible that one of the queues still has a good chunk of data > built up, and is flushing it out very slowly? It hasn't been > consistantly at the top of the hour either, and not every hour. But > the log lines themselves are sequentially written out, and usually the > lines are within a few seconds of each other. > > For example: > > syslog-myhost-20091102-18.txt.gz: 3 lines, 2 with TS Oct 21 18:46:34 > and one 18:46:35 > syslog-myhost-20091102-19.txt.gz: 17 lines, 3 Oct 21 18:46:35, 14 > lines Oct 21 18:46:36 > syslog-myhost-20091102-20.txt.gz: 12 lines, 8 Oct 21 18:46:36, 4 > lines Oct 21 18:46:37 > > Thoughts? > -Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 3 07:59:23 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 07:59:23 +0100 Subject: [rsyslog] Queue sizing References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103316@GRFEXC.intern.adiscon.com> Sorry for the late reply, but I was really busy with my article last week. Let me provide some rough numbers. For usual syslog traffic (< 100 bytes message size), 768 bytes is a good size to be expected for use by each message. If you have larger traffic, add the full average size to that picture (e.g. average size is 200 bytes, the typical size of a message object is around 968 bytes, NOT 868 - this has to do with some internal optimization). Then, simply multiply the configured size of the queue with that number. For example, you have a 2,000,000 message max configured, so 2,000,000 * 768 which gives you ~ 1.4 GB message store if the queue is full. The same applies to the action queues, here we have 200,000 entries per queue, that would sum up to ~140 MB per queue. Multiply that with the number of action queues. If, for example, you have 20 action queues, that would sum up to a total memory requirement of 20*.14+1.4 = 4.2 GB of total queue memory. Of course, this is an extreme number, and typically no system will have all action queues totally filled up. For omfile, with background writing (always enabled in ZIP mode), we use double-buffering with the configured buffers sizes (256KB being the default). So add 0.5 MB per open dynafile in this case. Multiply that by the maximum number of dynafiles you permit to be kept open. Add these numbers for each dynafile action. That gives you a (very theoretical) upper bound of memory use for the output system. Limiting the memory use is so simple that it probably is easy to overlook: just reduce the maximum numbers - that's why they exist ;) Depending on the input source, rsyslog then tries to slow down too-fast senders if it can't keep up. This works pretty well with TCP and not at all with UDP. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Wednesday, October 28, 2009 4:14 PM > To: rsyslog-users > Subject: [rsyslog] Queue sizing > > Greetings, > > I'm trying to get a good handle on queue sizing and memory usage. In > a higher-traffic environment, I have a dedicated rsyslog machine with > a large amount of ram. I want to try and find the right balance > between large queues for burst traffic, but I also want to make sure > that the queue sizes are small enough that we don't run out of memory > and crash. I'm using 4.5.5 currently. > > I'm currently using disk-backed memory queues, configured in > fixedarray, like so: > > $MainMsgQueueSize 2000000 > $MainMsgQueueMaxFileSize 50g > $MainMsgQueueSaveOnShutdown on > $MainMsgQueueHighWaterMark 15000000 > $MainMsgQueueLowWaterMark 10000000 > $MainMsgQueueType FixedArray > $MainMsgQueueWorkerThreads 2 > > $ActionQueueSize 200000 > $ActionQueueMaxFileSize 50g > $ActionQueueSaveOnShutdown on > $ActionQueueHighWaterMark 1500000 > $ActionQueueLowWaterMark 1000000 > $ActionQueueType FixedArray > $ActionQueueWorkerThreads 1 > > > I usually have upwards of a dozen action queue definitions, and my > omfile outputs are based on hostname of the incoming connection. Is > there some way I can calculate maximum memory use by the queuing > system, or force the system to a set memory maximum where I -know- > that it will simply start pushing back on the incoming connection > instead of adding more data to the system? > > Thoughts? > -Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 3 09:59:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 09:59:44 +0100 Subject: [rsyslog] queue configuration References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103318@GRFEXC.intern.adiscon.com> David and all, I have created a new output plugin, omruleset, which permits to nest rulesets. What it does is copy a message over from one ruleset to another and make it process in parallel. Some more details in the doc: http://www.rsyslog.com/doc-omruleset.html It permits to do what you asked for below. It also permits to do many more things and create very advanced configurations. But one must be VERY careful to receive the intended results. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 23, 2009 10:23 PM > To: rsyslog-users > Subject: [rsyslog] queue configuration > > I know that you can create an action queue for a specific output. > > is there any way to create an action queue for multiple outputs? > > for example, in my configuration I have > > :fromhost, !isequal, "127.0.0.1" > /var/log/messages;TraditionalFormat > :fromhost, isequal, "127.0.0.1" > @192.168.1.1;TraditionalForwardFormat > *.* @192.168.1.115 > *.* @192.168.1.241 > *.* @192.168.1.242 > *.* @192.168.1.6 > *.* @192.168.1.7 > *.* @192.168.1.122 > :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 > & @192.168.1.1;fixsnareForwardFormat2 > & ~ > :syslogtag, startswith, "MSWinEventLog#011" > /var/log/messages;fixsnareFormat > & @192.168.1.1;fixsnareForwardFormat > & ~ > *.* /var/log/messages;TraditionalFormat > *.* @192.168.1.1 > > it would be nice to move the outbound relays off to a different queue, > and > to put the list of rules that have order dependancies (because they > output > in different formats to fix up formatting problem) to a seperate queue > to > spread the work across multiple processes and not slow down the basic > writing to file. > > but as far as I can tell I would have to create a queue for each > individual item, and I can't create a queue for the part of the config > where I need to discard. > > am I missing something (including a better way to do everything ;-) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Nov 3 11:58:02 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 3 Nov 2009 11:58:02 +0100 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200910201639.43143.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> Message-ID: <200911031158.02666.marc.schiffbauer@mightycare.de> Hello Rainer, is there some news about this issue? TIA -Marc Am Dienstag, 20. Oktober 2009 18:38:16 schrieb Rainer Gerhards: > thanks! > > Interesting, I see that only one of the messages that should be in the .qi > are actually present. I wonder if this is related to the fact that ompgsql > emits an error message itself while being down (the others don't do this > AFIK). Will try to dig down to this (but I have to do so as time permits). > > Rainer > From mikel at irontec.com Tue Nov 3 12:20:18 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 03 Nov 2009 12:20:18 +0100 Subject: [rsyslog] working directoy dude Message-ID: <4AF011F2.7040501@irontec.com> Hi! I have a simple question: When you configure reliable TCP mesage forwarding, the "$WorkDirectory /rsyslog/work" have to be created? So, mkdir -p /rsyslog/work ? Thanks From rgerhards at hq.adiscon.com Tue Nov 3 12:28:19 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 12:28:19 +0100 Subject: [rsyslog] working directoy dude References: <4AF011F2.7040501@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710331C@GRFEXC.intern.adiscon.com> yes :) > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, November 03, 2009 12:20 PM > To: rsyslog-users > Subject: [rsyslog] working directoy dude > > Hi! > > I have a simple question: > > When you configure reliable TCP mesage forwarding, the "$WorkDirectory > /rsyslog/work" have to be created? > > So, mkdir -p /rsyslog/work ? > > Thanks > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mikel at irontec.com Tue Nov 3 12:36:02 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 03 Nov 2009 12:36:02 +0100 Subject: [rsyslog] working directoy dude In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710331C@GRFEXC.intern.adiscon.com> References: <4AF011F2.7040501@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA710331C@GRFEXC.intern.adiscon.com> Message-ID: <4AF015A2.6090303@irontec.com> Thanks Rainer!! Rainer Gerhards wrote: > yes :) > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Tuesday, November 03, 2009 12:20 PM >> To: rsyslog-users >> Subject: [rsyslog] working directoy dude >> >> Hi! >> >> I have a simple question: >> >> When you configure reliable TCP mesage forwarding, the "$WorkDirectory >> /rsyslog/work" have to be created? >> >> So, mkdir -p /rsyslog/work ? >> >> Thanks >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From epiphani at gmail.com Tue Nov 3 15:59:57 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 3 Nov 2009 09:59:57 -0500 Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> Message-ID: This is still taking place this hour - though I obviously can't restart onto a newer version without losing this case. Is there anything I can do in the current configuration to try and debug this situation? (We're up to 18:46:51 now!) -Aaron On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards wrote: > mhhh... This is obviously not intended behavior. There are some rate-limiting > features that can deliberately slow down a queue, but I guess you have not > configured these. So there either is a bug that activates them at some point > during processing (e.g. an invalid memory access could do that) or there is > some other bug that causes the dequeue to happen very slowly. In any case, it > is hard to guess. > > Given the volume of the messages, a debug log probably does not help. We > could gain a little insight in at least the queue sizes that rsyslog sees via > imdiag plus the (very rudamentary) v5 debug front-end (it doesn't require the > engine to be v5!). I would need to spend at least a little work on the > front-end to enable remote access, but that's not really a problem. > > One other thing is that I am still holding a v4-beta with additional fixes as > I didn't want to put these in the middle of some other debugging. However, > these fixes address potential memory problems, so the most appropriate course > of action would be to give that version at least a try. It needs to be > released in the next days in any case. > > I have uploaded that (pre-4.5.6) version so that you can give it a try if you > like: > > http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz > > I think it would good if you could try it at least once, because I think any > other troubleshooting will boil down to looking at the fixes this version > contains. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> Sent: Monday, November 02, 2009 11:52 PM >> To: rsyslog-users >> Subject: [rsyslog] Queuing bug (4.5.5) >> >> Greetings all, >> >> I appear to have run into an issue with 4.5.5 where queues are not >> being flushed in a timely manner. ?In this specific case, I have data >> from last wednesday that is being written to disk in small chunks >> today since last wednesday. ?Unfortunately I cannot be too specific in >> details, but here is what I can see: >> >> Using omfile+gzip, there appears to have been a decent burst in >> traffic sometime last wednesday. ?The rsyslog instance has grown to >> 1.8GB of memory and stayed there for a while. ?I have both main >> message and action queues defined using fixedarray, and I see no >> on-disk queues (appears to be entirely in memory). >> >> I've got templates writing out to filenames using an hourly timestamp >> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of those >> files, all of them less than 5k in size, there are between 5 and 15 >> lines of content, all of them from last wednesday, and within a few >> seconds of each other. ?It's almost like there was a significant queue >> built up, the hour rolled over, and only the first block of lines were >> pulled from the queue. ?Then the hour rolled over again, and another >> block of lines were pulled from the queue. ?Then the next hour, then >> another 5-15 lines. >> >> Is it possible that one of the queues still has a good chunk of data >> built up, and is flushing it out very slowly? ?It hasn't been >> consistantly at the top of the hour either, and not every hour. ?But >> the log lines themselves are sequentially written out, and usually the >> lines are within a few seconds of each other. >> >> For example: >> >> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 18:46:34 >> and one 18:46:35 >> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, 14 >> lines Oct 21 18:46:36 >> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, 4 >> lines Oct 21 18:46:37 >> >> Thoughts? >> -Aaron >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Tue Nov 3 16:06:15 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 3 Nov 2009 07:06:15 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 3 Nov 2009, Aaron Wiebe wrote: > This is still taking place this hour - though I obviously can't > restart onto a newer version without losing this case. Is there > anything I can do in the current configuration to try and debug this > situation? if you can strace each thread for a few seconds it may give you an idea what's happening. in the meantime you need to stop the system from getting further behind, can you redirect or reconfigure the senders so that they are not sending new logs to this box so that it can dig itself out (stopping the input may be enough by itself, rsyslog has historicly done a LOT of locking around the main queue, and if you have a full queue with more data trying to be delivered it can really slow things down. I wouldn't expect it to be this much, but it could be part of it) David Lang > (We're up to 18:46:51 now!) > > -Aaron > > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards > wrote: >> mhhh... This is obviously not intended behavior. There are some rate-limiting >> features that can deliberately slow down a queue, but I guess you have not >> configured these. So there either is a bug that activates them at some point >> during processing (e.g. an invalid memory access could do that) or there is >> some other bug that causes the dequeue to happen very slowly. In any case, it >> is hard to guess. >> >> Given the volume of the messages, a debug log probably does not help. We >> could gain a little insight in at least the queue sizes that rsyslog sees via >> imdiag plus the (very rudamentary) v5 debug front-end (it doesn't require the >> engine to be v5!). I would need to spend at least a little work on the >> front-end to enable remote access, but that's not really a problem. >> >> One other thing is that I am still holding a v4-beta with additional fixes as >> I didn't want to put these in the middle of some other debugging. However, >> these fixes address potential memory problems, so the most appropriate course >> of action would be to give that version at least a try. It needs to be >> released in the next days in any case. >> >> I have uploaded that (pre-4.5.6) version so that you can give it a try if you >> like: >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz >> >> I think it would good if you could try it at least once, because I think any >> other troubleshooting will boil down to looking at the fixes this version >> contains. >> >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>> Sent: Monday, November 02, 2009 11:52 PM >>> To: rsyslog-users >>> Subject: [rsyslog] Queuing bug (4.5.5) >>> >>> Greetings all, >>> >>> I appear to have run into an issue with 4.5.5 where queues are not >>> being flushed in a timely manner. ?In this specific case, I have data >>> from last wednesday that is being written to disk in small chunks >>> today since last wednesday. ?Unfortunately I cannot be too specific in >>> details, but here is what I can see: >>> >>> Using omfile+gzip, there appears to have been a decent burst in >>> traffic sometime last wednesday. ?The rsyslog instance has grown to >>> 1.8GB of memory and stayed there for a while. ?I have both main >>> message and action queues defined using fixedarray, and I see no >>> on-disk queues (appears to be entirely in memory). >>> >>> I've got templates writing out to filenames using an hourly timestamp >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of those >>> files, all of them less than 5k in size, there are between 5 and 15 >>> lines of content, all of them from last wednesday, and within a few >>> seconds of each other. ?It's almost like there was a significant queue >>> built up, the hour rolled over, and only the first block of lines were >>> pulled from the queue. ?Then the hour rolled over again, and another >>> block of lines were pulled from the queue. ?Then the next hour, then >>> another 5-15 lines. >>> >>> Is it possible that one of the queues still has a good chunk of data >>> built up, and is flushing it out very slowly? ?It hasn't been >>> consistantly at the top of the hour either, and not every hour. ?But >>> the log lines themselves are sequentially written out, and usually the >>> lines are within a few seconds of each other. >>> >>> For example: >>> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 18:46:34 >>> and one 18:46:35 >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, 14 >>> lines Oct 21 18:46:36 >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, 4 >>> lines Oct 21 18:46:37 >>> >>> Thoughts? >>> -Aaron >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 3 16:25:30 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 16:25:30 +0100 Subject: [rsyslog] Queuing bug (4.5.5) References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, November 03, 2009 4:06 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) > > On Tue, 3 Nov 2009, Aaron Wiebe wrote: > > > This is still taking place this hour - though I obviously can't > > restart onto a newer version without losing this case. Is there > > anything I can do in the current configuration to try and debug this > > situation? > > if you can strace each thread for a few seconds it may give you an idea > what's happening. This is a good suggestion. It would also potentially be enlightening to attach gdb to the process at various points in time and get a backtrace from all running threads. Other than that, there is little you can do on a running version. If it were compiled for debug, and the environment setup so that we could capture stdout/stderr, sending SIGUSR2 would yield to much the same information as the gdb backtrace BUT when runtime instrumentation is enabled, the build is operating at a third to a quarter of its normal speed. > in the meantime you need to stop the system from getting further > behind, > can you redirect or reconfigure the senders so that they are not > sending > new logs to this box so that it can dig itself out (stopping the input > may > be enough by itself, rsyslog has historicly done a LOT of locking > around > the main queue, and if you have a full queue with more data trying to > be > delivered it can really slow things down. I wouldn't expect it to be > this > much, but it could be part of it) > This may clean up things, but I really doubt it, given the magnitude of delays. Also, Aaron runs 4.4.5, which already has lots of the locking removed/restructure (not to compare with the recent v5-devel, but much more effcient than early v4 and v3). Rainer > David Lang > > > (We're up to 18:46:51 now!) > > > > -Aaron > > > > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards > > wrote: > >> mhhh... This is obviously not intended behavior. There are some > rate-limiting > >> features that can deliberately slow down a queue, but I guess you > have not > >> configured these. So there either is a bug that activates them at > some point > >> during processing (e.g. an invalid memory access could do that) or > there is > >> some other bug that causes the dequeue to happen very slowly. In any > case, it > >> is hard to guess. > >> > >> Given the volume of the messages, a debug log probably does not > help. We > >> could gain a little insight in at least the queue sizes that rsyslog > sees via > >> imdiag plus the (very rudamentary) v5 debug front-end (it doesn't > require the > >> engine to be v5!). I would need to spend at least a little work on > the > >> front-end to enable remote access, but that's not really a problem. > >> > >> One other thing is that I am still holding a v4-beta with additional > fixes as > >> I didn't want to put these in the middle of some other debugging. > However, > >> these fixes address potential memory problems, so the most > appropriate course > >> of action would be to give that version at least a try. It needs to > be > >> released in the next days in any case. > >> > >> I have uploaded that (pre-4.5.6) version so that you can give it a > try if you > >> like: > >> > >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz > >> > >> I think it would good if you could try it at least once, because I > think any > >> other troubleshooting will boil down to looking at the fixes this > version > >> contains. > >> > >> Rainer > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > >>> Sent: Monday, November 02, 2009 11:52 PM > >>> To: rsyslog-users > >>> Subject: [rsyslog] Queuing bug (4.5.5) > >>> > >>> Greetings all, > >>> > >>> I appear to have run into an issue with 4.5.5 where queues are not > >>> being flushed in a timely manner. ?In this specific case, I have > data > >>> from last wednesday that is being written to disk in small chunks > >>> today since last wednesday. ?Unfortunately I cannot be too specific > in > >>> details, but here is what I can see: > >>> > >>> Using omfile+gzip, there appears to have been a decent burst in > >>> traffic sometime last wednesday. ?The rsyslog instance has grown to > >>> 1.8GB of memory and stayed there for a while. ?I have both main > >>> message and action queues defined using fixedarray, and I see no > >>> on-disk queues (appears to be entirely in memory). > >>> > >>> I've got templates writing out to filenames using an hourly > timestamp > >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of > those > >>> files, all of them less than 5k in size, there are between 5 and 15 > >>> lines of content, all of them from last wednesday, and within a few > >>> seconds of each other. ?It's almost like there was a significant > queue > >>> built up, the hour rolled over, and only the first block of lines > were > >>> pulled from the queue. ?Then the hour rolled over again, and > another > >>> block of lines were pulled from the queue. ?Then the next hour, > then > >>> another 5-15 lines. > >>> > >>> Is it possible that one of the queues still has a good chunk of > data > >>> built up, and is flushing it out very slowly? ?It hasn't been > >>> consistantly at the top of the hour either, and not every hour. > ?But > >>> the log lines themselves are sequentially written out, and usually > the > >>> lines are within a few seconds of each other. > >>> > >>> For example: > >>> > >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 > 18:46:34 > >>> and one 18:46:35 > >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, 14 > >>> lines Oct 21 18:46:36 > >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, 4 > >>> lines Oct 21 18:46:37 > >>> > >>> Thoughts? > >>> -Aaron > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > From epiphani at gmail.com Tue Nov 3 18:27:07 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 3 Nov 2009 12:27:07 -0500 Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> Message-ID: Ok - I captured an strace. This appears to be the action thread. This specific set of calls took minutes. I'll re-run this with -t and/or -T. Note that this syslog instance is not actually receiving any data right now. -Aaron 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL 26365 <... futex resumed> ) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL 26365 <... futex resumed> ) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL 26365 <... futex resumed> ) = 0 26365 clock_gettime(CLOCK_REALTIME, 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 26365 <... futex resumed> ) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL 26365 <... futex resumed> ) = 0 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL 26365 <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable) 26365 clock_gettime(CLOCK_REALTIME, 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 26365 <... futex resumed> ) = 0 26365 clock_gettime(CLOCK_REALTIME, 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Tuesday, November 03, 2009 4:06 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >> >> > This is still taking place this hour - though I obviously can't >> > restart onto a newer version without losing this case. ?Is there >> > anything I can do in the current configuration to try and debug this >> > situation? >> >> if you can strace each thread for a few seconds it may give you an idea >> what's happening. > > This is a good suggestion. > > It would also potentially be enlightening to attach gdb to the process at > various points in time and get a backtrace from all running threads. > > Other than that, there is little you can do on a running version. If it were > compiled for debug, and the environment setup so that we could capture > stdout/stderr, sending SIGUSR2 would yield to much the same information as > the gdb backtrace BUT when runtime instrumentation is enabled, the build is > operating at a third to a quarter of its normal speed. > >> in the meantime you need to stop the system from getting further >> behind, >> can you redirect or reconfigure the senders so that they are not >> sending >> new logs to this box so that it can dig itself out (stopping the input >> may >> be enough by itself, rsyslog has historicly done a LOT of locking >> around >> the main queue, and if you have a full queue with more data trying to >> be >> delivered it can really slow things down. I wouldn't expect it to be >> this >> much, but it could be part of it) >> > > This may clean up things, but I really doubt it, given the magnitude of > delays. Also, Aaron runs 4.4.5, which already has lots of the locking > removed/restructure (not to compare with the recent v5-devel, but much more > effcient than early v4 and v3). > > Rainer >> David Lang >> >> > (We're up to 18:46:51 now!) >> > >> > -Aaron >> > >> > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >> > wrote: >> >> mhhh... This is obviously not intended behavior. There are some >> rate-limiting >> >> features that can deliberately slow down a queue, but I guess you >> have not >> >> configured these. So there either is a bug that activates them at >> some point >> >> during processing (e.g. an invalid memory access could do that) or >> there is >> >> some other bug that causes the dequeue to happen very slowly. In any >> case, it >> >> is hard to guess. >> >> >> >> Given the volume of the messages, a debug log probably does not >> help. We >> >> could gain a little insight in at least the queue sizes that rsyslog >> sees via >> >> imdiag plus the (very rudamentary) v5 debug front-end (it doesn't >> require the >> >> engine to be v5!). I would need to spend at least a little work on >> the >> >> front-end to enable remote access, but that's not really a problem. >> >> >> >> One other thing is that I am still holding a v4-beta with additional >> fixes as >> >> I didn't want to put these in the middle of some other debugging. >> However, >> >> these fixes address potential memory problems, so the most >> appropriate course >> >> of action would be to give that version at least a try. It needs to >> be >> >> released in the next days in any case. >> >> >> >> I have uploaded that (pre-4.5.6) version so that you can give it a >> try if you >> >> like: >> >> >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz >> >> >> >> I think it would good if you could try it at least once, because I >> think any >> >> other troubleshooting will boil down to looking at the fixes this >> version >> >> contains. >> >> >> >> Rainer >> >> >> >>> -----Original Message----- >> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> >>> Sent: Monday, November 02, 2009 11:52 PM >> >>> To: rsyslog-users >> >>> Subject: [rsyslog] Queuing bug (4.5.5) >> >>> >> >>> Greetings all, >> >>> >> >>> I appear to have run into an issue with 4.5.5 where queues are not >> >>> being flushed in a timely manner. ?In this specific case, I have >> data >> >>> from last wednesday that is being written to disk in small chunks >> >>> today since last wednesday. ?Unfortunately I cannot be too specific >> in >> >>> details, but here is what I can see: >> >>> >> >>> Using omfile+gzip, there appears to have been a decent burst in >> >>> traffic sometime last wednesday. ?The rsyslog instance has grown to >> >>> 1.8GB of memory and stayed there for a while. ?I have both main >> >>> message and action queues defined using fixedarray, and I see no >> >>> on-disk queues (appears to be entirely in memory). >> >>> >> >>> I've got templates writing out to filenames using an hourly >> timestamp >> >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of >> those >> >>> files, all of them less than 5k in size, there are between 5 and 15 >> >>> lines of content, all of them from last wednesday, and within a few >> >>> seconds of each other. ?It's almost like there was a significant >> queue >> >>> built up, the hour rolled over, and only the first block of lines >> were >> >>> pulled from the queue. ?Then the hour rolled over again, and >> another >> >>> block of lines were pulled from the queue. ?Then the next hour, >> then >> >>> another 5-15 lines. >> >>> >> >>> Is it possible that one of the queues still has a good chunk of >> data >> >>> built up, and is flushing it out very slowly? ?It hasn't been >> >>> consistantly at the top of the hour either, and not every hour. >> ?But >> >>> the log lines themselves are sequentially written out, and usually >> the >> >>> lines are within a few seconds of each other. >> >>> >> >>> For example: >> >>> >> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >> 18:46:34 >> >>> and one 18:46:35 >> >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, 14 >> >>> lines Oct 21 18:46:36 >> >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, 4 >> >>> lines Oct 21 18:46:37 >> >>> >> >>> Thoughts? >> >>> -Aaron >> >>> _______________________________________________ >> >>> rsyslog mailing list >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>> http://www.rsyslog.com >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com >> >> >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 3 18:30:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 18:30:17 +0100 Subject: [rsyslog] Queuing bug (4.5.5) References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> first shot at the data: this looks like a full queue where the enqueue operation is waiting on a drain that does not happen (fast enough). Of course, that doesn't explain why this happens... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Tuesday, November 03, 2009 6:27 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) > > Ok - I captured an strace. This appears to be the action thread. > This specific set of calls took minutes. I'll re-run this with -t > and/or -T. > > Note that this syslog instance is not actually receiving any data right > now. > > -Aaron > > 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- > 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL > 26365 <... futex resumed> ) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} > 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) > 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL > 26365 <... futex resumed> ) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} > 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) > 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL > 26365 <... futex resumed> ) = 0 > 26365 clock_gettime(CLOCK_REALTIME, > 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 > 26365 <... futex resumed> ) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} > 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) > 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL > 26365 <... futex resumed> ) = 0 > 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL > 26365 <... futex resumed> ) = -1 EAGAIN (Resource > temporarily unavailable) > 26365 clock_gettime(CLOCK_REALTIME, > 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 > 26365 <... futex resumed> ) = 0 > 26365 clock_gettime(CLOCK_REALTIME, > 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} > 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) > 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL > > > On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards > wrote: > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >> Sent: Tuesday, November 03, 2009 4:06 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Queuing bug (4.5.5) > >> > >> On Tue, 3 Nov 2009, Aaron Wiebe wrote: > >> > >> > This is still taking place this hour - though I obviously can't > >> > restart onto a newer version without losing this case. ?Is there > >> > anything I can do in the current configuration to try and debug > this > >> > situation? > >> > >> if you can strace each thread for a few seconds it may give you an > idea > >> what's happening. > > > > This is a good suggestion. > > > > It would also potentially be enlightening to attach gdb to the > process at > > various points in time and get a backtrace from all running threads. > > > > Other than that, there is little you can do on a running version. If > it were > > compiled for debug, and the environment setup so that we could > capture > > stdout/stderr, sending SIGUSR2 would yield to much the same > information as > > the gdb backtrace BUT when runtime instrumentation is enabled, the > build is > > operating at a third to a quarter of its normal speed. > > > >> in the meantime you need to stop the system from getting further > >> behind, > >> can you redirect or reconfigure the senders so that they are not > >> sending > >> new logs to this box so that it can dig itself out (stopping the > input > >> may > >> be enough by itself, rsyslog has historicly done a LOT of locking > >> around > >> the main queue, and if you have a full queue with more data trying > to > >> be > >> delivered it can really slow things down. I wouldn't expect it to be > >> this > >> much, but it could be part of it) > >> > > > > This may clean up things, but I really doubt it, given the magnitude > of > > delays. Also, Aaron runs 4.4.5, which already has lots of the locking > > removed/restructure (not to compare with the recent v5-devel, but > much more > > effcient than early v4 and v3). > > > > Rainer > >> David Lang > >> > >> > (We're up to 18:46:51 now!) > >> > > >> > -Aaron > >> > > >> > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards > >> > wrote: > >> >> mhhh... This is obviously not intended behavior. There are some > >> rate-limiting > >> >> features that can deliberately slow down a queue, but I guess you > >> have not > >> >> configured these. So there either is a bug that activates them at > >> some point > >> >> during processing (e.g. an invalid memory access could do that) > or > >> there is > >> >> some other bug that causes the dequeue to happen very slowly. In > any > >> case, it > >> >> is hard to guess. > >> >> > >> >> Given the volume of the messages, a debug log probably does not > >> help. We > >> >> could gain a little insight in at least the queue sizes that > rsyslog > >> sees via > >> >> imdiag plus the (very rudamentary) v5 debug front-end (it doesn't > >> require the > >> >> engine to be v5!). I would need to spend at least a little work > on > >> the > >> >> front-end to enable remote access, but that's not really a > problem. > >> >> > >> >> One other thing is that I am still holding a v4-beta with > additional > >> fixes as > >> >> I didn't want to put these in the middle of some other debugging. > >> However, > >> >> these fixes address potential memory problems, so the most > >> appropriate course > >> >> of action would be to give that version at least a try. It needs > to > >> be > >> >> released in the next days in any case. > >> >> > >> >> I have uploaded that (pre-4.5.6) version so that you can give it > a > >> try if you > >> >> like: > >> >> > >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz > >> >> > >> >> I think it would good if you could try it at least once, because > I > >> think any > >> >> other troubleshooting will boil down to looking at the fixes this > >> version > >> >> contains. > >> >> > >> >> Rainer > >> >> > >> >>> -----Original Message----- > >> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > >> >>> Sent: Monday, November 02, 2009 11:52 PM > >> >>> To: rsyslog-users > >> >>> Subject: [rsyslog] Queuing bug (4.5.5) > >> >>> > >> >>> Greetings all, > >> >>> > >> >>> I appear to have run into an issue with 4.5.5 where queues are > not > >> >>> being flushed in a timely manner. ?In this specific case, I have > >> data > >> >>> from last wednesday that is being written to disk in small > chunks > >> >>> today since last wednesday. ?Unfortunately I cannot be too > specific > >> in > >> >>> details, but here is what I can see: > >> >>> > >> >>> Using omfile+gzip, there appears to have been a decent burst in > >> >>> traffic sometime last wednesday. ?The rsyslog instance has grown > to > >> >>> 1.8GB of memory and stayed there for a while. ?I have both main > >> >>> message and action queues defined using fixedarray, and I see no > >> >>> on-disk queues (appears to be entirely in memory). > >> >>> > >> >>> I've got templates writing out to filenames using an hourly > >> timestamp > >> >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of > >> those > >> >>> files, all of them less than 5k in size, there are between 5 and > 15 > >> >>> lines of content, all of them from last wednesday, and within a > few > >> >>> seconds of each other. ?It's almost like there was a significant > >> queue > >> >>> built up, the hour rolled over, and only the first block of > lines > >> were > >> >>> pulled from the queue. ?Then the hour rolled over again, and > >> another > >> >>> block of lines were pulled from the queue. ?Then the next hour, > >> then > >> >>> another 5-15 lines. > >> >>> > >> >>> Is it possible that one of the queues still has a good chunk of > >> data > >> >>> built up, and is flushing it out very slowly? ?It hasn't been > >> >>> consistantly at the top of the hour either, and not every hour. > >> ?But > >> >>> the log lines themselves are sequentially written out, and > usually > >> the > >> >>> lines are within a few seconds of each other. > >> >>> > >> >>> For example: > >> >>> > >> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 > >> 18:46:34 > >> >>> and one 18:46:35 > >> >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, > 14 > >> >>> lines Oct 21 18:46:36 > >> >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, > 4 > >> >>> lines Oct 21 18:46:37 > >> >>> > >> >>> Thoughts? > >> >>> -Aaron > >> >>> _______________________________________________ > >> >>> rsyslog mailing list > >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >>> http://www.rsyslog.com > >> >> _______________________________________________ > >> >> rsyslog mailing list > >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> http://www.rsyslog.com > >> >> > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com > >> > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From epiphani at gmail.com Tue Nov 3 18:33:40 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 3 Nov 2009 12:33:40 -0500 Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> Message-ID: Here's one with times: 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, 3889000}) = 0 <0.000047> 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, 4147000}) = 0 <0.000013> 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) <2.002462> 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 250) = 250 <0.000305> 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, 7292000}) = 0 <0.000027> 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, 414932000}) = 0 <0.000048> 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, 415191000}) = 0 <0.000013> 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) <2.002220> 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 197) = 197 <0.000279> 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, 417992000}) = 0 <0.000027> 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, 608226000}) = 0 <0.000047> 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, 608486000}) = 0 <0.000014> 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} It looks like the locks are waiting a -very- long time. -Aaron On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards wrote: > first shot at the data: this looks like a full queue where the enqueue > operation is waiting on a drain that does not happen (fast enough). Of > course, that doesn't explain why this happens... > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> Sent: Tuesday, November 03, 2009 6:27 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> Ok - I captured an strace. ?This appears to be the action thread. >> This specific set of calls took minutes. ?I'll re-run this with -t >> and/or -T. >> >> Note that this syslog instance is not actually receiving any data right >> now. >> >> -Aaron >> >> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> timed out) >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> timed out) >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, ? >> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> timed out) >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource >> temporarily unavailable) >> 26365 clock_gettime(CLOCK_REALTIME, ? >> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, ? >> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> timed out) >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL >> >> >> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards >> wrote: >> >> -----Original Message----- >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >> Sent: Tuesday, November 03, 2009 4:06 PM >> >> To: rsyslog-users >> >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> >> >> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >> >> >> >> > This is still taking place this hour - though I obviously can't >> >> > restart onto a newer version without losing this case. ?Is there >> >> > anything I can do in the current configuration to try and debug >> this >> >> > situation? >> >> >> >> if you can strace each thread for a few seconds it may give you an >> idea >> >> what's happening. >> > >> > This is a good suggestion. >> > >> > It would also potentially be enlightening to attach gdb to the >> process at >> > various points in time and get a backtrace from all running threads. >> > >> > Other than that, there is little you can do on a running version. If >> it were >> > compiled for debug, and the environment setup so that we could >> capture >> > stdout/stderr, sending SIGUSR2 would yield to much the same >> information as >> > the gdb backtrace BUT when runtime instrumentation is enabled, the >> build is >> > operating at a third to a quarter of its normal speed. >> > >> >> in the meantime you need to stop the system from getting further >> >> behind, >> >> can you redirect or reconfigure the senders so that they are not >> >> sending >> >> new logs to this box so that it can dig itself out (stopping the >> input >> >> may >> >> be enough by itself, rsyslog has historicly done a LOT of locking >> >> around >> >> the main queue, and if you have a full queue with more data trying >> to >> >> be >> >> delivered it can really slow things down. I wouldn't expect it to be >> >> this >> >> much, but it could be part of it) >> >> >> > >> > This may clean up things, but I really doubt it, given the magnitude >> of >> > delays. Also, Aaron runs 4.4.5, which already has lots of the locking >> > removed/restructure (not to compare with the recent v5-devel, but >> much more >> > effcient than early v4 and v3). >> > >> > Rainer >> >> David Lang >> >> >> >> > (We're up to 18:46:51 now!) >> >> > >> >> > -Aaron >> >> > >> >> > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >> >> > wrote: >> >> >> mhhh... This is obviously not intended behavior. There are some >> >> rate-limiting >> >> >> features that can deliberately slow down a queue, but I guess you >> >> have not >> >> >> configured these. So there either is a bug that activates them at >> >> some point >> >> >> during processing (e.g. an invalid memory access could do that) >> or >> >> there is >> >> >> some other bug that causes the dequeue to happen very slowly. In >> any >> >> case, it >> >> >> is hard to guess. >> >> >> >> >> >> Given the volume of the messages, a debug log probably does not >> >> help. We >> >> >> could gain a little insight in at least the queue sizes that >> rsyslog >> >> sees via >> >> >> imdiag plus the (very rudamentary) v5 debug front-end (it doesn't >> >> require the >> >> >> engine to be v5!). I would need to spend at least a little work >> on >> >> the >> >> >> front-end to enable remote access, but that's not really a >> problem. >> >> >> >> >> >> One other thing is that I am still holding a v4-beta with >> additional >> >> fixes as >> >> >> I didn't want to put these in the middle of some other debugging. >> >> However, >> >> >> these fixes address potential memory problems, so the most >> >> appropriate course >> >> >> of action would be to give that version at least a try. It needs >> to >> >> be >> >> >> released in the next days in any case. >> >> >> >> >> >> I have uploaded that (pre-4.5.6) version so that you can give it >> a >> >> try if you >> >> >> like: >> >> >> >> >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz >> >> >> >> >> >> I think it would good if you could try it at least once, because >> I >> >> think any >> >> >> other troubleshooting will boil down to looking at the fixes this >> >> version >> >> >> contains. >> >> >> >> >> >> Rainer >> >> >> >> >> >>> -----Original Message----- >> >> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> >> >>> Sent: Monday, November 02, 2009 11:52 PM >> >> >>> To: rsyslog-users >> >> >>> Subject: [rsyslog] Queuing bug (4.5.5) >> >> >>> >> >> >>> Greetings all, >> >> >>> >> >> >>> I appear to have run into an issue with 4.5.5 where queues are >> not >> >> >>> being flushed in a timely manner. ?In this specific case, I have >> >> data >> >> >>> from last wednesday that is being written to disk in small >> chunks >> >> >>> today since last wednesday. ?Unfortunately I cannot be too >> specific >> >> in >> >> >>> details, but here is what I can see: >> >> >>> >> >> >>> Using omfile+gzip, there appears to have been a decent burst in >> >> >>> traffic sometime last wednesday. ?The rsyslog instance has grown >> to >> >> >>> 1.8GB of memory and stayed there for a while. ?I have both main >> >> >>> message and action queues defined using fixedarray, and I see no >> >> >>> on-disk queues (appears to be entirely in memory). >> >> >>> >> >> >>> I've got templates writing out to filenames using an hourly >> >> timestamp >> >> >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of >> >> those >> >> >>> files, all of them less than 5k in size, there are between 5 and >> 15 >> >> >>> lines of content, all of them from last wednesday, and within a >> few >> >> >>> seconds of each other. ?It's almost like there was a significant >> >> queue >> >> >>> built up, the hour rolled over, and only the first block of >> lines >> >> were >> >> >>> pulled from the queue. ?Then the hour rolled over again, and >> >> another >> >> >>> block of lines were pulled from the queue. ?Then the next hour, >> >> then >> >> >>> another 5-15 lines. >> >> >>> >> >> >>> Is it possible that one of the queues still has a good chunk of >> >> data >> >> >>> built up, and is flushing it out very slowly? ?It hasn't been >> >> >>> consistantly at the top of the hour either, and not every hour. >> >> ?But >> >> >>> the log lines themselves are sequentially written out, and >> usually >> >> the >> >> >>> lines are within a few seconds of each other. >> >> >>> >> >> >>> For example: >> >> >>> >> >> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >> >> 18:46:34 >> >> >>> and one 18:46:35 >> >> >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, >> 14 >> >> >>> lines Oct 21 18:46:36 >> >> >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, >> 4 >> >> >>> lines Oct 21 18:46:37 >> >> >>> >> >> >>> Thoughts? >> >> >>> -Aaron >> >> >>> _______________________________________________ >> >> >>> rsyslog mailing list >> >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >>> http://www.rsyslog.com >> >> >> _______________________________________________ >> >> >> rsyslog mailing list >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> http://www.rsyslog.com >> >> >> >> >> > _______________________________________________ >> >> > rsyslog mailing list >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> > http://www.rsyslog.com >> >> > >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 3 18:38:40 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 18:38:40 +0100 Subject: [rsyslog] Queuing bug (4.5.5) References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> It really looks like a full queue. The 2-second wait is the default wait interval before discarding a message when a queue is full. So for some reason the action queue seems to think it is full, so that the main queue worker can no longer add any data to it. Out of the strace, I unfortunately do not see why this happens. If possible, it would be good to try the 4.5.6 release, as this may actually be caused by the memory bug that is solved in that release. If it doesn't help, we would need to think about a way to get more in-depth info when the problem happens. I have an idea, but need to check if it can be done. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Tuesday, November 03, 2009 6:34 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) > > Here's one with times: > > 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- > 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL > > 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> > 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, > 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, > 3889000}) = 0 <0.000047> > 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 > 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> > 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, > 4147000}) = 0 <0.000013> > 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} > > 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) <2.002462> > 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 > WT-GARULF6"..., 250) = 250 <0.000305> > 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> > 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, > 7292000}) = 0 <0.000027> > 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL > > 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> > 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, > 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, > 414932000}) = 0 <0.000048> > 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 > 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> > 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, > 415191000}) = 0 <0.000013> > 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} > > 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) <2.002220> > 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 > WT-GARULF6"..., 197) = 197 <0.000279> > 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> > 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, > 417992000}) = 0 <0.000027> > 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL > > 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> > 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, > 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, > 608226000}) = 0 <0.000047> > 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 > 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> > 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, > 608486000}) = 0 <0.000014> > 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} > > > It looks like the locks are waiting a -very- long time. > > -Aaron > > On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards > wrote: > > first shot at the data: this looks like a full queue where the > enqueue > > operation is waiting on a drain that does not happen (fast enough). > Of > > course, that doesn't explain why this happens... > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > >> Sent: Tuesday, November 03, 2009 6:27 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Queuing bug (4.5.5) > >> > >> Ok - I captured an strace. ?This appears to be the action thread. > >> This specific set of calls took minutes. ?I'll re-run this with -t > >> and/or -T. > >> > >> Note that this syslog instance is not actually receiving any data > right > >> now. > >> > >> -Aaron > >> > >> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- > >> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} ...> > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection > >> timed out) > >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} ...> > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection > >> timed out) > >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, ? > >> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} ...> > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection > >> timed out) > >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource > >> temporarily unavailable) > >> 26365 clock_gettime(CLOCK_REALTIME, ? > >> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, ? > >> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} ...> > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection > >> timed out) > >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL > >> > >> > >> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards > >> wrote: > >> >> -----Original Message----- > >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >> >> Sent: Tuesday, November 03, 2009 4:06 PM > >> >> To: rsyslog-users > >> >> Subject: Re: [rsyslog] Queuing bug (4.5.5) > >> >> > >> >> On Tue, 3 Nov 2009, Aaron Wiebe wrote: > >> >> > >> >> > This is still taking place this hour - though I obviously can't > >> >> > restart onto a newer version without losing this case. ?Is > there > >> >> > anything I can do in the current configuration to try and debug > >> this > >> >> > situation? > >> >> > >> >> if you can strace each thread for a few seconds it may give you > an > >> idea > >> >> what's happening. > >> > > >> > This is a good suggestion. > >> > > >> > It would also potentially be enlightening to attach gdb to the > >> process at > >> > various points in time and get a backtrace from all running > threads. > >> > > >> > Other than that, there is little you can do on a running version. > If > >> it were > >> > compiled for debug, and the environment setup so that we could > >> capture > >> > stdout/stderr, sending SIGUSR2 would yield to much the same > >> information as > >> > the gdb backtrace BUT when runtime instrumentation is enabled, the > >> build is > >> > operating at a third to a quarter of its normal speed. > >> > > >> >> in the meantime you need to stop the system from getting further > >> >> behind, > >> >> can you redirect or reconfigure the senders so that they are not > >> >> sending > >> >> new logs to this box so that it can dig itself out (stopping the > >> input > >> >> may > >> >> be enough by itself, rsyslog has historicly done a LOT of locking > >> >> around > >> >> the main queue, and if you have a full queue with more data > trying > >> to > >> >> be > >> >> delivered it can really slow things down. I wouldn't expect it to > be > >> >> this > >> >> much, but it could be part of it) > >> >> > >> > > >> > This may clean up things, but I really doubt it, given the > magnitude > >> of > >> > delays. Also, Aaron runs 4.4.5, which already has lots of the > locking > >> > removed/restructure (not to compare with the recent v5-devel, but > >> much more > >> > effcient than early v4 and v3). > >> > > >> > Rainer > >> >> David Lang > >> >> > >> >> > (We're up to 18:46:51 now!) > >> >> > > >> >> > -Aaron > >> >> > > >> >> > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards > >> >> > wrote: > >> >> >> mhhh... This is obviously not intended behavior. There are > some > >> >> rate-limiting > >> >> >> features that can deliberately slow down a queue, but I guess > you > >> >> have not > >> >> >> configured these. So there either is a bug that activates them > at > >> >> some point > >> >> >> during processing (e.g. an invalid memory access could do > that) > >> or > >> >> there is > >> >> >> some other bug that causes the dequeue to happen very slowly. > In > >> any > >> >> case, it > >> >> >> is hard to guess. > >> >> >> > >> >> >> Given the volume of the messages, a debug log probably does > not > >> >> help. We > >> >> >> could gain a little insight in at least the queue sizes that > >> rsyslog > >> >> sees via > >> >> >> imdiag plus the (very rudamentary) v5 debug front-end (it > doesn't > >> >> require the > >> >> >> engine to be v5!). I would need to spend at least a little > work > >> on > >> >> the > >> >> >> front-end to enable remote access, but that's not really a > >> problem. > >> >> >> > >> >> >> One other thing is that I am still holding a v4-beta with > >> additional > >> >> fixes as > >> >> >> I didn't want to put these in the middle of some other > debugging. > >> >> However, > >> >> >> these fixes address potential memory problems, so the most > >> >> appropriate course > >> >> >> of action would be to give that version at least a try. It > needs > >> to > >> >> be > >> >> >> released in the next days in any case. > >> >> >> > >> >> >> I have uploaded that (pre-4.5.6) version so that you can give > it > >> a > >> >> try if you > >> >> >> like: > >> >> >> > >> >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog- > 4.5.6.tar.gz > >> >> >> > >> >> >> I think it would good if you could try it at least once, > because > >> I > >> >> think any > >> >> >> other troubleshooting will boil down to looking at the fixes > this > >> >> version > >> >> >> contains. > >> >> >> > >> >> >> Rainer > >> >> >> > >> >> >>> -----Original Message----- > >> >> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> >> >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > >> >> >>> Sent: Monday, November 02, 2009 11:52 PM > >> >> >>> To: rsyslog-users > >> >> >>> Subject: [rsyslog] Queuing bug (4.5.5) > >> >> >>> > >> >> >>> Greetings all, > >> >> >>> > >> >> >>> I appear to have run into an issue with 4.5.5 where queues > are > >> not > >> >> >>> being flushed in a timely manner. ?In this specific case, I > have > >> >> data > >> >> >>> from last wednesday that is being written to disk in small > >> chunks > >> >> >>> today since last wednesday. ?Unfortunately I cannot be too > >> specific > >> >> in > >> >> >>> details, but here is what I can see: > >> >> >>> > >> >> >>> Using omfile+gzip, there appears to have been a decent burst > in > >> >> >>> traffic sometime last wednesday. ?The rsyslog instance has > grown > >> to > >> >> >>> 1.8GB of memory and stayed there for a while. ?I have both > main > >> >> >>> message and action queues defined using fixedarray, and I see > no > >> >> >>> on-disk queues (appears to be entirely in memory). > >> >> >>> > >> >> >>> I've got templates writing out to filenames using an hourly > >> >> timestamp > >> >> >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some > of > >> >> those > >> >> >>> files, all of them less than 5k in size, there are between 5 > and > >> 15 > >> >> >>> lines of content, all of them from last wednesday, and within > a > >> few > >> >> >>> seconds of each other. ?It's almost like there was a > significant > >> >> queue > >> >> >>> built up, the hour rolled over, and only the first block of > >> lines > >> >> were > >> >> >>> pulled from the queue. ?Then the hour rolled over again, and > >> >> another > >> >> >>> block of lines were pulled from the queue. ?Then the next > hour, > >> >> then > >> >> >>> another 5-15 lines. > >> >> >>> > >> >> >>> Is it possible that one of the queues still has a good chunk > of > >> >> data > >> >> >>> built up, and is flushing it out very slowly? ?It hasn't been > >> >> >>> consistantly at the top of the hour either, and not every > hour. > >> >> ?But > >> >> >>> the log lines themselves are sequentially written out, and > >> usually > >> >> the > >> >> >>> lines are within a few seconds of each other. > >> >> >>> > >> >> >>> For example: > >> >> >>> > >> >> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 > >> >> 18:46:34 > >> >> >>> and one 18:46:35 > >> >> >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 > 18:46:35, > >> 14 > >> >> >>> lines Oct 21 18:46:36 > >> >> >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 > 18:46:36, > >> 4 > >> >> >>> lines Oct 21 18:46:37 > >> >> >>> > >> >> >>> Thoughts? > >> >> >>> -Aaron > >> >> >>> _______________________________________________ > >> >> >>> rsyslog mailing list > >> >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> >>> http://www.rsyslog.com > >> >> >> _______________________________________________ > >> >> >> rsyslog mailing list > >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> >> http://www.rsyslog.com > >> >> >> > >> >> > _______________________________________________ > >> >> > rsyslog mailing list > >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> > http://www.rsyslog.com > >> >> > > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com > >> > > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From epiphani at gmail.com Tue Nov 3 18:41:46 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 3 Nov 2009 12:41:46 -0500 Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> Message-ID: I'll see what I can do. I'm actually somewhat tied up with other stuff this week, but I may be able to find time to update. -Aaron On Tue, Nov 3, 2009 at 12:38 PM, Rainer Gerhards wrote: > It really looks like a full queue. The 2-second wait is the default wait > interval before discarding a message when a queue is full. So for some reason > the action queue seems to think it is full, so that the main queue worker can > no longer add any data to it. Out of the strace, I unfortunately do not see > why this happens. > > If possible, it would be good to try the 4.5.6 release, as this may actually > be caused by the memory bug that is solved in that release. If it doesn't > help, we would need to think about a way to get more in-depth info when the > problem happens. I have an idea, but need to check if it can be done. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> Sent: Tuesday, November 03, 2009 6:34 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> Here's one with times: >> >> 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >> 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL >> >> 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> >> 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, ? >> 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, >> 3889000}) = 0 <0.000047> >> 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> >> 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, >> 4147000}) = 0 <0.000013> >> 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} >> >> 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection >> timed out) <2.002462> >> 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 >> WT-GARULF6"..., 250) = 250 <0.000305> >> 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> >> 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, >> 7292000}) = 0 <0.000027> >> 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL >> >> 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> >> 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, ? >> 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, >> 414932000}) = 0 <0.000048> >> 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> >> 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, >> 415191000}) = 0 <0.000013> >> 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} >> >> 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection >> timed out) <2.002220> >> 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 >> WT-GARULF6"..., 197) = 197 <0.000279> >> 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> >> 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, >> 417992000}) = 0 <0.000027> >> 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL >> >> 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> >> 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, ? >> 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, >> 608226000}) = 0 <0.000047> >> 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> >> 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, >> 608486000}) = 0 <0.000014> >> 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} >> >> >> It looks like the locks are waiting a -very- long time. >> >> -Aaron >> >> On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards >> wrote: >> > first shot at the data: this looks like a full queue where the >> enqueue >> > operation is waiting on a drain that does not happen (fast enough). >> Of >> > course, that doesn't explain why this happens... >> > >> > Rainer >> > >> >> -----Original Message----- >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> >> Sent: Tuesday, November 03, 2009 6:27 PM >> >> To: rsyslog-users >> >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> >> >> Ok - I captured an strace. ?This appears to be the action thread. >> >> This specific set of calls took minutes. ?I'll re-run this with -t >> >> and/or -T. >> >> >> >> Note that this syslog instance is not actually receiving any data >> right >> >> now. >> >> >> >> -Aaron >> >> >> >> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} > ...> >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> >> timed out) >> >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} > ...> >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> >> timed out) >> >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, ? >> >> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} > ...> >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> >> timed out) >> >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource >> >> temporarily unavailable) >> >> 26365 clock_gettime(CLOCK_REALTIME, ? >> >> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, ? >> >> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} > ...> >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> >> timed out) >> >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL >> >> >> >> >> >> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards >> >> wrote: >> >> >> -----Original Message----- >> >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >> >> Sent: Tuesday, November 03, 2009 4:06 PM >> >> >> To: rsyslog-users >> >> >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> >> >> >> >> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >> >> >> >> >> >> > This is still taking place this hour - though I obviously can't >> >> >> > restart onto a newer version without losing this case. ?Is >> there >> >> >> > anything I can do in the current configuration to try and debug >> >> this >> >> >> > situation? >> >> >> >> >> >> if you can strace each thread for a few seconds it may give you >> an >> >> idea >> >> >> what's happening. >> >> > >> >> > This is a good suggestion. >> >> > >> >> > It would also potentially be enlightening to attach gdb to the >> >> process at >> >> > various points in time and get a backtrace from all running >> threads. >> >> > >> >> > Other than that, there is little you can do on a running version. >> If >> >> it were >> >> > compiled for debug, and the environment setup so that we could >> >> capture >> >> > stdout/stderr, sending SIGUSR2 would yield to much the same >> >> information as >> >> > the gdb backtrace BUT when runtime instrumentation is enabled, the >> >> build is >> >> > operating at a third to a quarter of its normal speed. >> >> > >> >> >> in the meantime you need to stop the system from getting further >> >> >> behind, >> >> >> can you redirect or reconfigure the senders so that they are not >> >> >> sending >> >> >> new logs to this box so that it can dig itself out (stopping the >> >> input >> >> >> may >> >> >> be enough by itself, rsyslog has historicly done a LOT of locking >> >> >> around >> >> >> the main queue, and if you have a full queue with more data >> trying >> >> to >> >> >> be >> >> >> delivered it can really slow things down. I wouldn't expect it to >> be >> >> >> this >> >> >> much, but it could be part of it) >> >> >> >> >> > >> >> > This may clean up things, but I really doubt it, given the >> magnitude >> >> of >> >> > delays. Also, Aaron runs 4.4.5, which already has lots of the >> locking >> >> > removed/restructure (not to compare with the recent v5-devel, but >> >> much more >> >> > effcient than early v4 and v3). >> >> > >> >> > Rainer >> >> >> David Lang >> >> >> >> >> >> > (We're up to 18:46:51 now!) >> >> >> > >> >> >> > -Aaron >> >> >> > >> >> >> > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >> >> >> > wrote: >> >> >> >> mhhh... This is obviously not intended behavior. There are >> some >> >> >> rate-limiting >> >> >> >> features that can deliberately slow down a queue, but I guess >> you >> >> >> have not >> >> >> >> configured these. So there either is a bug that activates them >> at >> >> >> some point >> >> >> >> during processing (e.g. an invalid memory access could do >> that) >> >> or >> >> >> there is >> >> >> >> some other bug that causes the dequeue to happen very slowly. >> In >> >> any >> >> >> case, it >> >> >> >> is hard to guess. >> >> >> >> >> >> >> >> Given the volume of the messages, a debug log probably does >> not >> >> >> help. We >> >> >> >> could gain a little insight in at least the queue sizes that >> >> rsyslog >> >> >> sees via >> >> >> >> imdiag plus the (very rudamentary) v5 debug front-end (it >> doesn't >> >> >> require the >> >> >> >> engine to be v5!). I would need to spend at least a little >> work >> >> on >> >> >> the >> >> >> >> front-end to enable remote access, but that's not really a >> >> problem. >> >> >> >> >> >> >> >> One other thing is that I am still holding a v4-beta with >> >> additional >> >> >> fixes as >> >> >> >> I didn't want to put these in the middle of some other >> debugging. >> >> >> However, >> >> >> >> these fixes address potential memory problems, so the most >> >> >> appropriate course >> >> >> >> of action would be to give that version at least a try. It >> needs >> >> to >> >> >> be >> >> >> >> released in the next days in any case. >> >> >> >> >> >> >> >> I have uploaded that (pre-4.5.6) version so that you can give >> it >> >> a >> >> >> try if you >> >> >> >> like: >> >> >> >> >> >> >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog- >> 4.5.6.tar.gz >> >> >> >> >> >> >> >> I think it would good if you could try it at least once, >> because >> >> I >> >> >> think any >> >> >> >> other troubleshooting will boil down to looking at the fixes >> this >> >> >> version >> >> >> >> contains. >> >> >> >> >> >> >> >> Rainer >> >> >> >> >> >> >> >>> -----Original Message----- >> >> >> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> >> >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> >> >> >>> Sent: Monday, November 02, 2009 11:52 PM >> >> >> >>> To: rsyslog-users >> >> >> >>> Subject: [rsyslog] Queuing bug (4.5.5) >> >> >> >>> >> >> >> >>> Greetings all, >> >> >> >>> >> >> >> >>> I appear to have run into an issue with 4.5.5 where queues >> are >> >> not >> >> >> >>> being flushed in a timely manner. ?In this specific case, I >> have >> >> >> data >> >> >> >>> from last wednesday that is being written to disk in small >> >> chunks >> >> >> >>> today since last wednesday. ?Unfortunately I cannot be too >> >> specific >> >> >> in >> >> >> >>> details, but here is what I can see: >> >> >> >>> >> >> >> >>> Using omfile+gzip, there appears to have been a decent burst >> in >> >> >> >>> traffic sometime last wednesday. ?The rsyslog instance has >> grown >> >> to >> >> >> >>> 1.8GB of memory and stayed there for a while. ?I have both >> main >> >> >> >>> message and action queues defined using fixedarray, and I see >> no >> >> >> >>> on-disk queues (appears to be entirely in memory). >> >> >> >>> >> >> >> >>> I've got templates writing out to filenames using an hourly >> >> >> timestamp >> >> >> >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some >> of >> >> >> those >> >> >> >>> files, all of them less than 5k in size, there are between 5 >> and >> >> 15 >> >> >> >>> lines of content, all of them from last wednesday, and within >> a >> >> few >> >> >> >>> seconds of each other. ?It's almost like there was a >> significant >> >> >> queue >> >> >> >>> built up, the hour rolled over, and only the first block of >> >> lines >> >> >> were >> >> >> >>> pulled from the queue. ?Then the hour rolled over again, and >> >> >> another >> >> >> >>> block of lines were pulled from the queue. ?Then the next >> hour, >> >> >> then >> >> >> >>> another 5-15 lines. >> >> >> >>> >> >> >> >>> Is it possible that one of the queues still has a good chunk >> of >> >> >> data >> >> >> >>> built up, and is flushing it out very slowly? ?It hasn't been >> >> >> >>> consistantly at the top of the hour either, and not every >> hour. >> >> >> ?But >> >> >> >>> the log lines themselves are sequentially written out, and >> >> usually >> >> >> the >> >> >> >>> lines are within a few seconds of each other. >> >> >> >>> >> >> >> >>> For example: >> >> >> >>> >> >> >> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >> >> >> 18:46:34 >> >> >> >>> and one 18:46:35 >> >> >> >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 >> 18:46:35, >> >> 14 >> >> >> >>> lines Oct 21 18:46:36 >> >> >> >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 >> 18:46:36, >> >> 4 >> >> >> >>> lines Oct 21 18:46:37 >> >> >> >>> >> >> >> >>> Thoughts? >> >> >> >>> -Aaron >> >> >> >>> _______________________________________________ >> >> >> >>> rsyslog mailing list >> >> >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> >>> http://www.rsyslog.com >> >> >> >> _______________________________________________ >> >> >> >> rsyslog mailing list >> >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> >> http://www.rsyslog.com >> >> >> >> >> >> >> > _______________________________________________ >> >> >> > rsyslog mailing list >> >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> > http://www.rsyslog.com >> >> >> > >> >> > _______________________________________________ >> >> > rsyslog mailing list >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> > http://www.rsyslog.com >> >> > >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From Luis.Fernando.Munoz.Mejias at cern.ch Tue Nov 3 19:04:36 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?utf-8?q?Mu=C3=B1oz_Mej=C3=ADas?=) Date: Tue, 3 Nov 2009 19:04:36 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5? Message-ID: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> Hello, world I must be doing something really dumb. I need to send some output to a named pipe that I have already created, but rsyslog is refusing to open it. My configuration states: *.* |/var/log/external/all/unknown;RSYSLOG_FileFormat And, of course, the named pipe exists: # ls -Z /var/log/external/all/unknown prw-r----- root logreaders user_u:object_r:var_log_t /var/log/external/all/unknown But when I start the syslog daemon it complains with this message: rsyslogd-2039: Could no open output file '|/var/log/external/all/unknown' [try http://www.rsyslog.com/e/2039 ] It may be a SELinux problem, but the logs show nothing about any denied requests. And the context for the named pipe is the exact same I have for the log files, which lie in the same directory as well. Am I the only one experiencing such problems? Thanks. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From david at lang.hm Tue Nov 3 19:20:57 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 3 Nov 2009 10:20:57 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> Message-ID: what is the configuration? does it have seperate action queues defined? if you have some way to stop the input, it would let the main queue drop down from being full (and eliminate the input modules from needing to lock it to try and deliver messages) I did see some times during my testing of 4.x stuff where the throughput would drop drasticly when the queue was full and there ws more incoming traffic hammering on it. it would drop from 500K logs/min to ~3k logs/min with a trivial config, slowing down the input would let it get out of this mode and speed up again after a little while. so much other stuff was happening at the time that I don't think that I reported it. David Lang On Tue, 3 Nov 2009, Rainer Gerhards wrote: > Date: Tue, 3 Nov 2009 18:38:40 +0100 > From: Rainer Gerhards > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) > > It really looks like a full queue. The 2-second wait is the default wait > interval before discarding a message when a queue is full. So for some reason > the action queue seems to think it is full, so that the main queue worker can > no longer add any data to it. Out of the strace, I unfortunately do not see > why this happens. > > If possible, it would be good to try the 4.5.6 release, as this may actually > be caused by the memory bug that is solved in that release. If it doesn't > help, we would need to think about a way to get more in-depth info when the > problem happens. I have an idea, but need to check if it can be done. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> Sent: Tuesday, November 03, 2009 6:34 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> Here's one with times: >> >> 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >> 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL >> >> 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> >> 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, >> 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, >> 3889000}) = 0 <0.000047> >> 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> >> 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, >> 4147000}) = 0 <0.000013> >> 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} >> >> 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection >> timed out) <2.002462> >> 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 >> WT-GARULF6"..., 250) = 250 <0.000305> >> 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> >> 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, >> 7292000}) = 0 <0.000027> >> 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL >> >> 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> >> 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, >> 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, >> 414932000}) = 0 <0.000048> >> 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> >> 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, >> 415191000}) = 0 <0.000013> >> 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} >> >> 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection >> timed out) <2.002220> >> 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 >> WT-GARULF6"..., 197) = 197 <0.000279> >> 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> >> 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, >> 417992000}) = 0 <0.000027> >> 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL >> >> 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> >> 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, >> 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, >> 608226000}) = 0 <0.000047> >> 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> >> 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, >> 608486000}) = 0 <0.000014> >> 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} >> >> >> It looks like the locks are waiting a -very- long time. >> >> -Aaron >> >> On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards >> wrote: >>> first shot at the data: this looks like a full queue where the >> enqueue >>> operation is waiting on a drain that does not happen (fast enough). >> Of >>> course, that doesn't explain why this happens... >>> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>> Sent: Tuesday, November 03, 2009 6:27 PM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>> >>>> Ok - I captured an strace. ?This appears to be the action thread. >>>> This specific set of calls took minutes. ?I'll re-run this with -t >>>> and/or -T. >>>> >>>> Note that this syslog instance is not actually receiving any data >> right >>>> now. >>>> >>>> -Aaron >>>> >>>> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} > ...> >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>> timed out) >>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} > ...> >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>> timed out) >>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} > ...> >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>> timed out) >>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource >>>> temporarily unavailable) >>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} > ...> >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>> timed out) >>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL >>>> >>>> >>>> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards >>>> wrote: >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>> Sent: Tuesday, November 03, 2009 4:06 PM >>>>>> To: rsyslog-users >>>>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>>>> >>>>>> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >>>>>> >>>>>>> This is still taking place this hour - though I obviously can't >>>>>>> restart onto a newer version without losing this case. ?Is >> there >>>>>>> anything I can do in the current configuration to try and debug >>>> this >>>>>>> situation? >>>>>> >>>>>> if you can strace each thread for a few seconds it may give you >> an >>>> idea >>>>>> what's happening. >>>>> >>>>> This is a good suggestion. >>>>> >>>>> It would also potentially be enlightening to attach gdb to the >>>> process at >>>>> various points in time and get a backtrace from all running >> threads. >>>>> >>>>> Other than that, there is little you can do on a running version. >> If >>>> it were >>>>> compiled for debug, and the environment setup so that we could >>>> capture >>>>> stdout/stderr, sending SIGUSR2 would yield to much the same >>>> information as >>>>> the gdb backtrace BUT when runtime instrumentation is enabled, the >>>> build is >>>>> operating at a third to a quarter of its normal speed. >>>>> >>>>>> in the meantime you need to stop the system from getting further >>>>>> behind, >>>>>> can you redirect or reconfigure the senders so that they are not >>>>>> sending >>>>>> new logs to this box so that it can dig itself out (stopping the >>>> input >>>>>> may >>>>>> be enough by itself, rsyslog has historicly done a LOT of locking >>>>>> around >>>>>> the main queue, and if you have a full queue with more data >> trying >>>> to >>>>>> be >>>>>> delivered it can really slow things down. I wouldn't expect it to >> be >>>>>> this >>>>>> much, but it could be part of it) >>>>>> >>>>> >>>>> This may clean up things, but I really doubt it, given the >> magnitude >>>> of >>>>> delays. Also, Aaron runs 4.4.5, which already has lots of the >> locking >>>>> removed/restructure (not to compare with the recent v5-devel, but >>>> much more >>>>> effcient than early v4 and v3). >>>>> >>>>> Rainer >>>>>> David Lang >>>>>> >>>>>>> (We're up to 18:46:51 now!) >>>>>>> >>>>>>> -Aaron >>>>>>> >>>>>>> On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >>>>>>> wrote: >>>>>>>> mhhh... This is obviously not intended behavior. There are >> some >>>>>> rate-limiting >>>>>>>> features that can deliberately slow down a queue, but I guess >> you >>>>>> have not >>>>>>>> configured these. So there either is a bug that activates them >> at >>>>>> some point >>>>>>>> during processing (e.g. an invalid memory access could do >> that) >>>> or >>>>>> there is >>>>>>>> some other bug that causes the dequeue to happen very slowly. >> In >>>> any >>>>>> case, it >>>>>>>> is hard to guess. >>>>>>>> >>>>>>>> Given the volume of the messages, a debug log probably does >> not >>>>>> help. We >>>>>>>> could gain a little insight in at least the queue sizes that >>>> rsyslog >>>>>> sees via >>>>>>>> imdiag plus the (very rudamentary) v5 debug front-end (it >> doesn't >>>>>> require the >>>>>>>> engine to be v5!). I would need to spend at least a little >> work >>>> on >>>>>> the >>>>>>>> front-end to enable remote access, but that's not really a >>>> problem. >>>>>>>> >>>>>>>> One other thing is that I am still holding a v4-beta with >>>> additional >>>>>> fixes as >>>>>>>> I didn't want to put these in the middle of some other >> debugging. >>>>>> However, >>>>>>>> these fixes address potential memory problems, so the most >>>>>> appropriate course >>>>>>>> of action would be to give that version at least a try. It >> needs >>>> to >>>>>> be >>>>>>>> released in the next days in any case. >>>>>>>> >>>>>>>> I have uploaded that (pre-4.5.6) version so that you can give >> it >>>> a >>>>>> try if you >>>>>>>> like: >>>>>>>> >>>>>>>> http://www.rsyslog.com/download/rsyslog/pre/rsyslog- >> 4.5.6.tar.gz >>>>>>>> >>>>>>>> I think it would good if you could try it at least once, >> because >>>> I >>>>>> think any >>>>>>>> other troubleshooting will boil down to looking at the fixes >> this >>>>>> version >>>>>>>> contains. >>>>>>>> >>>>>>>> Rainer >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>>>>>>> Sent: Monday, November 02, 2009 11:52 PM >>>>>>>>> To: rsyslog-users >>>>>>>>> Subject: [rsyslog] Queuing bug (4.5.5) >>>>>>>>> >>>>>>>>> Greetings all, >>>>>>>>> >>>>>>>>> I appear to have run into an issue with 4.5.5 where queues >> are >>>> not >>>>>>>>> being flushed in a timely manner. ?In this specific case, I >> have >>>>>> data >>>>>>>>> from last wednesday that is being written to disk in small >>>> chunks >>>>>>>>> today since last wednesday. ?Unfortunately I cannot be too >>>> specific >>>>>> in >>>>>>>>> details, but here is what I can see: >>>>>>>>> >>>>>>>>> Using omfile+gzip, there appears to have been a decent burst >> in >>>>>>>>> traffic sometime last wednesday. ?The rsyslog instance has >> grown >>>> to >>>>>>>>> 1.8GB of memory and stayed there for a while. ?I have both >> main >>>>>>>>> message and action queues defined using fixedarray, and I see >> no >>>>>>>>> on-disk queues (appears to be entirely in memory). >>>>>>>>> >>>>>>>>> I've got templates writing out to filenames using an hourly >>>>>> timestamp >>>>>>>>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some >> of >>>>>> those >>>>>>>>> files, all of them less than 5k in size, there are between 5 >> and >>>> 15 >>>>>>>>> lines of content, all of them from last wednesday, and within >> a >>>> few >>>>>>>>> seconds of each other. ?It's almost like there was a >> significant >>>>>> queue >>>>>>>>> built up, the hour rolled over, and only the first block of >>>> lines >>>>>> were >>>>>>>>> pulled from the queue. ?Then the hour rolled over again, and >>>>>> another >>>>>>>>> block of lines were pulled from the queue. ?Then the next >> hour, >>>>>> then >>>>>>>>> another 5-15 lines. >>>>>>>>> >>>>>>>>> Is it possible that one of the queues still has a good chunk >> of >>>>>> data >>>>>>>>> built up, and is flushing it out very slowly? ?It hasn't been >>>>>>>>> consistantly at the top of the hour either, and not every >> hour. >>>>>> ?But >>>>>>>>> the log lines themselves are sequentially written out, and >>>> usually >>>>>> the >>>>>>>>> lines are within a few seconds of each other. >>>>>>>>> >>>>>>>>> For example: >>>>>>>>> >>>>>>>>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >>>>>> 18:46:34 >>>>>>>>> and one 18:46:35 >>>>>>>>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 >> 18:46:35, >>>> 14 >>>>>>>>> lines Oct 21 18:46:36 >>>>>>>>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 >> 18:46:36, >>>> 4 >>>>>>>>> lines Oct 21 18:46:37 >>>>>>>>> >>>>>>>>> Thoughts? >>>>>>>>> -Aaron >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Tue Nov 3 19:52:12 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 3 Nov 2009 10:52:12 -0800 (PST) Subject: [rsyslog] queue configuration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103318@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103318@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 3 Nov 2009, Rainer Gerhards wrote: > David and all, > > I have created a new output plugin, omruleset, which permits to nest > rulesets. What it does is copy a message over from one ruleset to another and > make it process in parallel. Some more details in the doc: > > http://www.rsyslog.com/doc-omruleset.html this looks very interesting, however there is one thing mentioned here that concerns me. it sounds like you are saying that if you have multiple rules that write to the same file that you may get the content of various lines mixed togeather. am I understanding this correctly? if so, is there any way to work around this? David Lang > It permits to do what you asked for below. It also permits to do many more > things and create very advanced configurations. But one must be VERY careful > to receive the intended results. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Friday, October 23, 2009 10:23 PM >> To: rsyslog-users >> Subject: [rsyslog] queue configuration >> >> I know that you can create an action queue for a specific output. >> >> is there any way to create an action queue for multiple outputs? >> >> for example, in my configuration I have >> >> :fromhost, !isequal, "127.0.0.1" >> /var/log/messages;TraditionalFormat >> :fromhost, isequal, "127.0.0.1" >> @192.168.1.1;TraditionalForwardFormat >> *.* @192.168.1.115 >> *.* @192.168.1.241 >> *.* @192.168.1.242 >> *.* @192.168.1.6 >> *.* @192.168.1.7 >> *.* @192.168.1.122 >> :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 >> & @192.168.1.1;fixsnareForwardFormat2 >> & ~ >> :syslogtag, startswith, "MSWinEventLog#011" >> /var/log/messages;fixsnareFormat >> & @192.168.1.1;fixsnareForwardFormat >> & ~ >> *.* /var/log/messages;TraditionalFormat >> *.* @192.168.1.1 >> >> it would be nice to move the outbound relays off to a different queue, >> and >> to put the list of rules that have order dependancies (because they >> output >> in different formats to fix up formatting problem) to a seperate queue >> to >> spread the work across multiple processes and not slow down the basic >> writing to file. >> >> but as far as I can tell I would have to create a queue for each >> individual item, and I can't create a queue for the part of the config >> where I need to discard. >> >> am I missing something (including a better way to do everything ;-) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 3 20:20:38 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 20:20:38 +0100 Subject: [rsyslog] queue configuration Message-ID: <000201ca5cba$d08238ff$100013ac@intern.adiscon.com> Yes, thats right if you use the new buffered mode. The cure is either not to do that or use the new omruleset. ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 03.11.09 19:52 Betreff: Re: [rsyslog] queue configuration On Tue, 3 Nov 2009, Rainer Gerhards wrote: > David and all, > > I have created a new output plugin, omruleset, which permits to nest > rulesets. What it does is copy a message over from one ruleset to another and > make it process in parallel. Some more details in the doc: > > http://www.rsyslog.com/doc-omruleset.html this looks very interesting, however there is one thing mentioned here that concerns me. it sounds like you are saying that if you have multiple rules that write to the same file that you may get the content of various lines mixed togeather. am I understanding this correctly? if so, is there any way to work around this? David Lang > It permits to do what you asked for below. It also permits to do many more > things and create very advanced configurations. But one must be VERY careful > to receive the intended results. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Friday, October 23, 2009 10:23 PM >> To: rsyslog-users >> Subject: [rsyslog] queue configuration >> >> I know that you can create an action queue for a specific output. >> >> is there any way to create an action queue for multiple outputs? >> >> for example, in my configuration I have >> >> :fromhost, !isequal, "127.0.0.1" >> /var/log/messages;TraditionalFormat >> :fromhost, isequal, "127.0.0.1" >> @192.168.1.1;TraditionalForwardFormat >> *.* @192.168.1.115 >> *.* @192.168.1.241 >> *.* @192.168.1.242 >> *.* @192.168.1.6 >> *.* @192.168.1.7 >> *.* @192.168.1.122 >> :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 >> & @192.168.1.1;fixsnareForwardFormat2 >> & ~ >> :syslogtag, startswith, "MSWinEventLog#011" >> /var/log/messages;fixsnareFormat >> & @192.168.1.1;fixsnareForwardFormat >> & ~ >> *.* /var/log/messages;TraditionalFormat >> *.* @192.168.1.1 >> >> it would be nice to move the outbound relays off to a different queue, >> and >> to put the list of rules that have order dependancies (because they >> output >> in different formats to fix up formatting problem) to a seperate queue >> to >> spread the work across multiple processes and not slow down the basic >> writing to file. >> >> but as far as I can tell I would have to create a queue for each >> individual item, and I can't create a queue for the part of the config >> where I need to discard. >> >> am I missing something (including a better way to do everything ;-) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Tue Nov 3 20:29:30 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 3 Nov 2009 11:29:30 -0800 (PST) Subject: [rsyslog] queue configuration In-Reply-To: <000201ca5cba$d08238ff$100013ac@intern.adiscon.com> References: <000201ca5cba$d08238ff$100013ac@intern.adiscon.com> Message-ID: On Tue, 3 Nov 2009, Rainer Gerhards wrote: > Yes, thats right if you use the new buffered mode. The cure is either not to do that or use the new omruleset. so the new buffered mode even without seperate action queues or omruleset can have this problem? David Lang > ----- Urspr?ngliche Nachricht ----- > Von: "david at lang.hm" > An: "rsyslog-users" > Gesendet: 03.11.09 19:52 > Betreff: Re: [rsyslog] queue configuration > > On Tue, 3 Nov 2009, Rainer Gerhards wrote: > >> David and all, >> >> I have created a new output plugin, omruleset, which permits to nest >> rulesets. What it does is copy a message over from one ruleset to another and >> make it process in parallel. Some more details in the doc: >> >> http://www.rsyslog.com/doc-omruleset.html > > this looks very interesting, however there is one thing mentioned here > that concerns me. > > it sounds like you are saying that if you have multiple rules that write > to the same file that you may get the content of various lines mixed > togeather. > > am I understanding this correctly? > > if so, is there any way to work around this? > > David Lang > >> It permits to do what you asked for below. It also permits to do many more >> things and create very advanced configurations. But one must be VERY careful >> to receive the intended results. >> >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Friday, October 23, 2009 10:23 PM >>> To: rsyslog-users >>> Subject: [rsyslog] queue configuration >>> >>> I know that you can create an action queue for a specific output. >>> >>> is there any way to create an action queue for multiple outputs? >>> >>> for example, in my configuration I have >>> >>> :fromhost, !isequal, "127.0.0.1" >>> /var/log/messages;TraditionalFormat >>> :fromhost, isequal, "127.0.0.1" >>> @192.168.1.1;TraditionalForwardFormat >>> *.* @192.168.1.115 >>> *.* @192.168.1.241 >>> *.* @192.168.1.242 >>> *.* @192.168.1.6 >>> *.* @192.168.1.7 >>> *.* @192.168.1.122 >>> :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 >>> & @192.168.1.1;fixsnareForwardFormat2 >>> & ~ >>> :syslogtag, startswith, "MSWinEventLog#011" >>> /var/log/messages;fixsnareFormat >>> & @192.168.1.1;fixsnareForwardFormat >>> & ~ >>> *.* /var/log/messages;TraditionalFormat >>> *.* @192.168.1.1 >>> >>> it would be nice to move the outbound relays off to a different queue, >>> and >>> to put the list of rules that have order dependancies (because they >>> output >>> in different formats to fix up formatting problem) to a seperate queue >>> to >>> spread the work across multiple processes and not slow down the basic >>> writing to file. >>> >>> but as far as I can tell I would have to create a queue for each >>> individual item, and I can't create a queue for the part of the config >>> where I need to discard. >>> >>> am I missing something (including a better way to do everything ;-) >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 3 21:25:04 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 21:25:04 +0100 Subject: [rsyslog] queue configuration Message-ID: <000301ca5cc3$d0e60e48$100013ac@intern.adiscon.com> Think about this: actions are independent of each other. If different actions write to the same file, they do that independently. If these actions now buffer before they write, they do that independently. Consequently, the writes get mixed up. Without buffering, each os write is done seperately, so the OS keeps each of the records appended to the file. ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 03.11.09 20:30 Betreff: Re: [rsyslog] queue configuration On Tue, 3 Nov 2009, Rainer Gerhards wrote: > Yes, thats right if you use the new buffered mode. The cure is either not to do that or use the new omruleset. so the new buffered mode even without seperate action queues or omruleset can have this problem? David Lang > ----- Urspr?ngliche Nachricht ----- > Von: "david at lang.hm" > An: "rsyslog-users" > Gesendet: 03.11.09 19:52 > Betreff: Re: [rsyslog] queue configuration > > On Tue, 3 Nov 2009, Rainer Gerhards wrote: > >> David and all, >> >> I have created a new output plugin, omruleset, which permits to nest >> rulesets. What it does is copy a message over from one ruleset to another and >> make it process in parallel. Some more details in the doc: >> >> http://www.rsyslog.com/doc-omruleset.html > > this looks very interesting, however there is one thing mentioned here > that concerns me. > > it sounds like you are saying that if you have multiple rules that write > to the same file that you may get the content of various lines mixed > togeather. > > am I understanding this correctly? > > if so, is there any way to work around this? > > David Lang > >> It permits to do what you asked for below. It also permits to do many more >> things and create very advanced configurations. But one must be VERY careful >> to receive the intended results. >> >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Friday, October 23, 2009 10:23 PM >>> To: rsyslog-users >>> Subject: [rsyslog] queue configuration >>> >>> I know that you can create an action queue for a specific output. >>> >>> is there any way to create an action queue for multiple outputs? >>> >>> for example, in my configuration I have >>> >>> :fromhost, !isequal, "127.0.0.1" >>> /var/log/messages;TraditionalFormat >>> :fromhost, isequal, "127.0.0.1" >>> @192.168.1.1;TraditionalForwardFormat >>> *.* @192.168.1.115 >>> *.* @192.168.1.241 >>> *.* @192.168.1.242 >>> *.* @192.168.1.6 >>> *.* @192.168.1.7 >>> *.* @192.168.1.122 >>> :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 >>> & @192.168.1.1;fixsnareForwardFormat2 >>> & ~ >>> :syslogtag, startswith, "MSWinEventLog#011" >>> /var/log/messages;fixsnareFormat >>> & @192.168.1.1;fixsnareForwardFormat >>> & ~ >>> *.* /var/log/messages;TraditionalFormat >>> *.* @192.168.1.1 >>> >>> it would be nice to move the outbound relays off to a different queue, >>> and >>> to put the list of rules that have order dependancies (because they >>> output >>> in different formats to fix up formatting problem) to a seperate queue >>> to >>> spread the work across multiple processes and not slow down the basic >>> writing to file. >>> >>> but as far as I can tell I would have to create a queue for each >>> individual item, and I can't create a queue for the part of the config >>> where I need to discard. >>> >>> am I missing something (including a better way to do everything ;-) >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From epiphani at gmail.com Wed Nov 4 00:53:05 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 3 Nov 2009 18:53:05 -0500 Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> Message-ID: There is an action queue defined per rule set.. of which there are four. Here's the tricky part: input has been stopped. For days. The queues just seem to continue to flush very very slowly. I have a feeling that if I increased the traffic flow, the queues would flush faster. I'm not 100% sure about that though, and I can't test in this environment. Unfortunately, due to outside requirements, I'm going to have to pull rsyslog out of this deployment until we can sort out these problems... I'll work on replicating the issue in another environment, if I can. -Aaron On Tue, Nov 3, 2009 at 1:20 PM, wrote: > what is the configuration? does it have seperate action queues defined? > > if you have some way to stop the input, it would let the main queue drop > down from being full (and eliminate the input modules from needing to lock > it to try and deliver messages) > > I did see some times during my testing of 4.x stuff where the throughput > would drop drasticly when the queue was full and there ws more incoming > traffic hammering on it. it would drop from 500K logs/min to ~3k logs/min > with a trivial config, slowing down the input would let it get out of this > mode and speed up again after a little while. so much other stuff was > happening at the time that I don't think that I reported it. > > David Lang > > On Tue, 3 Nov 2009, Rainer Gerhards wrote: > >> Date: Tue, 3 Nov 2009 18:38:40 +0100 >> From: Rainer Gerhards >> Reply-To: rsyslog-users >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> It really looks like a full queue. The 2-second wait is the default wait >> interval before discarding a message when a queue is full. So for some >> reason >> the action queue seems to think it is full, so that the main queue worker >> can >> no longer add any data to it. Out of the strace, I unfortunately do not >> see >> why this happens. >> >> If possible, it would be good to try the 4.5.6 release, as this may >> actually >> be caused by the memory bug that is solved in that release. If it doesn't >> help, we would need to think about a way to get more in-depth info when >> the >> problem happens. I have an idea, but need to check if it can be done. >> >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>> Sent: Tuesday, November 03, 2009 6:34 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>> >>> Here's one with times: >>> >>> 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >>> 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL >>> >>> 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> >>> 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, ? >>> 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, >>> 3889000}) = 0 <0.000047> >>> 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 >>> 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> >>> 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, >>> 4147000}) = 0 <0.000013> >>> 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} >>> >>> 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection >>> timed out) <2.002462> >>> 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 >>> WT-GARULF6"..., 250) = 250 <0.000305> >>> 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> >>> 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, >>> 7292000}) = 0 <0.000027> >>> 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL >>> >>> 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> >>> 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, ? >>> 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, >>> 414932000}) = 0 <0.000048> >>> 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 >>> 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> >>> 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, >>> 415191000}) = 0 <0.000013> >>> 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} >>> >>> 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection >>> timed out) <2.002220> >>> 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 >>> WT-GARULF6"..., 197) = 197 <0.000279> >>> 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> >>> 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, >>> 417992000}) = 0 <0.000027> >>> 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL >>> >>> 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> >>> 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, ? >>> 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, >>> 608226000}) = 0 <0.000047> >>> 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 >>> 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> >>> 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, >>> 608486000}) = 0 <0.000014> >>> 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} >>> >>> >>> It looks like the locks are waiting a -very- long time. >>> >>> -Aaron >>> >>> On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards >>> wrote: >>>> >>>> first shot at the data: this looks like a full queue where the >>> >>> enqueue >>>> >>>> operation is waiting on a drain that does not happen (fast enough). >>> >>> Of >>>> >>>> course, that doesn't explain why this happens... >>>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>>> Sent: Tuesday, November 03, 2009 6:27 PM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>>> >>>>> Ok - I captured an strace. ?This appears to be the action thread. >>>>> This specific set of calls took minutes. ?I'll re-run this with -t >>>>> and/or -T. >>>>> >>>>> Note that this syslog instance is not actually receiving any data >>> >>> right >>>>> >>>>> now. >>>>> >>>>> -Aaron >>>>> >>>>> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} >> >>> ...> >>>>> >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>> timed out) >>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} >> >>> ...> >>>>> >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>> timed out) >>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} >> >>> ...> >>>>> >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>> timed out) >>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource >>>>> temporarily unavailable) >>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} >> >>> ...> >>>>> >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>> timed out) >>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL >>>>> >>>>> >>>>> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards >>>>> wrote: >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>> Sent: Tuesday, November 03, 2009 4:06 PM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>>>>> >>>>>>> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >>>>>>> >>>>>>>> This is still taking place this hour - though I obviously can't >>>>>>>> restart onto a newer version without losing this case. ?Is >>> >>> there >>>>>>>> >>>>>>>> anything I can do in the current configuration to try and debug >>>>> >>>>> this >>>>>>>> >>>>>>>> situation? >>>>>>> >>>>>>> if you can strace each thread for a few seconds it may give you >>> >>> an >>>>> >>>>> idea >>>>>>> >>>>>>> what's happening. >>>>>> >>>>>> This is a good suggestion. >>>>>> >>>>>> It would also potentially be enlightening to attach gdb to the >>>>> >>>>> process at >>>>>> >>>>>> various points in time and get a backtrace from all running >>> >>> threads. >>>>>> >>>>>> Other than that, there is little you can do on a running version. >>> >>> If >>>>> >>>>> it were >>>>>> >>>>>> compiled for debug, and the environment setup so that we could >>>>> >>>>> capture >>>>>> >>>>>> stdout/stderr, sending SIGUSR2 would yield to much the same >>>>> >>>>> information as >>>>>> >>>>>> the gdb backtrace BUT when runtime instrumentation is enabled, the >>>>> >>>>> build is >>>>>> >>>>>> operating at a third to a quarter of its normal speed. >>>>>> >>>>>>> in the meantime you need to stop the system from getting further >>>>>>> behind, >>>>>>> can you redirect or reconfigure the senders so that they are not >>>>>>> sending >>>>>>> new logs to this box so that it can dig itself out (stopping the >>>>> >>>>> input >>>>>>> >>>>>>> may >>>>>>> be enough by itself, rsyslog has historicly done a LOT of locking >>>>>>> around >>>>>>> the main queue, and if you have a full queue with more data >>> >>> trying >>>>> >>>>> to >>>>>>> >>>>>>> be >>>>>>> delivered it can really slow things down. I wouldn't expect it to >>> >>> be >>>>>>> >>>>>>> this >>>>>>> much, but it could be part of it) >>>>>>> >>>>>> >>>>>> This may clean up things, but I really doubt it, given the >>> >>> magnitude >>>>> >>>>> of >>>>>> >>>>>> delays. Also, Aaron runs 4.4.5, which already has lots of the >>> >>> locking >>>>>> >>>>>> removed/restructure (not to compare with the recent v5-devel, but >>>>> >>>>> much more >>>>>> >>>>>> effcient than early v4 and v3). >>>>>> >>>>>> Rainer >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>>> (We're up to 18:46:51 now!) >>>>>>>> >>>>>>>> -Aaron >>>>>>>> >>>>>>>> On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> mhhh... This is obviously not intended behavior. There are >>> >>> some >>>>>>> >>>>>>> rate-limiting >>>>>>>>> >>>>>>>>> features that can deliberately slow down a queue, but I guess >>> >>> you >>>>>>> >>>>>>> have not >>>>>>>>> >>>>>>>>> configured these. So there either is a bug that activates them >>> >>> at >>>>>>> >>>>>>> some point >>>>>>>>> >>>>>>>>> during processing (e.g. an invalid memory access could do >>> >>> that) >>>>> >>>>> or >>>>>>> >>>>>>> there is >>>>>>>>> >>>>>>>>> some other bug that causes the dequeue to happen very slowly. >>> >>> In >>>>> >>>>> any >>>>>>> >>>>>>> case, it >>>>>>>>> >>>>>>>>> is hard to guess. >>>>>>>>> >>>>>>>>> Given the volume of the messages, a debug log probably does >>> >>> not >>>>>>> >>>>>>> help. We >>>>>>>>> >>>>>>>>> could gain a little insight in at least the queue sizes that >>>>> >>>>> rsyslog >>>>>>> >>>>>>> sees via >>>>>>>>> >>>>>>>>> imdiag plus the (very rudamentary) v5 debug front-end (it >>> >>> doesn't >>>>>>> >>>>>>> require the >>>>>>>>> >>>>>>>>> engine to be v5!). I would need to spend at least a little >>> >>> work >>>>> >>>>> on >>>>>>> >>>>>>> the >>>>>>>>> >>>>>>>>> front-end to enable remote access, but that's not really a >>>>> >>>>> problem. >>>>>>>>> >>>>>>>>> One other thing is that I am still holding a v4-beta with >>>>> >>>>> additional >>>>>>> >>>>>>> fixes as >>>>>>>>> >>>>>>>>> I didn't want to put these in the middle of some other >>> >>> debugging. >>>>>>> >>>>>>> However, >>>>>>>>> >>>>>>>>> these fixes address potential memory problems, so the most >>>>>>> >>>>>>> appropriate course >>>>>>>>> >>>>>>>>> of action would be to give that version at least a try. It >>> >>> needs >>>>> >>>>> to >>>>>>> >>>>>>> be >>>>>>>>> >>>>>>>>> released in the next days in any case. >>>>>>>>> >>>>>>>>> I have uploaded that (pre-4.5.6) version so that you can give >>> >>> it >>>>> >>>>> a >>>>>>> >>>>>>> try if you >>>>>>>>> >>>>>>>>> like: >>>>>>>>> >>>>>>>>> http://www.rsyslog.com/download/rsyslog/pre/rsyslog- >>> >>> 4.5.6.tar.gz >>>>>>>>> >>>>>>>>> I think it would good if you could try it at least once, >>> >>> because >>>>> >>>>> I >>>>>>> >>>>>>> think any >>>>>>>>> >>>>>>>>> other troubleshooting will boil down to looking at the fixes >>> >>> this >>>>>>> >>>>>>> version >>>>>>>>> >>>>>>>>> contains. >>>>>>>>> >>>>>>>>> Rainer >>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>>>>>>>> Sent: Monday, November 02, 2009 11:52 PM >>>>>>>>>> To: rsyslog-users >>>>>>>>>> Subject: [rsyslog] Queuing bug (4.5.5) >>>>>>>>>> >>>>>>>>>> Greetings all, >>>>>>>>>> >>>>>>>>>> I appear to have run into an issue with 4.5.5 where queues >>> >>> are >>>>> >>>>> not >>>>>>>>>> >>>>>>>>>> being flushed in a timely manner. ?In this specific case, I >>> >>> have >>>>>>> >>>>>>> data >>>>>>>>>> >>>>>>>>>> from last wednesday that is being written to disk in small >>>>> >>>>> chunks >>>>>>>>>> >>>>>>>>>> today since last wednesday. ?Unfortunately I cannot be too >>>>> >>>>> specific >>>>>>> >>>>>>> in >>>>>>>>>> >>>>>>>>>> details, but here is what I can see: >>>>>>>>>> >>>>>>>>>> Using omfile+gzip, there appears to have been a decent burst >>> >>> in >>>>>>>>>> >>>>>>>>>> traffic sometime last wednesday. ?The rsyslog instance has >>> >>> grown >>>>> >>>>> to >>>>>>>>>> >>>>>>>>>> 1.8GB of memory and stayed there for a while. ?I have both >>> >>> main >>>>>>>>>> >>>>>>>>>> message and action queues defined using fixedarray, and I see >>> >>> no >>>>>>>>>> >>>>>>>>>> on-disk queues (appears to be entirely in memory). >>>>>>>>>> >>>>>>>>>> I've got templates writing out to filenames using an hourly >>>>>>> >>>>>>> timestamp >>>>>>>>>> >>>>>>>>>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some >>> >>> of >>>>>>> >>>>>>> those >>>>>>>>>> >>>>>>>>>> files, all of them less than 5k in size, there are between 5 >>> >>> and >>>>> >>>>> 15 >>>>>>>>>> >>>>>>>>>> lines of content, all of them from last wednesday, and within >>> >>> a >>>>> >>>>> few >>>>>>>>>> >>>>>>>>>> seconds of each other. ?It's almost like there was a >>> >>> significant >>>>>>> >>>>>>> queue >>>>>>>>>> >>>>>>>>>> built up, the hour rolled over, and only the first block of >>>>> >>>>> lines >>>>>>> >>>>>>> were >>>>>>>>>> >>>>>>>>>> pulled from the queue. ?Then the hour rolled over again, and >>>>>>> >>>>>>> another >>>>>>>>>> >>>>>>>>>> block of lines were pulled from the queue. ?Then the next >>> >>> hour, >>>>>>> >>>>>>> then >>>>>>>>>> >>>>>>>>>> another 5-15 lines. >>>>>>>>>> >>>>>>>>>> Is it possible that one of the queues still has a good chunk >>> >>> of >>>>>>> >>>>>>> data >>>>>>>>>> >>>>>>>>>> built up, and is flushing it out very slowly? ?It hasn't been >>>>>>>>>> consistantly at the top of the hour either, and not every >>> >>> hour. >>>>>>> >>>>>>> ?But >>>>>>>>>> >>>>>>>>>> the log lines themselves are sequentially written out, and >>>>> >>>>> usually >>>>>>> >>>>>>> the >>>>>>>>>> >>>>>>>>>> lines are within a few seconds of each other. >>>>>>>>>> >>>>>>>>>> For example: >>>>>>>>>> >>>>>>>>>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >>>>>>> >>>>>>> 18:46:34 >>>>>>>>>> >>>>>>>>>> and one 18:46:35 >>>>>>>>>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 >>> >>> 18:46:35, >>>>> >>>>> 14 >>>>>>>>>> >>>>>>>>>> lines Oct 21 18:46:36 >>>>>>>>>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 >>> >>> 18:46:36, >>>>> >>>>> 4 >>>>>>>>>> >>>>>>>>>> lines Oct 21 18:46:37 >>>>>>>>>> >>>>>>>>>> Thoughts? >>>>>>>>>> -Aaron >>>>>>>>>> _______________________________________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > From david at lang.hm Wed Nov 4 03:18:56 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 3 Nov 2009 18:18:56 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 3 Nov 2009, Aaron Wiebe wrote: > There is an action queue defined per rule set.. of which there are four. is it possible that something blocked one of the rule outputs? as I understand it, that would cause that action queue to fill up, and then the behavior of the main queue trying, then sleeping would make sense to me > Here's the tricky part: input has been stopped. For days. The > queues just seem to continue to flush very very slowly. I have a > feeling that if I increased the traffic flow, the queues would flush > faster. I'm not 100% sure about that though, and I can't test in this > environment. Unfortunately, due to outside requirements, I'm going to > have to pull rsyslog out of this deployment until we can sort out > these problems... did you configure this with HUPisRestart off? if so you may try sending it a HUP and see if that gets a few log messages out. if it does, hammer it with HUPs to drain the queue it is really a pain to run into problems only in production. David Lang > I'll work on replicating the issue in another environment, if I can. > > -Aaron > > On Tue, Nov 3, 2009 at 1:20 PM, wrote: >> what is the configuration? does it have seperate action queues defined? >> >> if you have some way to stop the input, it would let the main queue drop >> down from being full (and eliminate the input modules from needing to lock >> it to try and deliver messages) >> >> I did see some times during my testing of 4.x stuff where the throughput >> would drop drasticly when the queue was full and there ws more incoming >> traffic hammering on it. it would drop from 500K logs/min to ~3k logs/min >> with a trivial config, slowing down the input would let it get out of this >> mode and speed up again after a little while. so much other stuff was >> happening at the time that I don't think that I reported it. >> >> David Lang >> >> On Tue, 3 Nov 2009, Rainer Gerhards wrote: >> >>> Date: Tue, 3 Nov 2009 18:38:40 +0100 >>> From: Rainer Gerhards >>> Reply-To: rsyslog-users >>> To: rsyslog-users >>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>> >>> It really looks like a full queue. The 2-second wait is the default wait >>> interval before discarding a message when a queue is full. So for some >>> reason >>> the action queue seems to think it is full, so that the main queue worker >>> can >>> no longer add any data to it. Out of the strace, I unfortunately do not >>> see >>> why this happens. >>> >>> If possible, it would be good to try the 4.5.6 release, as this may >>> actually >>> be caused by the memory bug that is solved in that release. If it doesn't >>> help, we would need to think about a way to get more in-depth info when >>> the >>> problem happens. I have an idea, but need to check if it can be done. >>> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>> Sent: Tuesday, November 03, 2009 6:34 PM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>> >>>> Here's one with times: >>>> >>>> 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >>>> 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL >>>> >>>> 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> >>>> 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, >>>> 3889000}) = 0 <0.000047> >>>> 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>> 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> >>>> 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, >>>> 4147000}) = 0 <0.000013> >>>> 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} >>>> >>>> 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection >>>> timed out) <2.002462> >>>> 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 >>>> WT-GARULF6"..., 250) = 250 <0.000305> >>>> 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> >>>> 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, >>>> 7292000}) = 0 <0.000027> >>>> 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL >>>> >>>> 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> >>>> 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, >>>> 414932000}) = 0 <0.000048> >>>> 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>> 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> >>>> 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, >>>> 415191000}) = 0 <0.000013> >>>> 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} >>>> >>>> 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection >>>> timed out) <2.002220> >>>> 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 >>>> WT-GARULF6"..., 197) = 197 <0.000279> >>>> 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> >>>> 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, >>>> 417992000}) = 0 <0.000027> >>>> 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL >>>> >>>> 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> >>>> 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, >>>> 608226000}) = 0 <0.000047> >>>> 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>> 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> >>>> 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, >>>> 608486000}) = 0 <0.000014> >>>> 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} >>>> >>>> >>>> It looks like the locks are waiting a -very- long time. >>>> >>>> -Aaron >>>> >>>> On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards >>>> wrote: >>>>> >>>>> first shot at the data: this looks like a full queue where the >>>> >>>> enqueue >>>>> >>>>> operation is waiting on a drain that does not happen (fast enough). >>>> >>>> Of >>>>> >>>>> course, that doesn't explain why this happens... >>>>> >>>>> Rainer >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>>>> Sent: Tuesday, November 03, 2009 6:27 PM >>>>>> To: rsyslog-users >>>>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>>>> >>>>>> Ok - I captured an strace. ?This appears to be the action thread. >>>>>> This specific set of calls took minutes. ?I'll re-run this with -t >>>>>> and/or -T. >>>>>> >>>>>> Note that this syslog instance is not actually receiving any data >>>> >>>> right >>>>>> >>>>>> now. >>>>>> >>>>>> -Aaron >>>>>> >>>>>> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} >>> >>>> ...> >>>>>> >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>>> timed out) >>>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} >>> >>>> ...> >>>>>> >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>>> timed out) >>>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>>> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} >>> >>>> ...> >>>>>> >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>>> timed out) >>>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource >>>>>> temporarily unavailable) >>>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>>> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>>> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} >>> >>>> ...> >>>>>> >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>>> timed out) >>>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL >>>>>> >>>>>> >>>>>> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards >>>>>> wrote: >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>>> Sent: Tuesday, November 03, 2009 4:06 PM >>>>>>>> To: rsyslog-users >>>>>>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>>>>>> >>>>>>>> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >>>>>>>> >>>>>>>>> This is still taking place this hour - though I obviously can't >>>>>>>>> restart onto a newer version without losing this case. ?Is >>>> >>>> there >>>>>>>>> >>>>>>>>> anything I can do in the current configuration to try and debug >>>>>> >>>>>> this >>>>>>>>> >>>>>>>>> situation? >>>>>>>> >>>>>>>> if you can strace each thread for a few seconds it may give you >>>> >>>> an >>>>>> >>>>>> idea >>>>>>>> >>>>>>>> what's happening. >>>>>>> >>>>>>> This is a good suggestion. >>>>>>> >>>>>>> It would also potentially be enlightening to attach gdb to the >>>>>> >>>>>> process at >>>>>>> >>>>>>> various points in time and get a backtrace from all running >>>> >>>> threads. >>>>>>> >>>>>>> Other than that, there is little you can do on a running version. >>>> >>>> If >>>>>> >>>>>> it were >>>>>>> >>>>>>> compiled for debug, and the environment setup so that we could >>>>>> >>>>>> capture >>>>>>> >>>>>>> stdout/stderr, sending SIGUSR2 would yield to much the same >>>>>> >>>>>> information as >>>>>>> >>>>>>> the gdb backtrace BUT when runtime instrumentation is enabled, the >>>>>> >>>>>> build is >>>>>>> >>>>>>> operating at a third to a quarter of its normal speed. >>>>>>> >>>>>>>> in the meantime you need to stop the system from getting further >>>>>>>> behind, >>>>>>>> can you redirect or reconfigure the senders so that they are not >>>>>>>> sending >>>>>>>> new logs to this box so that it can dig itself out (stopping the >>>>>> >>>>>> input >>>>>>>> >>>>>>>> may >>>>>>>> be enough by itself, rsyslog has historicly done a LOT of locking >>>>>>>> around >>>>>>>> the main queue, and if you have a full queue with more data >>>> >>>> trying >>>>>> >>>>>> to >>>>>>>> >>>>>>>> be >>>>>>>> delivered it can really slow things down. I wouldn't expect it to >>>> >>>> be >>>>>>>> >>>>>>>> this >>>>>>>> much, but it could be part of it) >>>>>>>> >>>>>>> >>>>>>> This may clean up things, but I really doubt it, given the >>>> >>>> magnitude >>>>>> >>>>>> of >>>>>>> >>>>>>> delays. Also, Aaron runs 4.4.5, which already has lots of the >>>> >>>> locking >>>>>>> >>>>>>> removed/restructure (not to compare with the recent v5-devel, but >>>>>> >>>>>> much more >>>>>>> >>>>>>> effcient than early v4 and v3). >>>>>>> >>>>>>> Rainer >>>>>>>> >>>>>>>> David Lang >>>>>>>> >>>>>>>>> (We're up to 18:46:51 now!) >>>>>>>>> >>>>>>>>> -Aaron >>>>>>>>> >>>>>>>>> On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> mhhh... This is obviously not intended behavior. There are >>>> >>>> some >>>>>>>> >>>>>>>> rate-limiting >>>>>>>>>> >>>>>>>>>> features that can deliberately slow down a queue, but I guess >>>> >>>> you >>>>>>>> >>>>>>>> have not >>>>>>>>>> >>>>>>>>>> configured these. So there either is a bug that activates them >>>> >>>> at >>>>>>>> >>>>>>>> some point >>>>>>>>>> >>>>>>>>>> during processing (e.g. an invalid memory access could do >>>> >>>> that) >>>>>> >>>>>> or >>>>>>>> >>>>>>>> there is >>>>>>>>>> >>>>>>>>>> some other bug that causes the dequeue to happen very slowly. >>>> >>>> In >>>>>> >>>>>> any >>>>>>>> >>>>>>>> case, it >>>>>>>>>> >>>>>>>>>> is hard to guess. >>>>>>>>>> >>>>>>>>>> Given the volume of the messages, a debug log probably does >>>> >>>> not >>>>>>>> >>>>>>>> help. We >>>>>>>>>> >>>>>>>>>> could gain a little insight in at least the queue sizes that >>>>>> >>>>>> rsyslog >>>>>>>> >>>>>>>> sees via >>>>>>>>>> >>>>>>>>>> imdiag plus the (very rudamentary) v5 debug front-end (it >>>> >>>> doesn't >>>>>>>> >>>>>>>> require the >>>>>>>>>> >>>>>>>>>> engine to be v5!). I would need to spend at least a little >>>> >>>> work >>>>>> >>>>>> on >>>>>>>> >>>>>>>> the >>>>>>>>>> >>>>>>>>>> front-end to enable remote access, but that's not really a >>>>>> >>>>>> problem. >>>>>>>>>> >>>>>>>>>> One other thing is that I am still holding a v4-beta with >>>>>> >>>>>> additional >>>>>>>> >>>>>>>> fixes as >>>>>>>>>> >>>>>>>>>> I didn't want to put these in the middle of some other >>>> >>>> debugging. >>>>>>>> >>>>>>>> However, >>>>>>>>>> >>>>>>>>>> these fixes address potential memory problems, so the most >>>>>>>> >>>>>>>> appropriate course >>>>>>>>>> >>>>>>>>>> of action would be to give that version at least a try. It >>>> >>>> needs >>>>>> >>>>>> to >>>>>>>> >>>>>>>> be >>>>>>>>>> >>>>>>>>>> released in the next days in any case. >>>>>>>>>> >>>>>>>>>> I have uploaded that (pre-4.5.6) version so that you can give >>>> >>>> it >>>>>> >>>>>> a >>>>>>>> >>>>>>>> try if you >>>>>>>>>> >>>>>>>>>> like: >>>>>>>>>> >>>>>>>>>> http://www.rsyslog.com/download/rsyslog/pre/rsyslog- >>>> >>>> 4.5.6.tar.gz >>>>>>>>>> >>>>>>>>>> I think it would good if you could try it at least once, >>>> >>>> because >>>>>> >>>>>> I >>>>>>>> >>>>>>>> think any >>>>>>>>>> >>>>>>>>>> other troubleshooting will boil down to looking at the fixes >>>> >>>> this >>>>>>>> >>>>>>>> version >>>>>>>>>> >>>>>>>>>> contains. >>>>>>>>>> >>>>>>>>>> Rainer >>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>>>>>>>>> Sent: Monday, November 02, 2009 11:52 PM >>>>>>>>>>> To: rsyslog-users >>>>>>>>>>> Subject: [rsyslog] Queuing bug (4.5.5) >>>>>>>>>>> >>>>>>>>>>> Greetings all, >>>>>>>>>>> >>>>>>>>>>> I appear to have run into an issue with 4.5.5 where queues >>>> >>>> are >>>>>> >>>>>> not >>>>>>>>>>> >>>>>>>>>>> being flushed in a timely manner. ?In this specific case, I >>>> >>>> have >>>>>>>> >>>>>>>> data >>>>>>>>>>> >>>>>>>>>>> from last wednesday that is being written to disk in small >>>>>> >>>>>> chunks >>>>>>>>>>> >>>>>>>>>>> today since last wednesday. ?Unfortunately I cannot be too >>>>>> >>>>>> specific >>>>>>>> >>>>>>>> in >>>>>>>>>>> >>>>>>>>>>> details, but here is what I can see: >>>>>>>>>>> >>>>>>>>>>> Using omfile+gzip, there appears to have been a decent burst >>>> >>>> in >>>>>>>>>>> >>>>>>>>>>> traffic sometime last wednesday. ?The rsyslog instance has >>>> >>>> grown >>>>>> >>>>>> to >>>>>>>>>>> >>>>>>>>>>> 1.8GB of memory and stayed there for a while. ?I have both >>>> >>>> main >>>>>>>>>>> >>>>>>>>>>> message and action queues defined using fixedarray, and I see >>>> >>>> no >>>>>>>>>>> >>>>>>>>>>> on-disk queues (appears to be entirely in memory). >>>>>>>>>>> >>>>>>>>>>> I've got templates writing out to filenames using an hourly >>>>>>>> >>>>>>>> timestamp >>>>>>>>>>> >>>>>>>>>>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some >>>> >>>> of >>>>>>>> >>>>>>>> those >>>>>>>>>>> >>>>>>>>>>> files, all of them less than 5k in size, there are between 5 >>>> >>>> and >>>>>> >>>>>> 15 >>>>>>>>>>> >>>>>>>>>>> lines of content, all of them from last wednesday, and within >>>> >>>> a >>>>>> >>>>>> few >>>>>>>>>>> >>>>>>>>>>> seconds of each other. ?It's almost like there was a >>>> >>>> significant >>>>>>>> >>>>>>>> queue >>>>>>>>>>> >>>>>>>>>>> built up, the hour rolled over, and only the first block of >>>>>> >>>>>> lines >>>>>>>> >>>>>>>> were >>>>>>>>>>> >>>>>>>>>>> pulled from the queue. ?Then the hour rolled over again, and >>>>>>>> >>>>>>>> another >>>>>>>>>>> >>>>>>>>>>> block of lines were pulled from the queue. ?Then the next >>>> >>>> hour, >>>>>>>> >>>>>>>> then >>>>>>>>>>> >>>>>>>>>>> another 5-15 lines. >>>>>>>>>>> >>>>>>>>>>> Is it possible that one of the queues still has a good chunk >>>> >>>> of >>>>>>>> >>>>>>>> data >>>>>>>>>>> >>>>>>>>>>> built up, and is flushing it out very slowly? ?It hasn't been >>>>>>>>>>> consistantly at the top of the hour either, and not every >>>> >>>> hour. >>>>>>>> >>>>>>>> ?But >>>>>>>>>>> >>>>>>>>>>> the log lines themselves are sequentially written out, and >>>>>> >>>>>> usually >>>>>>>> >>>>>>>> the >>>>>>>>>>> >>>>>>>>>>> lines are within a few seconds of each other. >>>>>>>>>>> >>>>>>>>>>> For example: >>>>>>>>>>> >>>>>>>>>>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >>>>>>>> >>>>>>>> 18:46:34 >>>>>>>>>>> >>>>>>>>>>> and one 18:46:35 >>>>>>>>>>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 >>>> >>>> 18:46:35, >>>>>> >>>>>> 14 >>>>>>>>>>> >>>>>>>>>>> lines Oct 21 18:46:36 >>>>>>>>>>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 >>>> >>>> 18:46:36, >>>>>> >>>>>> 4 >>>>>>>>>>> >>>>>>>>>>> lines Oct 21 18:46:37 >>>>>>>>>>> >>>>>>>>>>> Thoughts? >>>>>>>>>>> -Aaron >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> rsyslog mailing list >>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>> http://www.rsyslog.com >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Nov 4 07:22:01 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 07:22:01 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5? References: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103323@GRFEXC.intern.adiscon.com> Hi, that actually could be a regression caused by the rewrite of omfile. I'll look at it as soon as I have time, but it probably is a good idea to file a bug tracker with bugzilla. I first need to finish what I am currently implementing (custom parser interface) and then I need to look into which bugs are open and have priority. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luis Fernando Mu?oz Mej?as > Sent: Tuesday, November 03, 2009 7:05 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Output to named pipes not working on v4.5.5? > > Hello, world > > I must be doing something really dumb. I need to send some output to a > named pipe that I have already created, but rsyslog is refusing to > open it. > > My configuration states: > > *.* |/var/log/external/all/unknown;RSYSLOG_FileFormat > > And, of course, the named pipe exists: > > # ls -Z /var/log/external/all/unknown > > prw-r----- root logreaders user_u:object_r:var_log_t > /var/log/external/all/unknown > > But when I start the syslog daemon it complains with this message: > > rsyslogd-2039: Could no open output file > '|/var/log/external/all/unknown' [try http://www.rsyslog.com/e/2039 ] > > It may be a SELinux problem, but the logs show nothing about any > denied requests. And the context for the named pipe is the exact same > I have for the log files, which lie in the same directory as well. > > Am I the only one experiencing such problems? > > Thanks. > -- > Luis Fernando Mu?oz Mej?as > Luis.Fernando.Munoz.Mejias at cern.ch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Nov 4 07:34:43 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 07:34:43 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> > it is really a pain to run into problems only in production. The problem with hunting such bugs is that we cannot get the diagnostic information we need. I have thought about a solution. Could you please comment if this one would be acceptable for your environment: I add the capability to send the debug log out via UDP syslog (not via the usual channels, but rather via a special send function inside the debug printf system). Then, when a problem occurs, we could enable the debug printf (which is present in production builds as well) and make it output the data to a remote server (or a special listener, even netcat, running on the local machine). I could turn on/off this via a signal (as we currently do for sending debug message to stdout). Alternatively, one could load imdiag into a suspect instance and use that to gain more in-depth information. However, I am not sure if that would be suitable for use in production, at least it is questionable from a security POV. Any feedback is appreciated. While I like to have better ability to dig into problems as they occur, I don't think it makes sense to invest time into coding something if it is unclear whether or not it could be utilized... So unless I get sufficient positive feedback, I'll stay away from implementing that. Rainer From correo at miguelangelnieto.net Wed Nov 4 10:16:42 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Wed, 4 Nov 2009 10:16:42 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory Message-ID: Hi, I have a problem with the attached client-configuration. When I stop the server, the client computer hangs-up some minutes later and didn't write logs on $WorkDirectory /var/log/queue. Where is the mistake? Thank you and sorry for my bad english :P -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From epiphani at gmail.com Wed Nov 4 14:09:29 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Wed, 4 Nov 2009 08:09:29 -0500 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> Message-ID: On Wed, Nov 4, 2009 at 1:34 AM, Rainer Gerhards wrote: > > I add the capability to send the debug log out via UDP syslog (not via the > usual channels, but rather via a special send function inside the debug > printf system). Then, when a problem occurs, we could enable the debug printf > (which is present in production builds as well) and make it output the data > to a remote server (or a special listener, even netcat, running on the local > machine). > > I could turn on/off this via a signal (as we currently do for sending debug > message to stdout). That could work - I could see that being an acceptable step for some of our environments here. > Alternatively, one could load imdiag into a suspect instance and use that to > gain more in-depth information. However, I am not sure if that would be > suitable for use in production, at least it is questionable from a security > POV. Is there more docs on imdiag and the enhancements you've made to it recently kicking around? I'll see if I can't make use if it in my efforts to reproduce the issue (which probably won't start until next week). -Aaron From rgerhards at hq.adiscon.com Wed Nov 4 14:13:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 14:13:56 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103333@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Wednesday, November 04, 2009 2:09 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) / Problem in production > > On Wed, Nov 4, 2009 at 1:34 AM, Rainer Gerhards > wrote: > > > > I add the capability to send the debug log out via UDP syslog (not > via the > > usual channels, but rather via a special send function inside the > debug > > printf system). Then, when a problem occurs, we could enable the > debug printf > > (which is present in production builds as well) and make it output > the data > > to a remote server (or a special listener, even netcat, running on > the local > > machine). > > > > I could turn on/off this via a signal (as we currently do for sending > debug > > message to stdout). > > That could work - I could see that being an acceptable step for some > of our environments here. OK, that sounds good. Will see what I can do (I think this could be generally useful). > > > Alternatively, one could load imdiag into a suspect instance and use > that to > > gain more in-depth information. However, I am not sure if that would > be > > suitable for use in production, at least it is questionable from a > security > > POV. > > Is there more docs on imdiag and the enhancements you've made to it > recently kicking around? I'll see if I can't make use if it in my > efforts to reproduce the issue (which probably won't start until next > week). No, actually not. Initially it was intended to be a thing mostly for Rainer and the testbench ;) I'll see that I do a write-up on production troubleshooting and include it. Any more feedback from anyone on this issue is appreciated. Rainer > > -Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From Luis.Fernando.Munoz.Mejias at cern.ch Wed Nov 4 14:14:34 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?utf-8?q?Mu=C3=B1oz_Mej=C3=ADas?=) Date: Wed, 4 Nov 2009 14:14:34 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5? In-Reply-To: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> References: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <200911041414.34449.Luis.Fernando.Munoz.Mejias@cern.ch> So, this has two parts: 1) A SELinux problem on RHEL5 and similar (but the logs show nothing!!). It seems the syslog context is not allowed to access FIFOs. I'll have to fix that. 2) Even when there are no permission problems, rsyslog 4.5.5 just doesn't write to a named pipe. From the following consecutive lines: *.* |/var/log/external/fifo *.* /var/log/external/file the file receives data, the FIFO doesn't. I'll paste all this to bugzilla. Cheers. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From david at lang.hm Wed Nov 4 14:28:57 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 05:28:57 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 4 Nov 2009, Rainer Gerhards wrote: >> it is really a pain to run into problems only in production. > > The problem with hunting such bugs is that we cannot get the diagnostic > information we need. I have thought about a solution. Could you please > comment if this one would be acceptable for your environment: > > I add the capability to send the debug log out via UDP syslog (not via the > usual channels, but rather via a special send function inside the debug > printf system). Then, when a problem occurs, we could enable the debug printf > (which is present in production builds as well) and make it output the data > to a remote server (or a special listener, even netcat, running on the local > machine). > > I could turn on/off this via a signal (as we currently do for sending debug > message to stdout). that would be a LOT easier than having to run in debug mode the entire time. would this have similar performance impact to running in debug mode? or does this need less serialization, and therefor less impact? > Alternatively, one could load imdiag into a suspect instance and use that to > gain more in-depth information. However, I am not sure if that would be > suitable for use in production, at least it is questionable from a security > POV. is loading this on the fly even possible? david Lang > Any feedback is appreciated. While I like to have better ability to dig into > problems as they occur, I don't think it makes sense to invest time into > coding something if it is unclear whether or not it could be utilized... So > unless I get sufficient positive feedback, I'll stay away from implementing > that. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Nov 4 14:32:22 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 05:32:22 -0800 (PST) Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > I have a problem with the attached client-configuration. When I stop > the server, the client computer hangs-up some minutes later and didn't > write logs on $WorkDirectory /var/log/queue. this list strips attachments, please re-send with the config in the body of the message. david Lang -------------- next part -------------- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Nov 4 14:33:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 14:33:53 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103334@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, November 04, 2009 2:29 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) / Problem in production > > On Wed, 4 Nov 2009, Rainer Gerhards wrote: > > >> it is really a pain to run into problems only in production. > > > > The problem with hunting such bugs is that we cannot get the > diagnostic > > information we need. I have thought about a solution. Could you > please > > comment if this one would be acceptable for your environment: > > > > I add the capability to send the debug log out via UDP syslog (not > via the > > usual channels, but rather via a special send function inside the > debug > > printf system). Then, when a problem occurs, we could enable the > debug printf > > (which is present in production builds as well) and make it output > the data > > to a remote server (or a special listener, even netcat, running on > the local > > machine). > > > > I could turn on/off this via a signal (as we currently do for sending > debug > > message to stdout). > > that would be a LOT easier than having to run in debug mode the entire > time. The problem is that you would not get the information BEFORE you turned in on. In theory, it should even be possible with the current system and a file, but I need to check this out (I never used it that way, but the engine probably is capable of doing it). > would this have similar performance impact to running in debug mode? or > does this need less serialization, and therefor less impact? Think of it as turning on debug mode on the fly. That exactly is what it does. So before the signal, there is no impact, but after it is the same impact as usual. > > Alternatively, one could load imdiag into a suspect instance and use > that to > > gain more in-depth information. However, I am not sure if that would > be > > suitable for use in production, at least it is questionable from a > security > > POV. > > is loading this on the fly even possible? NO, that's the point. If we intend to pursue that path, imdiag always needs to be loaded. That does not have any performance impact (but a security impact!). Once imdiag is loaded, it could be used to do all sorts of "interesting" things (most of which need to be implemented first...). Rainer > > david Lang > > > Any feedback is appreciated. While I like to have better ability to > dig into > > problems as they occur, I don't think it makes sense to invest time > into > > coding something if it is unclear whether or not it could be > utilized... So > > unless I get sufficient positive feedback, I'll stay away from > implementing > > that. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Nov 4 14:55:20 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 05:55:20 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103334@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103334@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 4 Nov 2009, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >>> >>> I could turn on/off this via a signal (as we currently do for sending >> debug >>> message to stdout). >> >> that would be a LOT easier than having to run in debug mode the entire >> time. > > The problem is that you would not get the information BEFORE you turned in > on. In theory, it should even be possible with the current system and a file, > but I need to check this out (I never used it that way, but the engine > probably is capable of doing it). yes, this would not help see what triggered the problem, but could be a huge help in figuring out what the current state of things are. going to a file could work as well. in either case a config option would have to be created to specify the destination. >> would this have similar performance impact to running in debug mode? or >> does this need less serialization, and therefor less impact? > > Think of it as turning on debug mode on the fly. That exactly is what it > does. So before the signal, there is no impact, but after it is the same > impact as usual. even in a high volume production environment this can be tolorated for short time periods. currently there is no way (short of stop, reconfigure, start, stop, reconfigure, start) to try and gather this info, and each stop will loose logs, so this ability would be much better. >>> Alternatively, one could load imdiag into a suspect instance and use >> that to >>> gain more in-depth information. However, I am not sure if that would >> be >>> suitable for use in production, at least it is questionable from a >> security >>> POV. >> >> is loading this on the fly even possible? > > NO, that's the point. If we intend to pursue that path, imdiag always needs > to be loaded. That does not have any performance impact (but a security > impact!). Once imdiag is loaded, it could be used to do all sorts of > "interesting" things (most of which need to be implemented first...). it all depends on what imdiag can do, and where the control signals come from. if the contol is via normal syslog messages it is a problem, on the other hand if it creates a new socket that it listens on that can only be written to by root it's probably far less of a problem. I guess the real response is that I don't know enough about imdiag to know the risks. David Lang From correo at miguelangelnieto.net Wed Nov 4 15:00:35 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Wed, 4 Nov 2009 15:00:35 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # kernel logging (formerly provided by rklogd) $WorkDirectory /var/log/queue $MainMsgQueueFileName mainq $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "TVC" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "TVB" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "TTD" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "KCD" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "LPT" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "ABT" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "XET" @@10.10.0.100 & ~ *.* /var/log/syslog kern.* /dev/console *.info;mail.none;authpriv.none;cron.none -/var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* -/var/log/cron uucp,news.crit -/var/log/spooler local7.* /var/log/boot.log 2009/11/4 : > On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > >> I have a problem with the attached client-configuration. When I stop >> the server, the client computer hangs-up some minutes later and didn't >> write logs on $WorkDirectory /var/log/queue. > > this list strips attachments, please re-send with the config in the body of > the message. > > david Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From rgerhards at hq.adiscon.com Wed Nov 4 15:11:58 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 15:11:58 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103334@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> > > The problem is that you would not get the information BEFORE you > turned in > > on. In theory, it should even be possible with the current system and > a file, > > but I need to check this out (I never used it that way, but the > engine > > probably is capable of doing it). > > yes, this would not help see what triggered the problem, but could be a > huge help in figuring out what the current state of things are. definitely > going to a file could work as well. in either case a config option > would > have to be created to specify the destination. traditionally, this is done via environment options, please see: http://www.rsyslog.com/doc-debug.html It may very well work just be specifying a file and sending SIGUSR1 - but I have never tried that in a production run (which also means I *never* tested it). When I added this mechanism, I selected envrionment variables because that enables to change the settings without touching the config file. Obviously it wouldn't be much of a problem to create config options for them as well. > > >> would this have similar performance impact to running in debug mode? > or > >> does this need less serialization, and therefor less impact? > > > > Think of it as turning on debug mode on the fly. That exactly is what > it > > does. So before the signal, there is no impact, but after it is the > same > > impact as usual. > > even in a high volume production environment this can be tolorated for > short time periods. currently there is no way (short of stop, > reconfigure, > start, stop, reconfigure, start) to try and gather this info, and each > stop will loose logs, so this ability would be much better. > > >>> Alternatively, one could load imdiag into a suspect instance and > use > >> that to > >>> gain more in-depth information. However, I am not sure if that > would > >> be > >>> suitable for use in production, at least it is questionable from a > >> security > >>> POV. > >> > >> is loading this on the fly even possible? > > > > NO, that's the point. If we intend to pursue that path, imdiag always > needs > > to be loaded. That does not have any performance impact (but a > security > > impact!). Once imdiag is loaded, it could be used to do all sorts of > > "interesting" things (most of which need to be implemented first...). > > it all depends on what imdiag can do, and where the control signals > come > from. if the contol is via normal syslog messages it is a problem, on > the > other hand if it creates a new socket that it listens on that can only > be > written to by root it's probably far less of a problem. > > I guess the real response is that I don't know enough about imdiag to > know > the risks. Most of that needs to be written, so we can specify what is acceptable. So far it listens to a network socket. This was intended so that testcases can be build without the need to fiddle with local socket creation policies. One potential was would also to selectively enable/disable imdiag functionality inside the config file. The GUI front-end I (barely) started to write talks to imdiag, but this is not really anything decent so far. In the long term, I'd like to be able to pull things like queue status, modules loaded, action status and the like from a running instance as well as inject not only messages but change some settings on the fly. My initial need was to provide a consistent rsyslog/UDP monitoring solution (for performance testing) as well as problem analysis. Rainer > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Nov 4 15:13:37 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 15:13:37 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103336@GRFEXC.intern.adiscon.com> just a quick look: the file name is always "dbq" and that will quickly result in an abort when actions go to the queue. Make sure the file names are different. Could be the source of your problem, but not sure... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Miguel Angel Nieto > Sent: Wednesday, November 04, 2009 3:01 PM > To: rsyslog-users > Subject: Re: [rsyslog] computer hang-up and WorkDirectory > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > $WorkDirectory /var/log/queue > $MainMsgQueueFileName mainq > > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TVC" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TVB" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TTD" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "KCD" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "LPT" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "ABT" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "XET" @@10.10.0.100 > & ~ > > *.* /var/log/syslog > kern.* /dev/console > *.info;mail.none;authpriv.none;cron.none - > /var/log/messages > authpriv.* /var/log/secure > mail.* - > /var/log/maillog > cron.* -/var/log/cron > uucp,news.crit - > /var/log/spooler > local7.* > /var/log/boot.log > > > 2009/11/4 : > > On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > > > >> I have a problem with the attached client-configuration. When I stop > >> the server, the client computer hangs-up some minutes later and > didn't > >> write logs on $WorkDirectory /var/log/queue. > > > > this list strips attachments, please re-send with the config in the > body of > > the message. > > > > david Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > > > > -- > Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que > hablar. Si quer?an decirme algo, tendr?an que escribirlo en un > papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que > hablar el resto de mi vida. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Nov 4 15:27:03 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 06:27:03 -0800 (PST) Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: ok, looking at this I don't see that you have any commands that would use the work directory. now when you say the client computer locks up do you mean the following? you have a server writing logs you have a seperate client sending logs to the server you shut down the server later the client machine stops responding. is this config for the client or for the server? one possible explination for the freeze you are seeing is that if you have the client configured to send via TCP (the @@ option) and the server does not accept the message, the client will queue the message, when the client queue fills up it will not accept any more messages. many processes (including login) will block until syslog accepts the message causeing the machine to 'freeze' or 'lock up' does this match what you are seeing? if so, turning the server back on should un-freeze the client machines. if this is the case you need to decide your priorities how critical is it to get the logs off the machine? in some cases they are a real security issue and you must get them off (in which case you really should be using relp, not tcp, but that's a different discussion that rainer did a write-up on), and your only real answer is to setup multiple servers so that one is always up. in other cases you are willing to spill over to disk and risk having an intruder tamper with the logs before they get sent off to another machine and set the main queu type to disk assisted mode in other cases you are willing to loose logs rather than freezing the machine and can configure rsyslog to accept messages, even when it can't do anything with them to avoid this sort of lockup. Daivd Lang On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > $WorkDirectory /var/log/queue > $MainMsgQueueFileName mainq > > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TVC" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TVB" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TTD" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "KCD" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "LPT" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "ABT" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "XET" @@10.10.0.100 > & ~ > > *.* /var/log/syslog > kern.* /dev/console > *.info;mail.none;authpriv.none;cron.none -/var/log/messages > authpriv.* /var/log/secure > mail.* -/var/log/maillog > cron.* -/var/log/cron > uucp,news.crit -/var/log/spooler > local7.* /var/log/boot.log > > > 2009/11/4 : >> On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: >> >>> I have a problem with the attached client-configuration. When I stop >>> the server, the client computer hangs-up some minutes later and didn't >>> write logs on $WorkDirectory /var/log/queue. >> >> this list strips attachments, please re-send with the config in the body of >> the message. >> >> david Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > > > > From david at lang.hm Wed Nov 4 15:38:54 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 06:38:54 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 4 Nov 2009, Rainer Gerhards wrote: >>> The problem is that you would not get the information BEFORE you >> turned in >>> on. In theory, it should even be possible with the current system and >> a file, >>> but I need to check this out (I never used it that way, but the >> engine >>> probably is capable of doing it). >> >> yes, this would not help see what triggered the problem, but could be a >> huge help in figuring out what the current state of things are. > > definitely > >> going to a file could work as well. in either case a config option >> would >> have to be created to specify the destination. > > traditionally, this is done via environment options, please see: > > http://www.rsyslog.com/doc-debug.html > > It may very well work just be specifying a file and sending SIGUSR1 - but I > have never tried that in a production run (which also means I *never* tested > it). > > When I added this mechanism, I selected envrionment variables because that > enables to change the settings without touching the config file. Obviously it > wouldn't be much of a problem to create config options for them as well. given that in a production environment, rsyslog is started from an init script of some sort, setting environment variables before it starts is just editing a different config file. it would probably be much better to have it in the rsyslog config than to hunt down the correct (distro specific) init file and edit it to set an environemnt variable there. >>> NO, that's the point. If we intend to pursue that path, imdiag always >> needs >>> to be loaded. That does not have any performance impact (but a >> security >>> impact!). Once imdiag is loaded, it could be used to do all sorts of >>> "interesting" things (most of which need to be implemented first...). >> >> it all depends on what imdiag can do, and where the control signals >> come >> from. if the contol is via normal syslog messages it is a problem, on >> the >> other hand if it creates a new socket that it listens on that can only >> be >> written to by root it's probably far less of a problem. >> >> I guess the real response is that I don't know enough about imdiag to >> know >> the risks. > > Most of that needs to be written, so we can specify what is acceptable. So > far it listens to a network socket. This was intended so that testcases can > be build without the need to fiddle with local socket creation policies. One > potential was would also to selectively enable/disable imdiag functionality > inside the config file. > > The GUI front-end I (barely) started to write talks to imdiag, but this is > not really anything decent so far. In the long term, I'd like to be able to > pull things like queue status, modules loaded, action status and the like > from a running instance as well as inject not only messages but change some > settings on the fly. My initial need was to provide a consistent rsyslog/UDP > monitoring solution (for performance testing) as well as problem analysis. in a production environement you may not be able to use a GUI, and a network socket (on anything other than localhost) is definantly a problem. if you have it create a network listener on localhost that can be connected to via telnet it's not nearly as big of a security problem. then it can be scripted or a user can cut-n-paste to it while still allowing a nice GUI (or text based equivalent) to be created as time permits. but for now just use a simple line-based interface and it would be usable. if the listening address and port are configurable you can have the address default to 127.0.0.1 (relativly safe), but be configurable to 0.0.0.0 so that you can hit it over the network with other machines as needed. David Lang From rgerhards at hq.adiscon.com Wed Nov 4 15:57:37 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 15:57:37 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com> > given that in a production environment, rsyslog is started from an init > script of some sort, setting environment variables before it starts is > just editing a different config file. it would probably be much better > to > have it in the rsyslog config than to hunt down the correct (distro > specific) init file and edit it to set an environemnt variable there. I'll see if there is anything that prevents this, if not, I'll simply add the relevant statements. > > The GUI front-end I (barely) started to write talks to imdiag, but > this is > > not really anything decent so far. In the long term, I'd like to be > able to > > pull things like queue status, modules loaded, action status and the > like > > from a running instance as well as inject not only messages but > change some > > settings on the fly. My initial need was to provide a consistent > rsyslog/UDP > > monitoring solution (for performance testing) as well as problem > analysis. > > in a production environement you may not be able to use a GUI, and a > network socket (on anything other than localhost) is definantly a > problem. > > if you have it create a network listener on localhost that can be > connected to via telnet it's not nearly as big of a security problem. > then > it can be scripted or a user can cut-n-paste to it while still allowing > a > nice GUI (or text based equivalent) to be created as time permits. but > for > now just use a simple line-based interface and it would be usable. > > if the listening address and port are configurable you can have the > address default to 127.0.0.1 (relativly safe), but be configurable to > 0.0.0.0 so that you can hit it over the network with other machines as > needed. That's along the lines after which the current imdiag is designed. However, I was thinking to exchange some data via XML streams as it can be very lengthy (think of a complete queue stats dump for all queues in a system). XML would probably not work very well with the frontend, something readable would probably not work very well with the GUI - maybe the request should just specify the output format ;) Rainer From david at lang.hm Wed Nov 4 16:01:34 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 07:01:34 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 4 Nov 2009, Rainer Gerhards wrote: >>> The GUI front-end I (barely) started to write talks to imdiag, but >> this is >>> not really anything decent so far. In the long term, I'd like to be >> able to >>> pull things like queue status, modules loaded, action status and the >> like >>> from a running instance as well as inject not only messages but >> change some >>> settings on the fly. My initial need was to provide a consistent >> rsyslog/UDP >>> monitoring solution (for performance testing) as well as problem >> analysis. >> >> in a production environement you may not be able to use a GUI, and a >> network socket (on anything other than localhost) is definantly a >> problem. >> >> if you have it create a network listener on localhost that can be >> connected to via telnet it's not nearly as big of a security problem. >> then >> it can be scripted or a user can cut-n-paste to it while still allowing >> a >> nice GUI (or text based equivalent) to be created as time permits. but >> for >> now just use a simple line-based interface and it would be usable. >> >> if the listening address and port are configurable you can have the >> address default to 127.0.0.1 (relativly safe), but be configurable to >> 0.0.0.0 so that you can hit it over the network with other machines as >> needed. > > That's along the lines after which the current imdiag is designed. However, I > was thinking to exchange some data via XML streams as it can be very lengthy > (think of a complete queue stats dump for all queues in a system). XML would > probably not work very well with the frontend, something readable would > probably not work very well with the GUI - maybe the request should just > specify the output format ;) that can work, however XML output may not be too bad, people entering commands manually are probably not going to be asking for that much data. the biggest problem I see with XML is the need to send requests and responses via e-mail for debugging. if one end uses a HTML enabled e-mail client it may not work well to paste XML text into it. David Lang From rgerhards at hq.adiscon.com Wed Nov 4 16:06:36 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 16:06:36 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103338@GRFEXC.intern.adiscon.com> > that can work, however XML output may not be too bad, people entering > commands manually are probably not going to be asking for that much > data. If I look at time constraints, I will probably not able to define a big CLI with a real language. So I think you might send something like "queuestatus" and it will reply with all status information on all queues it has. > the biggest problem I see with XML is the need to send requests and > responses via e-mail for debugging. if one end uses a HTML enabled e- > mail > client it may not work well to paste XML text into it. That's a very good point and I really should begin to think about this right now. Some kind of "debug package" that's easily mailable would not be a bad thing ;) Rainer From correo at miguelangelnieto.net Wed Nov 4 16:14:03 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Wed, 4 Nov 2009 16:14:03 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: Hi, 2009/11/4 : > ok, looking at this I don't see that you have any commands that would use > the work directory. Ok, here I have the first bug. I supposed that rsyslog will use work directory to save queues of all TCP forwards. What I missed to get this functionality? > now when you say the client computer locks up do you mean the following? > > you have a server writing logs > you have a seperate client sending logs to the server > you shut down the server > later the client machine stops responding. The client stops responding. > is this config for the client or for the server? Client. > one possible explination for the freeze you are seeing is that if you have > the client configured to send via TCP (the @@ option) and the server does > not accept the message, the client will queue the message, when the client > queue fills up it will not accept any more messages. many processes > (including login) will block until syslog accepts the message causeing the > machine to 'freeze' or 'lock up' > > does this match what you are seeing? Yes. So, activating the disk queue should fix this problem. Is it true? I would set a limit of 5GB. > if this is the case you need to decide your priorities > > how critical is it to get the logs off the machine? We have a lot of computers logging to a centralized rsyslog server. In each computer runs a application logging diferent activities (webcams, doors, etc.) We need all the data in one computer to analize it. > in other cases you are willing to loose logs rather than freezing the > machine and can configure rsyslog to accept messages, even when it can't > do anything with them to avoid this sort of lockup. How Can I do that? Thank you! > Daivd Lang > > > On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > >> $ModLoad immark.so # provides --MARK-- message capability >> $ModLoad imuxsock.so # provides support for local system logging (e.g. >> via logger command) >> $ModLoad imklog.so # kernel logging (formerly provided by rklogd) >> >> $WorkDirectory /var/log/queue >> $MainMsgQueueFileName mainq >> >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TVC" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TVB" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TTD" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "KCD" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "LPT" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "ABT" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "XET" @@10.10.0.100 >> & ~ >> >> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? /var/log/syslog >> kern.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? /dev/console >> *.info;mail.none;authpriv.none;cron.none ? ? ? ? ? ? ? ?-/var/log/messages >> authpriv.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/secure >> mail.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/maillog >> cron.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/cron >> uucp,news.crit ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/spooler >> local7.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/boot.log >> >> >> 2009/11/4 ?: >>> On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: >>> >>>> I have a problem with the attached client-configuration. When I stop >>>> the server, the client computer hangs-up some minutes later and didn't >>>> write logs on $WorkDirectory /var/log/queue. >>> >>> this list strips attachments, please re-send with the config in the body of >>> the message. >>> >>> david Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> >> >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From mbiebl at gmail.com Wed Nov 4 19:30:20 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 4 Nov 2009 19:30:20 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 Message-ID: Hi Rainer, hi list I tried the latest v5 stable release. First thing I noticed is, that it still uses -c 4 as native mode. I also noticed problems with logging to a pipe. The default rsyslog.conf file on Debian has a daemon.*;mail.*;\ news.err;\ *.=debug;*.=info;\ *.=notice;*.=warn |/dev/xconsole But I get the following error: 9042.679173747:b7607b70: file to log to: |/dev/xconsole 9042.679183525:b7607b70: doWrite, pData->pStrm 0x814d8c0, lenBuf 131 9042.679194699:b7607b70: strm 0x814d8c0: file -1 flush, buflen 208 9042.679225429:b7607b70: strm 0x814d8c0: open error 2, file '|/dev/xconsole' I've put a complete debug log + config files at http://debs.michaelbiebl.de/rsyslog/v5 Rainer, maybe you can take a look. Thanks, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Wed Nov 4 21:17:09 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 21:17:09 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Michael, That is very probably the same regression that Luis Fernando mentioned for 4.5.5. All in all, I would be quite careful with 5.2.0, I really have the impression that folks simply didn't try it out. But as I wrote, I need to somehow get the process started to come to make v5 mainstream. It is really important to me to get rid of v4-devel, as it takes considerable time (and bug potential!) to have two branches that are in development state. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Wednesday, November 04, 2009 7:30 PM > To: rsyslog-users > Subject: [rsyslog] rsyslog v5 5.2.0 > > Hi Rainer, hi list > > I tried the latest v5 stable release. > First thing I noticed is, that it still uses -c 4 as native mode. > I also noticed problems with logging to a pipe. The default > rsyslog.conf file on Debian has a > daemon.*;mail.*;\ > news.err;\ > *.=debug;*.=info;\ > *.=notice;*.=warn |/dev/xconsole > > But I get the following error: > > 9042.679173747:b7607b70: file to log to: |/dev/xconsole > 9042.679183525:b7607b70: doWrite, pData->pStrm 0x814d8c0, lenBuf 131 > 9042.679194699:b7607b70: strm 0x814d8c0: file -1 flush, buflen 208 > 9042.679225429:b7607b70: strm 0x814d8c0: open error 2, file > '|/dev/xconsole' > > I've put a complete debug log + config files at > http://debs.michaelbiebl.de/rsyslog/v5 > > Rainer, maybe you can take a look. > > Thanks, > Michael > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From correo at miguelangelnieto.net Wed Nov 4 22:32:43 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Wed, 4 Nov 2009 22:32:43 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: Hi again, I simplified the rsyslog config file: $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # kernel logging (formerly provided by rklogd) $WorkDirectory /var/log/queue $ActionQueueType LinkedList $ActionQueueFileName prueba $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on *.* @@10.10.0.210 & ~ *.* /var/log/messages kern.* /dev/console *.info;mail.none;authpriv.none;cron.none -/var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* -/var/log/cron uucp,news.crit -/var/log/spooler local7.* /var/log/boot.log Now I have disk queue, but rsyslog only logs up to 3000 entries. 0719.224892620:imuxsock.c: --------imuxsock calling select, active file descriptors (max 3): 3 0719.233534436:imuxsock.c: Message from UNIX socket: 0000003 0719.233573550:imuxsock.c: logmsg: flags 4, from 'linux-92wq', msg Nov 4 22:38:39 punisher: pruebaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa 0719.233589475:imuxsock.c: Message has legacy syslog format. 0719.233643675:imuxsock.c: main queue: entry added, size now 3000 entries 0719.233659599:imuxsock.c: main queue: EnqueueMsg signaled condition (0) 0719.233674127:imuxsock.c: wtpAdviseMaxWorkers signals busy 0719.233688096:imuxsock.c: --------imuxsock calling select, active file descriptors (max 3): 3 0719.244104262:imuxsock.c: Message from UNIX socket: 0000003 0719.244145331:imuxsock.c: logmsg: flags 4, from 'linux-92wq', msg Nov 4 22:38:39 punisher: pruebaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa 0719.244161256:imuxsock.c: Message has legacy syslog format. 0719.244186680:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0720.818283853:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0721.819871573:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0722.828967380:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0723.832482270:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0724.839541947:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0725.848415367:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0726.855937980:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0727.858592935:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0728.862998771:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. Rsyslog 3.19.7 (Suse) 2009/11/4 : > ok, looking at this I don't see that you have any commands that would use > the work directory. > > now when you say the client computer locks up do you mean the following? > > you have a server writing logs > you have a seperate client sending logs to the server > you shut down the server > later the client machine stops responding. > > is this config for the client or for the server? > > one possible explination for the freeze you are seeing is that if you have > the client configured to send via TCP (the @@ option) and the server does > not accept the message, the client will queue the message, when the client > queue fills up it will not accept any more messages. many processes > (including login) will block until syslog accepts the message causeing the > machine to 'freeze' or 'lock up' > > does this match what you are seeing? > > if so, turning the server back on should un-freeze the client machines. > > > if this is the case you need to decide your priorities > > how critical is it to get the logs off the machine? in some cases they are > a real security issue and you must get them off (in which case you really > should be using relp, not tcp, but that's a different discussion that > rainer did a write-up on), and your only real answer is to setup multiple > servers so that one is always up. > > in other cases you are willing to spill over to disk and risk having an > intruder tamper with the logs before they get sent off to another machine > and set the main queu type to disk assisted mode > > in other cases you are willing to loose logs rather than freezing the > machine and can configure rsyslog to accept messages, even when it can't > do anything with them to avoid this sort of lockup. > > Daivd Lang > > > On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > >> $ModLoad immark.so # provides --MARK-- message capability >> $ModLoad imuxsock.so # provides support for local system logging (e.g. >> via logger command) >> $ModLoad imklog.so # kernel logging (formerly provided by rklogd) >> >> $WorkDirectory /var/log/queue >> $MainMsgQueueFileName mainq >> >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TVC" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TVB" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TTD" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "KCD" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "LPT" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "ABT" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "XET" @@10.10.0.100 >> & ~ >> >> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? /var/log/syslog >> kern.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? /dev/console >> *.info;mail.none;authpriv.none;cron.none ? ? ? ? ? ? ? ?-/var/log/messages >> authpriv.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/secure >> mail.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/maillog >> cron.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/cron >> uucp,news.crit ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/spooler >> local7.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/boot.log >> >> >> 2009/11/4 ?: >>> On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: >>> >>>> I have a problem with the attached client-configuration. When I stop >>>> the server, the client computer hangs-up some minutes later and didn't >>>> write logs on $WorkDirectory /var/log/queue. >>> >>> this list strips attachments, please re-send with the config in the body of >>> the message. >>> >>> david Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> >> >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From tbergfeld at hq.adiscon.com Thu Nov 5 08:08:00 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Thu, 5 Nov 2009 08:08:00 +0100 Subject: [rsyslog] rsyslog 5.3.4 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103340@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.3.4, a member of the v5-devel branch. Today's release offers a number of important improvements. I would like to highlight the ability to nest rulesets [1] (I already announced that) and the ability to define custom parsers [2]. Let me elaborate a bit on the later. In the syslog world exists a myriad of incompatible and invalidly formatted messages. We have had numerous support requests on how to parse this or that malformed message format. With custom parsers, we now have a vehicel to integrate all these invalid formats in an elegant way that is also offering high performance ([2] has a concrete sample). The core idea now is that parsers are plugins just like input and output modules. It is relatively easy to write such a parser (roughly a day, I expect) and custom parsers can be integrated into rsyslogd in a way that they only affect messages received on specific port. So far, I have not actually created a custom parser, but I hope that over time many of them will become available. My hope here is that we actually can build a directory, which other folks can browse so that they are able to find a solution to the mess their vendor has provided ;) This release also introduces the capability to run rulesets off their own queue (also already mentioned on the mailing list). Plus, it contains the usual set of bug fixes. We plan to make this version the basis for the next v5-stable. [1] http://www.rsyslog.com/doc-omruleset.html [2] http://www.rsyslog.com/doc-rsconf1_rulesetparser.html ChangeLog: http://www.rsyslog.com/Article423.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-185.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From david at lang.hm Thu Nov 5 08:53:12 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 23:53:12 -0800 (PST) Subject: [rsyslog] rsyslog 5.3.4 (devel) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103340@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103340@GRFEXC.intern.adiscon.com> Message-ID: yOn Thu, 5 Nov 2009, Tom Bergfeld wrote: > We have just released rsyslog 5.3.4, a member of the v5-devel branch. > Today's release offers a number of important improvements. I would like to > highlight the ability to nest rulesets [1] (I already announced that) and the > ability to define custom parsers [2]. > > Let me elaborate a bit on the later. In the syslog world exists a myriad of > incompatible and invalidly formatted messages. We have had numerous support > requests on how to parse this or that malformed message format. With custom > parsers, we now have a vehicel to integrate all these invalid formats in an > elegant way that is also offering high performance ([2] has a concrete > sample). The core idea now is that parsers are plugins just like input and > output modules. It is relatively easy to write such a parser (roughly a day, > I expect) and custom parsers can be integrated into rsyslogd in a way that > they only affect messages received on specific port. So far, I have not > actually created a custom parser, but I hope that over time many of them will > become available. My hope here is that we actually can build a directory, > which other folks can browse so that they are able to find a solution to the > mess their vendor has provided ;) this sounds very interesting. however, reading the link I'm a little confused. on the one hand it talks a lot about the priority of parsers, but then it talks about binding different parsers to different ports. It's not clear if these are two different ways to use parsers, or how these two would interact. where can these parsers be used? obviously the imudp module can use them. can all input modules use them? what are the limits of the parser? a couple of extreme examples: could you implement relp as a parser for imtcp, or since relp sends replies that can't be done (limiting it to different ways of processing a message that's already been received) could a parser detect that it had a piece of a multi-line messsage and stash the piece until it receives the rest of the message so that it could submit the complete message? a coouple questions from trying to look through the code at almost 3am local time if it can easily be done, may I suggest changing the santization flag from a yes/no boolean to an enum so that there can be more than one sanitation routine do you have the default parser broken out as a seperate file tha canbe used as an example? David Lang > This release also introduces the capability to run rulesets off their own > queue (also already mentioned on the mailing list). Plus, it contains the > usual set of bug fixes. > > We plan to make this version the basis for the next v5-stable. > > > [1] http://www.rsyslog.com/doc-omruleset.html > [2] http://www.rsyslog.com/doc-rsconf1_rulesetparser.html > > > > ChangeLog: > > http://www.rsyslog.com/Article423.phtml > > Download: > > http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-185.phtml > > As always, feedback is appreciated. > > Best regards, > Tom Bergfeld > -- > Support > ======= > > Improving rsyslog is costly, but you can help! We are looking for > organizations that find rsyslog useful and wish to contribute back. You can > contribute by reporting bugs, improve the software, or donate money or > equipment. > > Commercial support contracts for rsyslog are available, and they help finance > continued maintenance. Adiscon GmbH, a privately held German company, is > currently funding rsyslog development. We are always looking for interesting > development projects. For details on how to help, please see > http://www.rsyslog.com/doc-how2help.html . > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Nov 5 09:29:26 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 09:29:26 +0100 Subject: [rsyslog] rsyslog 5.3.4 (devel) released References: <9B6E2A8877C38245BFB15CC491A11DA7103340@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103342@GRFEXC.intern.adiscon.com> Good questions, thanks! :) I'll add some of the answers to the doc, what probably makes most sense. Just one thing: These are message parsers, not frame parsers. So you cannot parse RELP out of plain tcp, simply because that would be a frame parser (and even more, as these are totally different protocols). Message parsers work on the raw message, once it is extracted from the transport framing. I don't see any use case for permitting different frame parsers, as the framing is always bound to a specific protocol and a protocol has many more attributes than just the framing. The right way to implement a new protocol is via an input/output plugins. Currently, all message parsers are built into the core, but this is just a delivery mechanism (like with omfile, ...). The code is the same for loadable parsers. The two parsers I currently provide can be seen here: http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/pmrfc3164.c;h=5b684af56e 6d5e8ea2bcd616a06cc39e4cbb4a09;hb=6d9c54c7a2d4f07b0414082ef9681bd197ed6bde http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/pmrfc5424.c;h=07994adeba fb7e40d597e417dd1215874972971c;hb=6d9c54c7a2d4f07b0414082ef9681bd197ed6bde Rest comes either via doc or with a later reply... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, November 05, 2009 8:53 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 5.3.4 (devel) released > > yOn Thu, 5 Nov 2009, Tom Bergfeld wrote: > > > We have just released rsyslog 5.3.4, a member of the v5-devel branch. > > Today's release offers a number of important improvements. I would > like to > > highlight the ability to nest rulesets [1] (I already announced that) > and the > > ability to define custom parsers [2]. > > > > Let me elaborate a bit on the later. In the syslog world exists a > myriad of > > incompatible and invalidly formatted messages. We have had numerous > support > > requests on how to parse this or that malformed message format. With > custom > > parsers, we now have a vehicel to integrate all these invalid formats > in an > > elegant way that is also offering high performance ([2] has a > concrete > > sample). The core idea now is that parsers are plugins just like > input and > > output modules. It is relatively easy to write such a parser (roughly > a day, > > I expect) and custom parsers can be integrated into rsyslogd in a way > that > > they only affect messages received on specific port. So far, I have > not > > actually created a custom parser, but I hope that over time many of > them will > > become available. My hope here is that we actually can build a > directory, > > which other folks can browse so that they are able to find a solution > to the > > mess their vendor has provided ;) > > this sounds very interesting. however, reading the link I'm a little > confused. > > on the one hand it talks a lot about the priority of parsers, but then > it > talks about binding different parsers to different ports. It's not > clear > if these are two different ways to use parsers, or how these two would > interact. > > where can these parsers be used? > > obviously the imudp module can use them. can all input modules use > them? > > what are the limits of the parser? > > a couple of extreme examples: > > could you implement relp as a parser for imtcp, or since relp sends > replies that can't be done (limiting it to different ways of processing > a > message that's already been received) > > could a parser detect that it had a piece of a multi-line messsage and > stash the piece until it receives the rest of the message so that it > could > submit the complete message? > > > a coouple questions from trying to look through the code at almost 3am > local time > > if it can easily be done, may I suggest changing the santization flag > from > a yes/no boolean to an enum so that there can be more than one > sanitation > routine > > do you have the default parser broken out as a seperate file tha canbe > used as an example? > > David Lang > > > This release also introduces the capability to run rulesets off their > own > > queue (also already mentioned on the mailing list). Plus, it contains > the > > usual set of bug fixes. > > > > We plan to make this version the basis for the next v5-stable. > > > > > > [1] http://www.rsyslog.com/doc-omruleset.html > > [2] http://www.rsyslog.com/doc-rsconf1_rulesetparser.html > > > > > > > > ChangeLog: > > > > http://www.rsyslog.com/Article423.phtml > > > > Download: > > > > http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid- > 185.phtml > > > > As always, feedback is appreciated. > > > > Best regards, > > Tom Bergfeld > > -- > > Support > > ======= > > > > Improving rsyslog is costly, but you can help! We are looking for > > organizations that find rsyslog useful and wish to contribute back. > You can > > contribute by reporting bugs, improve the software, or donate money > or > > equipment. > > > > Commercial support contracts for rsyslog are available, and they help > finance > > continued maintenance. Adiscon GmbH, a privately held German > company, is > > currently funding rsyslog development. We are always looking for > interesting > > development projects. For details on how to help, please see > > http://www.rsyslog.com/doc-how2help.html . > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Nov 5 09:36:14 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 09:36:14 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103343@GRFEXC.intern.adiscon.com> > 0726.855937980:imuxsock.c: main queue: enqueueMsg: LightDelay mark > reached for light delayble message - blocking a bit. > 0727.858592935:imuxsock.c: main queue: enqueueMsg: LightDelay mark > reached for light delayble message - blocking a bit. > 0728.862998771:imuxsock.c: main queue: enqueueMsg: LightDelay mark > reached for light delayble message - blocking a bit. > > Rsyslog 3.19.7 (Suse) The version is the issue - that's an outdated beta version ;) Some versions had unix sockets flagged as sources that could lightly be delayed, resulting in what you see. That was made optional in later builds (and I guess nobody has turned it on since then ;)). Cure: upgrade to the current (3.22.1) v3-stable. Rainer From rgerhards at hq.adiscon.com Thu Nov 5 12:47:39 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 12:47:39 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5/5.2.0? References: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> <200911041414.34449.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103353@GRFEXC.intern.adiscon.com> Looks like a trivial regression, fix here: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=a5cd509be736fcff3c8ae310 4712d2fe85776416 I'll now do some more checks and see if I can include an automatted test for the pipe functionality so that a similar problem won't slip through in the future :) I will later apply the same fix to 5.2.0, it is the same problem. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luis Fernando Mu?oz Mej?as > Sent: Wednesday, November 04, 2009 2:15 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Output to named pipes not working on v4.5.5? > > So, this has two parts: > > 1) A SELinux problem on RHEL5 and similar (but the logs show > nothing!!). It seems the syslog context is not allowed to access > FIFOs. I'll have to fix that. > > 2) Even when there are no permission problems, rsyslog 4.5.5 just > doesn't write to a named pipe. From the following consecutive lines: > > *.* |/var/log/external/fifo > *.* /var/log/external/file > > the file receives data, the FIFO doesn't. > > I'll paste all this to bugzilla. > > Cheers. > -- > Luis Fernando Mu?oz Mej?as > Luis.Fernando.Munoz.Mejias at cern.ch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From martinmie at PartyGaming.com Thu Nov 5 13:51:38 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 5 Nov 2009 13:51:38 +0100 Subject: [rsyslog] rsyslog 5.2.0 meets signal 11. Message-ID: <0B1B9163138571439EAF804646F3F6AA08A586E8@GIBSVWIN004X.partygaming.local> Hi, Yesterday I successfully upgraded our logservers to the stable rsyslog 5.2.0. This morning I found out that the rsyslog process on the primary server was dead. Digging a bit in the logs I found this: -- # grep stop logserver.20091104.log 2009-11-04T07:32:01.970288-05:00 logserver kernel: Kernel logging (proc) stopped. -- According to the audit.log, it received a SIGSEGV (kill -11): -- type=ANOM_ABEND msg=audit(1257340663.009:19009): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=25881 comm="rsyslogd" sig=11 -- ...so, the application died on nov. 4th 2009 @ 8:17:43 EST -- # date -d @1257340663.009 Wed Nov 4 08:17:43 EST 2009 -- Has this been observed for this v5 stable branch?? And is there any fix? And, auditd was needed in order to determine the reason why rsyslogd was stopped. Could it be possible to have a more verbose message when the process stops due to an anomaly? Cheers, Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From Luis.Fernando.Munoz.Mejias at cern.ch Thu Nov 5 15:49:40 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?utf-8?q?Mu=C3=B1oz_Mej=C3=ADas?=) Date: Thu, 5 Nov 2009 15:49:40 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5/5.2.0? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103353@GRFEXC.intern.adiscon.com> References: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> <200911041414.34449.Luis.Fernando.Munoz.Mejias@cern.ch> <9B6E2A8877C38245BFB15CC491A11DA7103353@GRFEXC.intern.adiscon.com> Message-ID: <200911051549.40352.Luis.Fernando.Munoz.Mejias@cern.ch> Rainer, > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=a5cd509be736fcff3c8ae3 >10 4712d2fe85776416 I've cherry-picked this commit and I confirm it's fixed on v4.5.X. Thanks a lot for the quick action. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From rgerhards at hq.adiscon.com Thu Nov 5 15:51:15 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 15:51:15 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5/5.2.0? References: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch><200911041414.34449.Luis.Fernando.Munoz.Mejias@cern.ch><9B6E2A8877C38245BFB15CC491A11DA7103353@GRFEXC.intern.adiscon.com> <200911051549.40352.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103357@GRFEXC.intern.adiscon.com> Thanks for the feedback, much appreciated. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luis Fernando Mu?oz Mej?as > Sent: Thursday, November 05, 2009 3:50 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Output to named pipes not working on > v4.5.5/5.2.0? > > Rainer, > > > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=a5cd509be736fcff3c > 8ae3 > >10 4712d2fe85776416 > > I've cherry-picked this commit and I confirm it's fixed on > v4.5.X. Thanks a lot for the quick action. > -- > Luis Fernando Mu?oz Mej?as > Luis.Fernando.Munoz.Mejias at cern.ch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Thu Nov 5 16:50:16 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 5 Nov 2009 16:50:16 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: 2009/11/4 Rainer Gerhards : > Michael, > > That is very probably the same regression that Luis Fernando mentioned for > 4.5.5. All in all, I would be quite careful with 5.2.0, I really have the > impression that folks simply didn't try it out. But as I wrote, I need to > somehow get the process started to come to make v5 mainstream. It is really > important to me to get rid of v4-devel, as it takes considerable time (and > bug potential!) to have two branches that are in development state. I can confirm that a5cd509be736fcff3c8ae3104712d2fe85776416 fixes this issue for me using 5.2.0. What about the -c compat flag? Is it intentional that there is no -c 5? I will probably upload a 5.2.* to unstable (or experimental) soon, which should give it some wider exposure. Cheers, Michael > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Michael Biebl >> Sent: Wednesday, November 04, 2009 7:30 PM >> To: rsyslog-users >> Subject: [rsyslog] rsyslog v5 5.2.0 >> >> Hi Rainer, hi list >> >> I tried the latest v5 stable release. >> First thing I noticed is, that it still uses -c 4 as native mode. >> I also noticed problems with logging to a pipe. The default >> rsyslog.conf file on Debian has a >> daemon.*;mail.*;\ >> ? ? ? ? news.err;\ >> ? ? ? ? *.=debug;*.=info;\ >> ? ? ? ? *.=notice;*.=warn ? ? ? |/dev/xconsole >> >> But I get the following error: >> >> 9042.679173747:b7607b70: file to log to: |/dev/xconsole >> 9042.679183525:b7607b70: doWrite, pData->pStrm 0x814d8c0, lenBuf 131 >> 9042.679194699:b7607b70: strm 0x814d8c0: file -1 flush, buflen 208 >> 9042.679225429:b7607b70: strm 0x814d8c0: open error 2, file >> '|/dev/xconsole' >> >> I've put a complete debug log + config files at >> http://debs.michaelbiebl.de/rsyslog/v5 >> >> Rainer, maybe you can take a look. >> >> Thanks, >> Michael >> >> -- >> Why is it that all of the instruments seeking intelligent life in the >> universe are pointed away from Earth? >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Nov 5 16:53:34 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 16:53:34 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, November 05, 2009 4:50 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog v5 5.2.0 > > 2009/11/4 Rainer Gerhards : > > Michael, > > > > That is very probably the same regression that Luis Fernando > mentioned for > > 4.5.5. All in all, I would be quite careful with 5.2.0, I really have > the > > impression that folks simply didn't try it out. But as I wrote, I > need to > > somehow get the process started to come to make v5 mainstream. It is > really > > important to me to get rid of v4-devel, as it takes considerable time > (and > > bug potential!) to have two branches that are in development state. > > I can confirm that a5cd509be736fcff3c8ae3104712d2fe85776416 fixes this > issue for me using 5.2.0. > What about the -c compat flag? Is it intentional that there is no -c 5? > I will probably upload a 5.2.* to unstable (or experimental) soon, > which should give it some wider exposure. I'll look into that, soon. My hopes are to get past 5.2.0 as quickly as possible. I have done a lot of work on the queue engine in the recent development version and there are definitely a couple of issues in 5.2.0 which cannot be fixed without upgrading the 5.3 version. That just FYI. I will probably not work on any queue-related issues in 5.2 but rather recommend to move on to a newer version. However, for normal operations that do not require intensive use of the queues, 5.2.0 should work OK and be faster than pervious versions. Rainer > Cheers, > Michael > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Michael > Biebl > >> Sent: Wednesday, November 04, 2009 7:30 PM > >> To: rsyslog-users > >> Subject: [rsyslog] rsyslog v5 5.2.0 > >> > >> Hi Rainer, hi list > >> > >> I tried the latest v5 stable release. > >> First thing I noticed is, that it still uses -c 4 as native mode. > >> I also noticed problems with logging to a pipe. The default > >> rsyslog.conf file on Debian has a > >> daemon.*;mail.*;\ > >> ? ? ? ? news.err;\ > >> ? ? ? ? *.=debug;*.=info;\ > >> ? ? ? ? *.=notice;*.=warn ? ? ? |/dev/xconsole > >> > >> But I get the following error: > >> > >> 9042.679173747:b7607b70: file to log to: |/dev/xconsole > >> 9042.679183525:b7607b70: doWrite, pData->pStrm 0x814d8c0, lenBuf 131 > >> 9042.679194699:b7607b70: strm 0x814d8c0: file -1 flush, buflen 208 > >> 9042.679225429:b7607b70: strm 0x814d8c0: open error 2, file > >> '|/dev/xconsole' > >> > >> I've put a complete debug log + config files at > >> http://debs.michaelbiebl.de/rsyslog/v5 > >> > >> Rainer, maybe you can take a look. > >> > >> Thanks, > >> Michael > >> > >> -- > >> Why is it that all of the instruments seeking intelligent life in > the > >> universe are pointed away from Earth? > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Thu Nov 5 17:02:32 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 5 Nov 2009 17:02:32 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710335A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710335A@GRFEXC.intern.adiscon.com> Message-ID: Another issue I noticed with 5.2.0 (+said patch) is, that it no longer shuts down cleany on SIGTERM/SIGINT If I run it with -c 4 -d and press STRG+C, I get ... 6895.847489380:b7692940: Terminating outputs... 6895.847516758:b7692940: destructing ruleset 0x89c7648, name 0x89c7678 6895.847553634:b7692940: strm 0x89c97b8: file -1 flush, buflen 0 6895.847595539:b7692940: strm 0x89ca1d8: file 5 flush, buflen 0 6895.847627386:b7692940: strm 0x89ca1d8: file 5 closing 6895.847688567:b7692940: strm 0x89cac48: file -1 flush, buflen 0 6895.847728796:b7692940: strm 0x89cb680: file 6 flush, buflen 0 6895.847758967:b7692940: strm 0x89cb680: file 6 closing 6895.847805062:b7692940: strm 0x89cc108: file -1 flush, buflen 0 6895.847846129:b7692940: strm 0x89ccb78: file -1 flush, buflen 0 6895.847886916:b7692940: strm 0x89cd5c0: file -1 flush, buflen 0 6895.847927145:b7692940: strm 0x89ce060: file -1 flush, buflen 0 6895.847967653:b7692940: strm 0x89cea98: file -1 flush, buflen 0 6895.848007882:b7692940: strm 0x89cf500: file -1 flush, buflen 0 6895.848048110:b7692940: strm 0x89cff68: file -1 flush, buflen 0 6895.848088897:b7692940: strm 0x89d09d8: file -1 flush, buflen 0 6895.848129126:b7692940: strm 0x89d1478: file -1 flush, buflen 0 6895.848169355:b7692940: strm 0x89d1ee8: file -1 flush, buflen 0 6895.848220758:b7692940: strm 0x89d2990: file 7 flush, buflen 0 6895.848251488:b7692940: strm 0x89d2990: file 7 closing 6895.848300098:b7692940: strm 0x89d38c0: file -1 flush, buflen 77 pressing STRG+C a second time will then terminate rsyslogd. From mrdemeanour at jackpot.uk.net Thu Nov 5 17:14:05 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Thu, 05 Nov 2009 16:14:05 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls Message-ID: <4AF2F9CD.9000402@jackpot.uk.net> Hi, I'm running a central rsyslog server with a couple of remote WAN (internet) clients and several remote LAN clients. Traffic is low - of the order of 10,000 messages per day. Internet clients communicate with the server using gnutls. LAN clients are currently using UDP. The server writes client logs to mysql, and also writes messages of local origin to disk. After some interval x::x <= 24h, the server consumes all memory (and swap) in the system. It looks as if the OS then tries to evict rsyslogd, and hangs - there's what looks like a stacktrace displayed on the (dead) console. I've stopped using gnutls for now, because the server machine is also a NFS file-server, and other systems depend on it not shutting down like this. It seems the problem doesn't occur with plain tcp. # uname -a Linux prajna 2.6.30-2-686 #1 SMP Sat Sep 26 01:16:22 UTC 2009 i686 GNU/Linux # rsyslogd -v rsyslogd 4.4.2, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: Yes FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: Yes FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: Yes Runtime Instrumentation (slow code): No Here's some log output from around the time of one of these incidents; I don't know if it helps, but the data on the console after the incident looks a lot like the end of this set of entries. Nov 4 08:10:05 prajna rsyslogd: [origin software="rsyslogd" swVersion="4.4.2" x-pid="15425" x-info="http://www.rsyslog.com"] (re)start Nov 4 08:10:05 prajna kernel: [47633.000709] Killed process 13665 (rsyslogd) Nov 4 08:10:05 prajna kernel: [47633.000612] 191505 pages non-shared Nov 4 08:10:05 prajna kernel: [47633.000636] Out of memory: kill process 13665 (rsyslogd) score 14137 or a child Nov 4 08:10:05 prajna kernel: [47633.000575] 2870 pages reserved Nov 4 08:10:05 prajna kernel: [47633.000594] 221 pages shared Nov 4 08:10:05 prajna kernel: [47633.000529] 196608 pages RAM Nov 4 08:10:05 prajna kernel: [47633.000557] 0 pages HighMem Nov 4 08:10:05 prajna kernel: [47632.970129] Free swap = 0kB Nov 4 08:10:05 prajna kernel: [47632.970148] Total swap = 1951856kB Nov 4 08:10:05 prajna kernel: [47632.970081] 0 pages in swap cache Nov 4 08:10:05 prajna kernel: [47632.970103] Swap cache stats: add 497948, delete 497948, find 68248/68574 Nov 4 08:10:05 prajna kernel: [47632.970061] 209 total pagecache pages Nov 4 08:10:05 prajna kernel: [47632.969982] Normal: 6*4kB 1*8kB 1*16kB 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3440kB Nov 4 08:10:05 prajna kernel: [47632.969873] lowmem_reserve[]: 0 0 0 0 Nov 4 08:10:05 prajna kernel: [47632.969905] DMA: 1*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3052kB Nov 4 08:10:05 prajna kernel: [47632.969763] lowmem_reserve[]: 0 746 746 746 Nov 4 08:10:05 prajna kernel: [47632.969804] Normal free:3456kB min:3460kB low:4324kB high:5188kB active_anon:366232kB inactive_anon:366324kB active_file:608kB inactive_file:156kB unevictable:64kB present:764032kB pages_scanned:1512018 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47632.968951] free:1627 slab:1995 mapped:1 pagetables:944 bounce:0 Nov 4 08:10:05 prajna kernel: [47632.969024] DMA free:3052kB min:68kB low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB active_file:0kB inactive_file:16kB unevictable:0kB present:15868kB pages_scanned:15745 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47632.968946] inactive_file:43 unevictable:16 dirty:0 writeback:0 unstable:0 Nov 4 08:10:05 prajna kernel: [47632.968909] CPU 0: hi: 186, btch: 31 usd: 171 Nov 4 08:10:05 prajna kernel: [47632.968941] Active_anon:92646 active_file:152 inactive_anon:92691 Nov 4 08:10:05 prajna kernel: [47632.968889] Normal per-cpu: Nov 4 08:10:05 prajna kernel: [47632.968827] Mem-Info: Nov 4 08:10:05 prajna kernel: [47632.968846] DMA per-cpu: Nov 4 08:10:05 prajna kernel: [47632.968866] CPU 0: hi: 0, btch: 1 usd: 0 Nov 4 08:10:05 prajna kernel: [47632.968803] [] ? do_page_fault+0x0/0x1e7 Nov 4 08:10:05 prajna kernel: [47632.968731] [] ? do_page_fault+0x0/0x1e7 Nov 4 08:10:05 prajna kernel: [47632.968774] [] ? error_code+0x6d/0x74 Nov 4 08:10:05 prajna kernel: [47632.968702] [] ? do_page_fault+0x1d8/0x1e7 Nov 4 08:10:05 prajna kernel: [47632.968623] [] ? handle_mm_fault+0x2bb/0x652 Nov 4 08:10:05 prajna kernel: [47632.968658] [] ? fd_install+0x1e/0x3c Nov 4 08:10:05 prajna kernel: [47632.968592] [] ? irq_exit+0x31/0x53 Nov 4 08:10:05 prajna kernel: [47632.968549] [] ? __do_fault+0x47/0x355 Nov 4 08:10:05 prajna kernel: [47632.968508] [] ? kunmap_atomic+0x63/0x72 Nov 4 08:10:05 prajna kernel: [47632.968442] [] ? do_page_cache_readahead+0x3d/0x47 Nov 4 08:10:05 prajna kernel: [47632.968474] [] ? filemap_fault+0x154/0x315 Nov 4 08:10:05 prajna kernel: [47632.968410] [] ? __do_page_cache_readahead+0x98/0x16b Nov 4 08:10:05 prajna kernel: [47632.968376] [] ? __alloc_pages_internal+0x2ee/0x39d Nov 4 08:10:05 prajna kernel: [47632.968342] [] ? out_of_memory+0x5a/0x7c Nov 4 08:10:05 prajna kernel: [47632.968311] [] ? __out_of_memory+0x33/0x10b Nov 4 08:10:05 prajna kernel: [47632.968231] Call Trace: Nov 4 08:10:05 prajna kernel: [47632.968279] [] ? oom_kill_process+0x79/0x1f7 Nov 4 08:10:05 prajna kernel: [47632.968173] hald-addon-stor cpuset=/ mems_allowed=0 Nov 4 08:10:05 prajna kernel: [47632.968204] Pid: 1965, comm: hald-addon-stor Not tainted 2.6.30-2-686 #1 Nov 4 08:10:05 prajna kernel: [47632.968114] hald-addon-stor invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 Nov 4 08:10:05 prajna kernel: [47631.204598] Out of memory: kill process 6420 (rsyslogd) score 21205 or a child Nov 4 08:10:05 prajna kernel: [47631.204667] Killed process 6420 (rsyslogd) Nov 4 08:10:05 prajna kernel: [47631.204555] 207 pages shared Nov 4 08:10:05 prajna kernel: [47631.204573] 191533 pages non-shared Nov 4 08:10:05 prajna kernel: [47631.204518] 0 pages HighMem Nov 4 08:10:05 prajna kernel: [47631.204536] 2870 pages reserved Nov 4 08:10:05 prajna kernel: [47631.174161] Total swap = 1951856kB Nov 4 08:10:05 prajna kernel: [47631.204489] 196608 pages RAM Nov 4 08:10:05 prajna kernel: [47631.174142] Free swap = 0kB Nov 4 08:10:05 prajna kernel: [47631.174093] 14 pages in swap cache Nov 4 08:10:05 prajna kernel: [47631.174116] Swap cache stats: add 497939, delete 497925, find 68248/68574 Nov 4 08:10:05 prajna kernel: [47631.173995] Normal: 4*4kB 1*8kB 1*16kB 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3432kB Nov 4 08:10:05 prajna kernel: [47631.174073] 230 total pagecache pages Nov 4 08:10:05 prajna kernel: [47631.173917] DMA: 1*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3052kB Nov 4 08:10:05 prajna kernel: [47631.173817] Normal free:3448kB min:3460kB low:4324kB high:5188kB active_anon:366252kB inactive_anon:366360kB active_file:280kB inactive_file:584kB unevictable:64kB present:764032kB pages_scanned:1830820 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47631.173885] lowmem_reserve[]: 0 0 0 0 Nov 4 08:10:05 prajna kernel: [47631.173775] lowmem_reserve[]: 0 746 746 746 Nov 4 08:10:05 prajna kernel: [47631.173710] DMA free:3052kB min:68kB low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB active_file:0kB inactive_file:0kB unevictable:0kB present:15868kB pages_scanned:15745 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47631.173637] free:1625 slab:2000 mapped:1 pagetables:944 bounce:0 Nov 4 08:10:05 prajna kernel: [47631.173627] Active_anon:92651 active_file:70 inactive_anon:92700 Nov 4 08:10:05 prajna kernel: [47631.173632] inactive_file:146 unevictable:16 dirty:0 writeback:0 unstable:0 Nov 4 08:10:05 prajna kernel: [47631.173575] Normal per-cpu: Nov 4 08:10:05 prajna kernel: [47631.173596] CPU 0: hi: 186, btch: 31 usd: 154 Nov 4 08:10:05 prajna kernel: [47631.173532] DMA per-cpu: Nov 4 08:10:05 prajna kernel: [47631.173552] CPU 0: hi: 0, btch: 1 usd: 0 Nov 4 08:10:05 prajna kernel: [47631.173514] Mem-Info: Nov 4 08:10:05 prajna kernel: [47631.173461] [] ? error_code+0x6d/0x74 Nov 4 08:10:05 prajna kernel: [47631.173490] [] ? do_page_fault+0x0/0x1e7 Nov 4 08:10:05 prajna kernel: [47631.173420] [] ? do_page_fault+0x0/0x1e7 Nov 4 08:10:05 prajna kernel: [47631.173348] [] ? sys_socketcall+0x103/0x19f Nov 4 08:10:05 prajna kernel: [47631.173390] [] ? do_page_fault+0x1d8/0x1e7 Nov 4 08:10:05 prajna kernel: [47631.173319] [] ? sys_recv+0x19/0x1d Nov 4 08:10:05 prajna kernel: [47631.173280] [] ? handle_mm_fault+0x2bb/0x652 Nov 4 08:10:05 prajna kernel: [47631.173208] [] ? kunmap_atomic+0x63/0x72 Nov 4 08:10:05 prajna kernel: [47631.173248] [] ? __do_fault+0x47/0x355 Nov 4 08:10:05 prajna kernel: [47631.173173] [] ? filemap_fault+0x154/0x315 Nov 4 08:10:05 prajna kernel: [47631.173110] [] ? __do_page_cache_readahead+0x98/0x16b Nov 4 08:10:05 prajna kernel: [47631.173142] [] ? do_page_cache_readahead+0x3d/0x47 Nov 4 08:10:05 prajna kernel: [47631.173011] [] ? __out_of_memory+0x33/0x10b Nov 4 08:10:05 prajna kernel: [47631.173042] [] ? out_of_memory+0x5a/0x7c Nov 4 08:10:05 prajna kernel: [47631.173076] [] ? __alloc_pages_internal+0x2ee/0x39d Nov 4 08:10:05 prajna kernel: [47631.172979] [] ? oom_kill_process+0x79/0x1f7 Nov 4 08:10:05 prajna kernel: [47631.172935] Call Trace: Nov 4 08:10:05 prajna kernel: [47631.172909] Pid: 13665, comm: rsyslogd Not tainted 2.6.30-2-686 #1 Nov 4 08:10:05 prajna kernel: [47631.172829] rsyslogd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 Nov 4 08:10:05 prajna kernel: [47631.172881] rsyslogd cpuset=/ mems_allowed=0 Nov 4 08:10:05 prajna kernel: [47631.136403] Killed process 13664 (rsyslogd) Nov 4 08:10:05 prajna kernel: [47631.136310] 191499 pages non-shared Nov 4 08:10:05 prajna kernel: [47631.136334] Out of memory: kill process 13664 (rsyslogd) score 42411 or a child Nov 4 08:10:05 prajna kernel: [47631.136272] 2870 pages reserved Nov 4 08:10:05 prajna kernel: [47631.136291] 203 pages shared Nov 4 08:10:05 prajna kernel: [47631.136226] 196608 pages RAM Nov 4 08:10:05 prajna kernel: [47631.136254] 0 pages HighMem Nov 4 08:10:05 prajna kernel: [47631.105740] Total swap = 1951856kB Nov 4 08:10:05 prajna kernel: [47631.105722] Free swap = 0kB Nov 4 08:10:05 prajna kernel: [47631.105696] Swap cache stats: add 497920, delete 497920, find 68247/68572 Nov 4 08:10:05 prajna kernel: [47631.105674] 0 pages in swap cache Nov 4 08:10:05 prajna kernel: [47631.105653] 152 total pagecache pages Nov 4 08:10:05 prajna kernel: [47631.105574] Normal: 33*4kB 2*8kB 1*16kB 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3556kB Nov 4 08:10:05 prajna kernel: [47631.105496] DMA: 1*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3052kB Nov 4 08:10:05 prajna kernel: [47631.105465] lowmem_reserve[]: 0 0 0 0 Nov 4 08:10:05 prajna kernel: [47631.105396] Normal free:3556kB min:3460kB low:4324kB high:5188kB active_anon:366288kB inactive_anon:366268kB active_file:280kB inactive_file:324kB unevictable:64kB present:764032kB pages_scanned:1830304 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47631.105355] lowmem_reserve[]: 0 746 746 746 Nov 4 08:10:05 prajna kernel: [47631.105289] DMA free:3052kB min:68kB low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB active_file:0kB inactive_file:0kB unevictable:0kB present:15868kB pages_scanned:15745 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47631.105217] free:1652 slab:2000 mapped:1 pagetables:944 bounce:0 Nov 4 08:10:05 prajna kernel: [47631.105207] Active_anon:92660 active_file:70 inactive_anon:92677 Nov 4 08:10:05 prajna kernel: [47631.105212] inactive_file:81 unevictable:16 dirty:0 writeback:0 unstable:0 Nov 4 08:10:05 prajna kernel: [47631.105155] Normal per-cpu: Nov 4 08:10:05 prajna kernel: [47631.105175] CPU 0: hi: 186, btch: 31 usd: 161 Nov 4 08:10:05 prajna kernel: [47631.105132] CPU 0: hi: 0, btch: 1 usd: 0 Nov 4 08:10:05 prajna kernel: [47631.105093] Mem-Info: Nov 4 08:10:05 prajna kernel: [47631.105112] DMA per-cpu: Nov 4 08:10:05 prajna kernel: [47631.105069] [] ? do_page_fault+0x0/0x1e7 Nov 4 08:10:05 prajna kernel: imklog 4.4.2, log source = /proc/kmsg started. Nov 4 08:10:05 prajna kernel: [] ? error_code+0x6d/0x74 From rgerhards at hq.adiscon.com Thu Nov 5 17:17:16 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:17:16 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710335A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335B@GRFEXC.intern.adiscon.com> Hi Michael, Is this a standard debian config? I ask because it works well for me... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, November 05, 2009 5:03 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog v5 5.2.0 > > Another issue I noticed with 5.2.0 (+said patch) is, that it no longer > shuts down cleany on SIGTERM/SIGINT > If I run it with -c 4 -d and press STRG+C, I get > ... > 6895.847489380:b7692940: Terminating outputs... > 6895.847516758:b7692940: destructing ruleset 0x89c7648, name 0x89c7678 > 6895.847553634:b7692940: strm 0x89c97b8: file -1 flush, buflen 0 > 6895.847595539:b7692940: strm 0x89ca1d8: file 5 flush, buflen 0 > 6895.847627386:b7692940: strm 0x89ca1d8: file 5 closing > 6895.847688567:b7692940: strm 0x89cac48: file -1 flush, buflen 0 > 6895.847728796:b7692940: strm 0x89cb680: file 6 flush, buflen 0 > 6895.847758967:b7692940: strm 0x89cb680: file 6 closing > 6895.847805062:b7692940: strm 0x89cc108: file -1 flush, buflen 0 > 6895.847846129:b7692940: strm 0x89ccb78: file -1 flush, buflen 0 > 6895.847886916:b7692940: strm 0x89cd5c0: file -1 flush, buflen 0 > 6895.847927145:b7692940: strm 0x89ce060: file -1 flush, buflen 0 > 6895.847967653:b7692940: strm 0x89cea98: file -1 flush, buflen 0 > 6895.848007882:b7692940: strm 0x89cf500: file -1 flush, buflen 0 > 6895.848048110:b7692940: strm 0x89cff68: file -1 flush, buflen 0 > 6895.848088897:b7692940: strm 0x89d09d8: file -1 flush, buflen 0 > 6895.848129126:b7692940: strm 0x89d1478: file -1 flush, buflen 0 > 6895.848169355:b7692940: strm 0x89d1ee8: file -1 flush, buflen 0 > 6895.848220758:b7692940: strm 0x89d2990: file 7 flush, buflen 0 > 6895.848251488:b7692940: strm 0x89d2990: file 7 closing > 6895.848300098:b7692940: strm 0x89d38c0: file -1 flush, buflen 77 > > > pressing STRG+C a second time will then terminate rsyslogd. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Nov 5 17:23:32 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:23:32 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335C@GRFEXC.intern.adiscon.com> Michael, > I can confirm that a5cd509be736fcff3c8ae3104712d2fe85776416 fixes this > issue for me using 5.2.0. > What about the -c compat flag? Is it intentional that there is no -c 5? > I will probably upload a 5.2.* to unstable (or experimental) soon, > which should give it some wider exposure. one more side-note: to do this, do you need a "stable" tag attached to the version by the upstream? I am asking because it would be much more useful to have wider exposure for 5.3.4, whereas 5.2.0 will probably not provide that much useful feedback (the most interesting pieces inside the queue engine already have been rewritten in 5.3). Still, it is good to see that the stable tag now draws attention to the version, I begin to think that nobody ever tried the beta ;) Maybe I should attach stable tags earlier in the future. Rainer From david at lang.hm Thu Nov 5 17:19:17 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 5 Nov 2009 08:19:17 -0800 (PST) Subject: [rsyslog] rsyslog v5 5.2.0 In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 5 Nov 2009, Michael Biebl wrote: > 2009/11/4 Rainer Gerhards : >> Michael, >> >> That is very probably the same regression that Luis Fernando mentioned for >> 4.5.5. All in all, I would be quite careful with 5.2.0, I really have the >> impression that folks simply didn't try it out. But as I wrote, I need to >> somehow get the process started to come to make v5 mainstream. It is really >> important to me to get rid of v4-devel, as it takes considerable time (and >> bug potential!) to have two branches that are in development state. > > I can confirm that a5cd509be736fcff3c8ae3104712d2fe85776416 fixes this > issue for me using 5.2.0. > What about the -c compat flag? Is it intentional that there is no -c 5? are you sure there isn't? I've been running dev versions of 5.x and hav eneeded to use -c5 with them instead of -c4 > I will probably upload a 5.2.* to unstable (or experimental) soon, > which should give it some wider exposure. I would not use 5.2.0, it had many problems under load (and problems where invalid syslog messages could cause it to crash) I'm currently running a slightly later git snapshot reliably under fairly heavy load (>10k logs/sec sustained for hours at a time). unfortunantly I am 3 timezones away from the office or I would upgrade to the 5.3 that was just released. I will proabaly do so tuesday when I am back in the office. David Lang > Cheers, > Michael > > >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com >>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Michael Biebl >>> Sent: Wednesday, November 04, 2009 7:30 PM >>> To: rsyslog-users >>> Subject: [rsyslog] rsyslog v5 5.2.0 >>> >>> Hi Rainer, hi list >>> >>> I tried the latest v5 stable release. >>> First thing I noticed is, that it still uses -c 4 as native mode. >>> I also noticed problems with logging to a pipe. The default >>> rsyslog.conf file on Debian has a >>> daemon.*;mail.*;\ >>> ? ? ? ? news.err;\ >>> ? ? ? ? *.=debug;*.=info;\ >>> ? ? ? ? *.=notice;*.=warn ? ? ? |/dev/xconsole >>> >>> But I get the following error: >>> >>> 9042.679173747:b7607b70: file to log to: |/dev/xconsole >>> 9042.679183525:b7607b70: doWrite, pData->pStrm 0x814d8c0, lenBuf 131 >>> 9042.679194699:b7607b70: strm 0x814d8c0: file -1 flush, buflen 208 >>> 9042.679225429:b7607b70: strm 0x814d8c0: open error 2, file >>> '|/dev/xconsole' >>> >>> I've put a complete debug log + config files at >>> http://debs.michaelbiebl.de/rsyslog/v5 >>> >>> Rainer, maybe you can take a look. >>> >>> Thanks, >>> Michael >>> >>> -- >>> Why is it that all of the instruments seeking intelligent life in the >>> universe are pointed away from Earth? >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > > > From rgerhards at hq.adiscon.com Thu Nov 5 17:29:52 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:29:52 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335D@GRFEXC.intern.adiscon.com> > > I will probably upload a 5.2.* to unstable (or experimental) soon, > > which should give it some wider exposure. > > I would not use 5.2.0, it had many problems under load (and problems > where > invalid syslog messages could cause it to crash) The invalid message problem should be fixed in 5.2.0, this was merged in to all v5 versions from the v4 fix. But I won't touch any queue problems, and I am sure there exist some during shutdown. > I'm currently running a slightly later git snapshot reliably under > fairly > heavy load (>10k logs/sec sustained for hours at a time). unfortunantly > I > am 3 timezones away from the office or I would upgrade to the 5.3 that > was > just released. I will proabaly do so tuesday when I am back in the > office. Looking forward to the results. I intend to promote that version to beta soon and try to push it through a very short beta cycle. I do not plan any more enhancements within the short-term timeframe. I looks like the advanced features begin to get used and so finally bug reports come in, so it is bug fixing and instrumentation time :) Rainer From correo at miguelangelnieto.net Thu Nov 5 17:31:33 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Thu, 5 Nov 2009 17:31:33 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103343@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103343@GRFEXC.intern.adiscon.com> Message-ID: Thank you! It works now! :) This is the final rsyslog.conf ############## $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # kernel logging (formerly provided by rklogd) $WorkDirectory /var/log/queue $ActionQueueType LinkedList $ActionQueueFileName SL $ActionQueueMaxDiskSpace 5g $ActionQueueMaxFileSize 10m $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 if $msg contains 'TL' or $msg contains 'CD' or $msg contains 'TR' then @@192.168.1.222 & ~ *.* /var/log/syslog kern.* /dev/console *.info;mail.none;authpriv.none;cron.none -/var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* -/var/log/cron uucp,news.crit -/var/log/spooler local7.* /var/log/boot.log ############# 2009/11/5 Rainer Gerhards : >> 0726.855937980:imuxsock.c: main queue: enqueueMsg: LightDelay mark >> reached for light delayble message - blocking a bit. >> 0727.858592935:imuxsock.c: main queue: enqueueMsg: LightDelay mark >> reached for light delayble message - blocking a bit. >> 0728.862998771:imuxsock.c: main queue: enqueueMsg: LightDelay mark >> reached for light delayble message - blocking a bit. >> >> Rsyslog 3.19.7 (Suse) > > The version is the issue - that's an outdated beta version ;) Some versions > had unix sockets flagged as sources that could lightly be delayed, resulting > in what you see. That was made optional in later builds (and I guess nobody > has turned it on since then ;)). > > Cure: upgrade to the current (3.22.1) v3-stable. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From rgerhards at hq.adiscon.com Thu Nov 5 17:37:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:37:44 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> Can you send me your rsyslog.conf, so that I can run it under the memory debugger in my lab. I'll also take this as a motivation to finally add multi-daemon tests to the testbench (what may take me a little while...). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Thursday, November 05, 2009 5:14 PM > To: rsyslog at lists.adiscon.com >> rsyslog-users > Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Hi, > > I'm running a central rsyslog server with a couple of remote WAN > (internet) clients and several remote LAN clients. Traffic is low - of > the order of 10,000 messages per day. Internet clients communicate with > the server using gnutls. LAN clients are currently using UDP. The > server > writes client logs to mysql, and also writes messages of local origin > to > disk. > > After some interval x::x <= 24h, the server consumes all memory (and > swap) in the system. It looks as if the OS then tries to evict > rsyslogd, > and hangs - there's what looks like a stacktrace displayed on the > (dead) > console. > > I've stopped using gnutls for now, because the server machine is also a > NFS file-server, and other systems depend on it not shutting down like > this. It seems the problem doesn't occur with plain tcp. > > # uname -a > Linux prajna 2.6.30-2-686 #1 SMP Sat Sep 26 01:16:22 UTC 2009 i686 > GNU/Linux > > # rsyslogd -v > rsyslogd 4.4.2, compiled with: > FEATURE_REGEXP: Yes > FEATURE_LARGEFILE: Yes > FEATURE_NETZIP (message compression): Yes > GSSAPI Kerberos 5 support: Yes > FEATURE_DEBUG (debug build, slow code): No > Atomic operations supported: Yes > Runtime Instrumentation (slow code): No > > Here's some log output from around the time of one of these incidents; > I > don't know if it helps, but the data on the console after the incident > looks a lot like the end of this set of entries. > > Nov 4 08:10:05 prajna rsyslogd: [origin software="rsyslogd" > swVersion="4.4.2" x-pid="15425" x-info="http://www.rsyslog.com"] > (re)start > Nov 4 08:10:05 prajna kernel: [47633.000709] Killed process 13665 > (rsyslogd) > Nov 4 08:10:05 prajna kernel: [47633.000612] 191505 pages non-shared > Nov 4 08:10:05 prajna kernel: [47633.000636] Out of memory: kill > process 13665 (rsyslogd) score 14137 or a child > Nov 4 08:10:05 prajna kernel: [47633.000575] 2870 pages reserved > Nov 4 08:10:05 prajna kernel: [47633.000594] 221 pages shared > Nov 4 08:10:05 prajna kernel: [47633.000529] 196608 pages RAM > Nov 4 08:10:05 prajna kernel: [47633.000557] 0 pages HighMem > Nov 4 08:10:05 prajna kernel: [47632.970129] Free swap = 0kB > Nov 4 08:10:05 prajna kernel: [47632.970148] Total swap = 1951856kB > Nov 4 08:10:05 prajna kernel: [47632.970081] 0 pages in swap cache > Nov 4 08:10:05 prajna kernel: [47632.970103] Swap cache stats: add > 497948, delete 497948, find 68248/68574 > Nov 4 08:10:05 prajna kernel: [47632.970061] 209 total pagecache pages > Nov 4 08:10:05 prajna kernel: [47632.969982] Normal: 6*4kB 1*8kB > 1*16kB > 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = > 3440kB > Nov 4 08:10:05 prajna kernel: [47632.969873] lowmem_reserve[]: 0 0 0 0 > Nov 4 08:10:05 prajna kernel: [47632.969905] DMA: 1*4kB 1*8kB 0*16kB > 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = > 3052kB > Nov 4 08:10:05 prajna kernel: [47632.969763] lowmem_reserve[]: 0 746 > 746 746 > Nov 4 08:10:05 prajna kernel: [47632.969804] Normal free:3456kB > min:3460kB low:4324kB high:5188kB active_anon:366232kB > inactive_anon:366324kB active_file:608kB inactive_file:156kB > unevictable:64kB present:764032kB pages_scanned:1512018 > all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47632.968951] free:1627 slab:1995 > mapped:1 pagetables:944 bounce:0 > Nov 4 08:10:05 prajna kernel: [47632.969024] DMA free:3052kB min:68kB > low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB > active_file:0kB inactive_file:16kB unevictable:0kB present:15868kB > pages_scanned:15745 all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47632.968946] inactive_file:43 > unevictable:16 dirty:0 writeback:0 unstable:0 > Nov 4 08:10:05 prajna kernel: [47632.968909] CPU 0: hi: 186, btch: > 31 usd: 171 > Nov 4 08:10:05 prajna kernel: [47632.968941] Active_anon:92646 > active_file:152 inactive_anon:92691 > Nov 4 08:10:05 prajna kernel: [47632.968889] Normal per-cpu: > Nov 4 08:10:05 prajna kernel: [47632.968827] Mem-Info: > Nov 4 08:10:05 prajna kernel: [47632.968846] DMA per-cpu: > Nov 4 08:10:05 prajna kernel: [47632.968866] CPU 0: hi: 0, btch: > 1 usd: 0 > Nov 4 08:10:05 prajna kernel: [47632.968803] [] ? > do_page_fault+0x0/0x1e7 > Nov 4 08:10:05 prajna kernel: [47632.968731] [] ? > do_page_fault+0x0/0x1e7 > Nov 4 08:10:05 prajna kernel: [47632.968774] [] ? > error_code+0x6d/0x74 > Nov 4 08:10:05 prajna kernel: [47632.968702] [] ? > do_page_fault+0x1d8/0x1e7 > Nov 4 08:10:05 prajna kernel: [47632.968623] [] ? > handle_mm_fault+0x2bb/0x652 > Nov 4 08:10:05 prajna kernel: [47632.968658] [] ? > fd_install+0x1e/0x3c > Nov 4 08:10:05 prajna kernel: [47632.968592] [] ? > irq_exit+0x31/0x53 > Nov 4 08:10:05 prajna kernel: [47632.968549] [] ? > __do_fault+0x47/0x355 > Nov 4 08:10:05 prajna kernel: [47632.968508] [] ? > kunmap_atomic+0x63/0x72 > Nov 4 08:10:05 prajna kernel: [47632.968442] [] ? > do_page_cache_readahead+0x3d/0x47 > Nov 4 08:10:05 prajna kernel: [47632.968474] [] ? > filemap_fault+0x154/0x315 > Nov 4 08:10:05 prajna kernel: [47632.968410] [] ? > __do_page_cache_readahead+0x98/0x16b > Nov 4 08:10:05 prajna kernel: [47632.968376] [] ? > __alloc_pages_internal+0x2ee/0x39d > Nov 4 08:10:05 prajna kernel: [47632.968342] [] ? > out_of_memory+0x5a/0x7c > Nov 4 08:10:05 prajna kernel: [47632.968311] [] ? > __out_of_memory+0x33/0x10b > Nov 4 08:10:05 prajna kernel: [47632.968231] Call Trace: > Nov 4 08:10:05 prajna kernel: [47632.968279] [] ? > oom_kill_process+0x79/0x1f7 > Nov 4 08:10:05 prajna kernel: [47632.968173] hald-addon-stor cpuset=/ > mems_allowed=0 > Nov 4 08:10:05 prajna kernel: [47632.968204] Pid: 1965, comm: > hald-addon-stor Not tainted 2.6.30-2-686 #1 > Nov 4 08:10:05 prajna kernel: [47632.968114] hald-addon-stor invoked > oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 > Nov 4 08:10:05 prajna kernel: [47631.204598] Out of memory: kill > process 6420 (rsyslogd) score 21205 or a child > Nov 4 08:10:05 prajna kernel: [47631.204667] Killed process 6420 > (rsyslogd) > Nov 4 08:10:05 prajna kernel: [47631.204555] 207 pages shared > Nov 4 08:10:05 prajna kernel: [47631.204573] 191533 pages non-shared > Nov 4 08:10:05 prajna kernel: [47631.204518] 0 pages HighMem > Nov 4 08:10:05 prajna kernel: [47631.204536] 2870 pages reserved > Nov 4 08:10:05 prajna kernel: [47631.174161] Total swap = 1951856kB > Nov 4 08:10:05 prajna kernel: [47631.204489] 196608 pages RAM > Nov 4 08:10:05 prajna kernel: [47631.174142] Free swap = 0kB > Nov 4 08:10:05 prajna kernel: [47631.174093] 14 pages in swap cache > Nov 4 08:10:05 prajna kernel: [47631.174116] Swap cache stats: add > 497939, delete 497925, find 68248/68574 > Nov 4 08:10:05 prajna kernel: [47631.173995] Normal: 4*4kB 1*8kB > 1*16kB > 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = > 3432kB > Nov 4 08:10:05 prajna kernel: [47631.174073] 230 total pagecache pages > Nov 4 08:10:05 prajna kernel: [47631.173917] DMA: 1*4kB 1*8kB 0*16kB > 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = > 3052kB > Nov 4 08:10:05 prajna kernel: [47631.173817] Normal free:3448kB > min:3460kB low:4324kB high:5188kB active_anon:366252kB > inactive_anon:366360kB active_file:280kB inactive_file:584kB > unevictable:64kB present:764032kB pages_scanned:1830820 > all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47631.173885] lowmem_reserve[]: 0 0 0 0 > Nov 4 08:10:05 prajna kernel: [47631.173775] lowmem_reserve[]: 0 746 > 746 746 > Nov 4 08:10:05 prajna kernel: [47631.173710] DMA free:3052kB min:68kB > low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB > active_file:0kB inactive_file:0kB unevictable:0kB present:15868kB > pages_scanned:15745 all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47631.173637] free:1625 slab:2000 > mapped:1 pagetables:944 bounce:0 > Nov 4 08:10:05 prajna kernel: [47631.173627] Active_anon:92651 > active_file:70 inactive_anon:92700 > Nov 4 08:10:05 prajna kernel: [47631.173632] inactive_file:146 > unevictable:16 dirty:0 writeback:0 unstable:0 > Nov 4 08:10:05 prajna kernel: [47631.173575] Normal per-cpu: > Nov 4 08:10:05 prajna kernel: [47631.173596] CPU 0: hi: 186, btch: > 31 usd: 154 > Nov 4 08:10:05 prajna kernel: [47631.173532] DMA per-cpu: > Nov 4 08:10:05 prajna kernel: [47631.173552] CPU 0: hi: 0, btch: > 1 usd: 0 > Nov 4 08:10:05 prajna kernel: [47631.173514] Mem-Info: > Nov 4 08:10:05 prajna kernel: [47631.173461] [] ? > error_code+0x6d/0x74 > Nov 4 08:10:05 prajna kernel: [47631.173490] [] ? > do_page_fault+0x0/0x1e7 > Nov 4 08:10:05 prajna kernel: [47631.173420] [] ? > do_page_fault+0x0/0x1e7 > Nov 4 08:10:05 prajna kernel: [47631.173348] [] ? > sys_socketcall+0x103/0x19f > Nov 4 08:10:05 prajna kernel: [47631.173390] [] ? > do_page_fault+0x1d8/0x1e7 > Nov 4 08:10:05 prajna kernel: [47631.173319] [] ? > sys_recv+0x19/0x1d > Nov 4 08:10:05 prajna kernel: [47631.173280] [] ? > handle_mm_fault+0x2bb/0x652 > Nov 4 08:10:05 prajna kernel: [47631.173208] [] ? > kunmap_atomic+0x63/0x72 > Nov 4 08:10:05 prajna kernel: [47631.173248] [] ? > __do_fault+0x47/0x355 > Nov 4 08:10:05 prajna kernel: [47631.173173] [] ? > filemap_fault+0x154/0x315 > Nov 4 08:10:05 prajna kernel: [47631.173110] [] ? > __do_page_cache_readahead+0x98/0x16b > Nov 4 08:10:05 prajna kernel: [47631.173142] [] ? > do_page_cache_readahead+0x3d/0x47 > Nov 4 08:10:05 prajna kernel: [47631.173011] [] ? > __out_of_memory+0x33/0x10b > Nov 4 08:10:05 prajna kernel: [47631.173042] [] ? > out_of_memory+0x5a/0x7c > Nov 4 08:10:05 prajna kernel: [47631.173076] [] ? > __alloc_pages_internal+0x2ee/0x39d > Nov 4 08:10:05 prajna kernel: [47631.172979] [] ? > oom_kill_process+0x79/0x1f7 > Nov 4 08:10:05 prajna kernel: [47631.172935] Call Trace: > Nov 4 08:10:05 prajna kernel: [47631.172909] Pid: 13665, comm: > rsyslogd > Not tainted 2.6.30-2-686 #1 > Nov 4 08:10:05 prajna kernel: [47631.172829] rsyslogd invoked > oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 > Nov 4 08:10:05 prajna kernel: [47631.172881] rsyslogd cpuset=/ > mems_allowed=0 > Nov 4 08:10:05 prajna kernel: [47631.136403] Killed process 13664 > (rsyslogd) > Nov 4 08:10:05 prajna kernel: [47631.136310] 191499 pages non-shared > Nov 4 08:10:05 prajna kernel: [47631.136334] Out of memory: kill > process 13664 (rsyslogd) score 42411 or a child > Nov 4 08:10:05 prajna kernel: [47631.136272] 2870 pages reserved > Nov 4 08:10:05 prajna kernel: [47631.136291] 203 pages shared > Nov 4 08:10:05 prajna kernel: [47631.136226] 196608 pages RAM > Nov 4 08:10:05 prajna kernel: [47631.136254] 0 pages HighMem > Nov 4 08:10:05 prajna kernel: [47631.105740] Total swap = 1951856kB > Nov 4 08:10:05 prajna kernel: [47631.105722] Free swap = 0kB > Nov 4 08:10:05 prajna kernel: [47631.105696] Swap cache stats: add > 497920, delete 497920, find 68247/68572 > Nov 4 08:10:05 prajna kernel: [47631.105674] 0 pages in swap cache > Nov 4 08:10:05 prajna kernel: [47631.105653] 152 total pagecache pages > Nov 4 08:10:05 prajna kernel: [47631.105574] Normal: 33*4kB 2*8kB > 1*16kB 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB > = 3556kB > Nov 4 08:10:05 prajna kernel: [47631.105496] DMA: 1*4kB 1*8kB 0*16kB > 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = > 3052kB > Nov 4 08:10:05 prajna kernel: [47631.105465] lowmem_reserve[]: 0 0 0 0 > Nov 4 08:10:05 prajna kernel: [47631.105396] Normal free:3556kB > min:3460kB low:4324kB high:5188kB active_anon:366288kB > inactive_anon:366268kB active_file:280kB inactive_file:324kB > unevictable:64kB present:764032kB pages_scanned:1830304 > all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47631.105355] lowmem_reserve[]: 0 746 > 746 746 > Nov 4 08:10:05 prajna kernel: [47631.105289] DMA free:3052kB min:68kB > low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB > active_file:0kB inactive_file:0kB unevictable:0kB present:15868kB > pages_scanned:15745 all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47631.105217] free:1652 slab:2000 > mapped:1 pagetables:944 bounce:0 > Nov 4 08:10:05 prajna kernel: [47631.105207] Active_anon:92660 > active_file:70 inactive_anon:92677 > Nov 4 08:10:05 prajna kernel: [47631.105212] inactive_file:81 > unevictable:16 dirty:0 writeback:0 unstable:0 > Nov 4 08:10:05 prajna kernel: [47631.105155] Normal per-cpu: > Nov 4 08:10:05 prajna kernel: [47631.105175] CPU 0: hi: 186, btch: > 31 usd: 161 > Nov 4 08:10:05 prajna kernel: [47631.105132] CPU 0: hi: 0, btch: > 1 usd: 0 > Nov 4 08:10:05 prajna kernel: [47631.105093] Mem-Info: > Nov 4 08:10:05 prajna kernel: [47631.105112] DMA per-cpu: > Nov 4 08:10:05 prajna kernel: [47631.105069] [] ? > do_page_fault+0x0/0x1e7 > Nov 4 08:10:05 prajna kernel: imklog 4.4.2, log source = /proc/kmsg > started. > Nov 4 08:10:05 prajna kernel: [] ? error_code+0x6d/0x74 > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mrdemeanour at jackpot.uk.net Thu Nov 5 17:41:15 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Thu, 05 Nov 2009 16:41:15 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> References: <4AF2F9CD.9000402@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> Message-ID: <4AF3002B.9060302@jackpot.uk.net> Rainer Gerhards wrote: > Can you send me your rsyslog.conf, so that I can run it under the memory > debugger in my lab. I'll also take this as a motivation to finally add > multi-daemon tests to the testbench (what may take me a little while...). This is the server config (some of the remarks are misleading). # /etc/rsyslog.conf Configuration file for rsyslog v3. # # For more information see # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html # $DebugPrintTemplateList on # $ActionFileDefaultTemplate mysql-template ################# #### MODULES #### ################# $ModLoad imuxsock # provides support for local system logging # $ModLoad ommysql.so # provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 $ModLoad imklog # provides kernel logging support (previously done by rklogd) # provides TCP syslog reception $ModLoad imtcp # make gtls driver the default # $DefaultNetstreamDriver gtls $DefaultNetstreamDriver ptcp # certificate files $DefaultNetstreamDriverCAFile /etc/rsyslog.d/.ssl/gnu-ca-cert.pem $DefaultNetstreamDriverCertFile /etc/rsyslog.d/.ssl/saraha-rsyslog-cert.pem $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/.ssl/saraha-rsyslog-key.pem # $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode # $InputTCPServerStreamDriverAuthMode anon # client is NOT authenticated $InputTCPServerRun 10514 # start up listener at port 10514 $ModLoad MySQL ########################### ########################### #### GLOBAL DIRECTIVES #### ########################### # # Use default timestamp format. # To enable high precision timestamps, comment out the following line. # $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # # Set the default permissions for all log files. # $FileOwner root $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $WorkDirectory /var/log/rsyslog $template mysql-template, "insert into logs(host, facility, priority, level, tag, datetime, msg) values ('%source%', '%syslogfacility-text%', '%syslogpriority-text%', '%syslogseverity-text%', '%programname%', '%timereported:::date-mysql%', '%msg%')", sql, mysql # $template DEBUG,"Debug line with all properties:\nFROMHOST: '%FROMHOST%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nrawmsg: '%rawmsg%'\n\n" ############### #### RULES #### ############### #Discard some dross messages # authpriv.info ~ :HOSTNAME, isequal, "last" ~ #Discard router access messages for the script on Prajna that collects the router logs. if $msg contains 'User logged in on TELNET (192.168.1.2)' then ~ if $msg contains 'User logged out on TELNET (192.168.1.2)' then ~ # Log everything else to mysql. $ActionQueueType LinkedList # Number of elements... $ActionQueueSize 100 # $ActionQueueFileName mysql # $ActionQueueMaxDiskSpace 1M # $ActionQueueHighWaterMark 40 # $ActionQueueLowWaterMark 5 *.* >127.0.0.1,syslog,syslog,syslog;mysql-template $ActionExecOnlyWhenPreviousIsSuspended on & ~ $ActionExecOnlyWhenPreviousIsSuspended off #Log local stuff ONLY to /var/log/syslog :HOSTNAME, isequal, "prajna" -/var/log/syslog -- Jack. From rgerhards at hq.adiscon.com Thu Nov 5 17:42:01 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:42:01 +0100 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com><200910201639.43143.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> <200911031158.02666.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335F@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, November 03, 2009 11:58 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Hello Rainer, > > is there some news about this issue? Unfortuantely not at the moment (I guess you see it is a busy time). Could you give 5.3.4 a try? It would be very interesting (even for a v4-fix) to see if the problem persists or not... Rainer > > TIA > -Marc > > Am Dienstag, 20. Oktober 2009 18:38:16 schrieb Rainer Gerhards: > > thanks! > > > > Interesting, I see that only one of the messages that should be in > the .qi > > are actually present. I wonder if this is related to the fact that > ompgsql > > emits an error message itself while being down (the others don't do > this > > AFIK). Will try to dig down to this (but I have to do so as time > permits). > > > > Rainer > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Nov 5 17:45:07 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:45:07 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> Thanks, but I wasn't specific enough. For TLS, I also need to client config, because I need two machines to reproduce any issues (these two instances are also the challenge for the current testbench, what requires hopefully fewer than I expect changes ;)). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Thursday, November 05, 2009 5:41 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Rainer Gerhards wrote: > > Can you send me your rsyslog.conf, so that I can run it under the > memory > > debugger in my lab. I'll also take this as a motivation to finally > add > > multi-daemon tests to the testbench (what may take me a little > while...). > > This is the server config (some of the remarks are misleading). > > # /etc/rsyslog.conf Configuration file for rsyslog v3. > # > # For more information see > # /usr/share/doc/rsyslog- > doc/html/rsyslog_conf.html > > # $DebugPrintTemplateList on > # $ActionFileDefaultTemplate mysql-template > ################# > #### MODULES #### > ################# > > $ModLoad imuxsock # provides support for local system logging > > # $ModLoad ommysql.so > > # provides UDP syslog reception > $ModLoad imudp > $UDPServerRun 514 > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > > # provides TCP syslog reception > $ModLoad imtcp > > # make gtls driver the default > # $DefaultNetstreamDriver gtls > $DefaultNetstreamDriver ptcp > > # certificate files > $DefaultNetstreamDriverCAFile /etc/rsyslog.d/.ssl/gnu-ca-cert.pem > $DefaultNetstreamDriverCertFile /etc/rsyslog.d/.ssl/saraha-rsyslog- > cert.pem > $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/.ssl/saraha-rsyslog- > key.pem > > # $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode > # $InputTCPServerStreamDriverAuthMode anon # client is NOT > authenticated > $InputTCPServerRun 10514 # start up listener at port 10514 > > $ModLoad MySQL > > ########################### > ########################### > #### GLOBAL DIRECTIVES #### > ########################### > > # > # Use default timestamp format. > # To enable high precision timestamps, comment out the following line. > # > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > # > # Set the default permissions for all log files. > # > $FileOwner root > $FileGroup adm > $FileCreateMode 0640 > $DirCreateMode 0755 > $WorkDirectory /var/log/rsyslog > > $template mysql-template, "insert into logs(host, facility, priority, > level, tag, datetime, msg) values ('%source%', '%syslogfacility-text%', > '%syslogpriority-text%', '%syslogseverity-text%', '%programname%', > '%timereported:::date-mysql%', '%msg%')", sql, mysql > > # $template DEBUG,"Debug line with all properties:\nFROMHOST: > '%FROMHOST%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag > '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', > PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', > STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nrawmsg: > '%rawmsg%'\n\n" > > ############### > #### RULES #### > ############### > #Discard some dross messages > # authpriv.info ~ > :HOSTNAME, isequal, "last" ~ > > #Discard router access messages for the script on Prajna that collects > the router logs. > if $msg contains 'User logged in on TELNET (192.168.1.2)' then ~ > if $msg contains 'User logged out on TELNET (192.168.1.2)' then ~ > > # Log everything else to mysql. > $ActionQueueType LinkedList > # Number of elements... > $ActionQueueSize 100 > # $ActionQueueFileName mysql > # $ActionQueueMaxDiskSpace 1M > # $ActionQueueHighWaterMark 40 > # $ActionQueueLowWaterMark 5 > > > *.* > >127.0.0.1,syslog,syslog,syslog;mysql-template > > $ActionExecOnlyWhenPreviousIsSuspended on > & ~ > $ActionExecOnlyWhenPreviousIsSuspended off > > #Log local stuff ONLY to /var/log/syslog > :HOSTNAME, isequal, "prajna" -/var/log/syslog > > > -- > Jack. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Thu Nov 5 17:50:14 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 5 Nov 2009 17:50:14 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: 2009/11/5 : > On Thu, 5 Nov 2009, Michael Biebl wrote: > >> What about the -c compat flag? Is it intentional that there is no -c 5? > > are you sure there isn't? I've been running dev versions of 5.x and hav > eneeded to use -c5 with them instead of -c4 Well, at least with 5.2.0, rsyslogd does not complain if I start it with -c4 (complain in the sense, that rsyslog says it's running in non-native mode) >> I will probably upload a 5.2.* to unstable (or experimental) soon, >> which should give it some wider exposure. > > I would not use 5.2.0, it had many problems under load (and problems where > invalid syslog messages could cause it to crash) > > I'm currently running a slightly later git snapshot reliably under fairly > heavy load (>10k logs/sec sustained for hours at a time). unfortunantly I am > 3 timezones away from the office or I would upgrade to the 5.3 that was just > released. I will proabaly do so tuesday when I am back in the office. That's good to know. It seems the 5.2.* series is some kind of neglected so maybe it's best to just skip that branch. and wait for 5.4.* or stable 5.3.* beta releases. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Thu Nov 5 17:51:05 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 5 Nov 2009 17:51:05 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710335B@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710335A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710335B@GRFEXC.intern.adiscon.com> Message-ID: 2009/11/5 Rainer Gerhards : > Hi Michael, > > Is this a standard debian config? I ask because it works well for me... It's a standard Debian config as shipped with version 4.4.1-1 > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Michael Biebl >> Sent: Thursday, November 05, 2009 5:03 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] rsyslog v5 5.2.0 >> >> Another issue I noticed with 5.2.0 (+said patch) is, that it no longer >> shuts down cleany on SIGTERM/SIGINT >> If I run it with -c 4 -d and press STRG+C, I get >> ... >> 6895.847489380:b7692940: Terminating outputs... >> 6895.847516758:b7692940: destructing ruleset 0x89c7648, name 0x89c7678 >> 6895.847553634:b7692940: strm 0x89c97b8: file -1 flush, buflen 0 >> 6895.847595539:b7692940: strm 0x89ca1d8: file 5 flush, buflen 0 >> 6895.847627386:b7692940: strm 0x89ca1d8: file 5 closing >> 6895.847688567:b7692940: strm 0x89cac48: file -1 flush, buflen 0 >> 6895.847728796:b7692940: strm 0x89cb680: file 6 flush, buflen 0 >> 6895.847758967:b7692940: strm 0x89cb680: file 6 closing >> 6895.847805062:b7692940: strm 0x89cc108: file -1 flush, buflen 0 >> 6895.847846129:b7692940: strm 0x89ccb78: file -1 flush, buflen 0 >> 6895.847886916:b7692940: strm 0x89cd5c0: file -1 flush, buflen 0 >> 6895.847927145:b7692940: strm 0x89ce060: file -1 flush, buflen 0 >> 6895.847967653:b7692940: strm 0x89cea98: file -1 flush, buflen 0 >> 6895.848007882:b7692940: strm 0x89cf500: file -1 flush, buflen 0 >> 6895.848048110:b7692940: strm 0x89cff68: file -1 flush, buflen 0 >> 6895.848088897:b7692940: strm 0x89d09d8: file -1 flush, buflen 0 >> 6895.848129126:b7692940: strm 0x89d1478: file -1 flush, buflen 0 >> 6895.848169355:b7692940: strm 0x89d1ee8: file -1 flush, buflen 0 >> 6895.848220758:b7692940: strm 0x89d2990: file 7 flush, buflen 0 >> 6895.848251488:b7692940: strm 0x89d2990: file 7 closing >> 6895.848300098:b7692940: strm 0x89d38c0: file -1 flush, buflen 77 >> >> >> pressing STRG+C a second time will then terminate rsyslogd. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Nov 5 17:54:10 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:54:10 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103361@GRFEXC.intern.adiscon.com> > That's good to know. It seems the 5.2.* series is some kind of > neglected so maybe it's best to just skip that branch. and wait for > 5.4.* or stable 5.3.* beta releases. As I said on-list (and within the announcement), I needed to get a 5.2.0 version out. The problem was that there was so few feedback on 5.1.x that it looked like it worked. However, I knew that there are problems with queue termination. That all is solved (hopefully ;)) in the newer v5 versions, but I can't renumber them to a lower version to make them 5.2.x stable). Still it would be good if we could push out some of the new 5.3.x builds, maybe after they become beta, because otherwise we may end up with the same problem... (no testers, no feedback, no bugs, but still not working while I pursue for other things, thus never arriving at a really stable build...) Rainer From mrdemeanour at jackpot.uk.net Thu Nov 5 18:02:23 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Thu, 05 Nov 2009 17:02:23 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> Message-ID: <4AF3051F.5020501@jackpot.uk.net> Rainer Gerhards wrote: > Thanks, but I wasn't specific enough. For TLS, I also need to client config, > because I need two machines to reproduce any issues (these two instances are > also the challenge for the current testbench, what requires hopefully fewer > than I expect changes ;)). Sorry, Rainer. Anyway, I sent you the *current* config, i.e. using ptls. Here are the two configs using gnutls. But NOTE: I'm not using your default MySQL schema; you can't just drop this into your testlab. It should work if you ignore the custom MySQL template. I *really* doubt this has anything to do with MySQL - I've been using this MySQL setup for a year. Also note that there's nothing included from /etc/rsyslog.d - that directory is empty. These are the complete configs. There are a bunch of *Queue* directives in these files, both active and commented-out; I started playing around with queues to see if I could straighten it out that way, but it didn't work. That is, the problem should occur with a default queueing setup. ====== Start Server ========= # /etc/rsyslog.conf Configuration file for rsyslog v3. # # For more information see # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html # $DebugPrintTemplateList on # $ActionFileDefaultTemplate mysql-template ################# #### MODULES #### ################# $ModLoad imuxsock # provides support for local system logging # $ModLoad ommysql.so # provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 $ModLoad imklog # provides kernel logging support (previously done by rklogd) # provides TCP syslog reception $ModLoad imtcp # make gtls driver the default $DefaultNetstreamDriver gtls # $DefaultNetstreamDriver ptcp # certificate files $DefaultNetstreamDriverCAFile /etc/rsyslog.d/.ssl/gnu-ca-cert.pem $DefaultNetstreamDriverCertFile /etc/rsyslog.d/.ssl/saraha-rsyslog-cert.pem $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/.ssl/saraha-rsyslog-key.pem $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode $InputTCPServerStreamDriverAuthMode anon # client is NOT authenticated $InputTCPServerRun 10514 # start up listener at port 10514 $ModLoad MySQL ########################### #### GLOBAL DIRECTIVES #### ########################### # # Use default timestamp format. # To enable high precision timestamps, comment out the following line. # $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # # Set the default permissions for all log files. # $FileOwner root $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $WorkDirectory /var/log/rsyslog $template mysql-template, "insert into logs(host, facility, priority, level, tag, datetime, msg) values ('%source%', '%syslogfacility-text%', '%syslogpriority-text%', '%syslogseverity-text%', '%programname%', '%timereported:::date-mysql%', '%msg%')", sql, mysql # $template DEBUG,"Debug line with all properties:\nFROMHOST: '%FROMHOST%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nrawmsg: '%rawmsg%'\n\n" ############### #### RULES #### ############### #Discard some dross messages # authpriv.info ~ :HOSTNAME, isequal, "last" ~ #Discard router access messages for the script on Prajna that collects the router logs. if $msg contains 'User logged in on TELNET (192.168.1.2)' then ~ if $msg contains 'User logged out on TELNET (192.168.1.2)' then ~ # Log everything else to mysql. $ActionQueueType LinkedList # Number of elements... $ActionQueueSize 100 # $ActionQueueFileName mysql # $ActionQueueMaxDiskSpace 1M # $ActionQueueHighWaterMark 40 # $ActionQueueLowWaterMark 5 *.* >127.0.0.1,syslog,syslog,syslog;mysql-template $ActionExecOnlyWhenPreviousIsSuspended on & ~ $ActionExecOnlyWhenPreviousIsSuspended off #Log local stuff ONLY to /var/log/syslog :HOSTNAME, isequal, "prajna" -/var/log/syslog ====== End Server ========= ====== Start Client ========= # /etc/rsyslog.conf Configuration file for rsyslog v3. # # For more information see # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html ################# #### MODULES #### ################# $ModLoad imuxsock # provides support for local system logging $ModLoad imklog # provides kernel logging support (previously done by rklogd) #$ModLoad immark # provides --MARK-- message capability # provides UDP syslog reception #$ModLoad imudp #$UDPServerRun 514 # provides TCP syslog reception # $ModLoad imtcp # $InputTCPServerRun 514 # certificate files - just CA for a client $DefaultNetstreamDriverCAFile /etc/rsyslog.d/.ssl/gnu-ca-cert.pem # set up the action $DefaultNetstreamDriver gtls # use gtls netstream driver # $DefaultNetstreamDriver ptcp # use default netstream driver $ActionSendStreamDriverMode 1 # require TLS for the connection $ActionSendStreamDriverAuthMode anon # server is NOT authenticated ########################### #### GLOBAL DIRECTIVES #### ########################### # # Use traditional timestamp format. # To enable high precision timestamps, comment out the following line. # $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # # Set the default permissions for all log files. # $FileOwner root $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 # # Include all config files in /etc/rsyslog.d/ # $IncludeConfig /etc/rsyslog.d/*.conf ############### #### RULES #### ############### #Encrypted TCP log to database on Prajna *.* @@87.194.213.229:10514 # *.* -/var/log/syslog ====== End Client ========= > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour >> Sent: Thursday, November 05, 2009 5:41 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls >> >> Rainer Gerhards wrote: >>> Can you send me your rsyslog.conf, so that I can run it under the >> memory >>> debugger in my lab. I'll also take this as a motivation to finally >> add >>> multi-daemon tests to the testbench (what may take me a little >> while...). >> >> This is the server config (some of the remarks are misleading). >> >> # /etc/rsyslog.conf Configuration file for rsyslog v3. >> # >> # For more information see >> # /usr/share/doc/rsyslog- >> doc/html/rsyslog_conf.html >> >> # $DebugPrintTemplateList on >> # $ActionFileDefaultTemplate mysql-template >> ################# >> #### MODULES #### >> ################# >> >> $ModLoad imuxsock # provides support for local system logging >> >> # $ModLoad ommysql.so >> >> # provides UDP syslog reception >> $ModLoad imudp >> $UDPServerRun 514 >> $ModLoad imklog # provides kernel logging support (previously done by >> rklogd) >> >> # provides TCP syslog reception >> $ModLoad imtcp >> >> # make gtls driver the default >> # $DefaultNetstreamDriver gtls >> $DefaultNetstreamDriver ptcp >> >> # certificate files >> $DefaultNetstreamDriverCAFile /etc/rsyslog.d/.ssl/gnu-ca-cert.pem >> $DefaultNetstreamDriverCertFile /etc/rsyslog.d/.ssl/saraha-rsyslog- >> cert.pem >> $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/.ssl/saraha-rsyslog- >> key.pem >> >> # $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode >> # $InputTCPServerStreamDriverAuthMode anon # client is NOT >> authenticated >> $InputTCPServerRun 10514 # start up listener at port 10514 >> >> $ModLoad MySQL >> >> ########################### >> ########################### >> #### GLOBAL DIRECTIVES #### >> ########################### >> >> # >> # Use default timestamp format. >> # To enable high precision timestamps, comment out the following line. >> # >> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >> >> # >> # Set the default permissions for all log files. >> # >> $FileOwner root >> $FileGroup adm >> $FileCreateMode 0640 >> $DirCreateMode 0755 >> $WorkDirectory /var/log/rsyslog >> >> $template mysql-template, "insert into logs(host, facility, priority, >> level, tag, datetime, msg) values ('%source%', '%syslogfacility-text%', >> '%syslogpriority-text%', '%syslogseverity-text%', '%programname%', >> '%timereported:::date-mysql%', '%msg%')", sql, mysql >> >> # $template DEBUG,"Debug line with all properties:\nFROMHOST: >> '%FROMHOST%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag >> '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', >> PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', >> STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nrawmsg: >> '%rawmsg%'\n\n" >> >> ############### >> #### RULES #### >> ############### >> #Discard some dross messages >> # authpriv.info ~ >> :HOSTNAME, isequal, "last" ~ >> >> #Discard router access messages for the script on Prajna that collects >> the router logs. >> if $msg contains 'User logged in on TELNET (192.168.1.2)' then ~ >> if $msg contains 'User logged out on TELNET (192.168.1.2)' then ~ >> >> # Log everything else to mysql. >> $ActionQueueType LinkedList >> # Number of elements... >> $ActionQueueSize 100 >> # $ActionQueueFileName mysql >> # $ActionQueueMaxDiskSpace 1M >> # $ActionQueueHighWaterMark 40 >> # $ActionQueueLowWaterMark 5 >> >> >> *.* >> >127.0.0.1,syslog,syslog,syslog;mysql-template >> >> $ActionExecOnlyWhenPreviousIsSuspended on >> & ~ >> $ActionExecOnlyWhenPreviousIsSuspended off >> >> #Log local stuff ONLY to /var/log/syslog >> :HOSTNAME, isequal, "prajna" -/var/log/syslog >> >> >> -- >> Jack. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Thu Nov 5 23:22:04 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Thu, 5 Nov 2009 23:22:04 +0100 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710335F@GRFEXC.intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200911031158.02666.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA710335F@GRFEXC.intern.adiscon.com> Message-ID: <200911052322.04174.marc.schiffbauer@mightycare.de> Hello Rainer, I will give it a try and send you the feedback as soon as I am back from holiday. This will be in the beginning of december... Thanks -Marc Am Donnerstag, 5. November 2009 17:42:01 schrieb Rainer Gerhards: > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > > Sent: Tuesday, November 03, 2009 11:58 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > > Hello Rainer, > > > > is there some news about this issue? > > Unfortuantely not at the moment (I guess you see it is a busy time). Could > you give 5.3.4 a try? It would be very interesting (even for a v4-fix) to > see if the problem persists or not... > > Rainer > From rgerhards at hq.adiscon.com Fri Nov 6 11:54:50 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 6 Nov 2009 11:54:50 +0100 Subject: [rsyslog] message parser info - was: rsyslog 5.3.4 (devel) released References: <9B6E2A8877C38245BFB15CC491A11DA7103340@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710336D@GRFEXC.intern.adiscon.com> David, all, I hope I have answered all questions in this new document: http://www.rsyslog.com/doc-messageparser.html I would also appreciate feedback on that part: ====== Note that it is currently under evaluation if rsyslog will support binding parser chains to specific inputs directly, without depending on the ruleset. There are some concerns that this may not be necessary but adds considerable complexity to the configuration. So this may or may not be possible in the future. In any case, if we decide to add it, input modules need to support it, so this functionality would require some time to implement. ====== If I implement this, a listener would probably have a parser chain assigned, it not, the parser chain from the ruleset is used, if not, the parser chain from the default ruleset is used. I it (fairly...) trivial to add this capability, but I am really concerned that the config options get more and more complex... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, November 05, 2009 8:53 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 5.3.4 (devel) released > > yOn Thu, 5 Nov 2009, Tom Bergfeld wrote: > > > We have just released rsyslog 5.3.4, a member of the v5-devel branch. > > Today's release offers a number of important improvements. I would > like to > > highlight the ability to nest rulesets [1] (I already announced that) > and the > > ability to define custom parsers [2]. > > > > Let me elaborate a bit on the later. In the syslog world exists a > myriad of > > incompatible and invalidly formatted messages. We have had numerous > support > > requests on how to parse this or that malformed message format. With > custom > > parsers, we now have a vehicel to integrate all these invalid formats > in an > > elegant way that is also offering high performance ([2] has a > concrete > > sample). The core idea now is that parsers are plugins just like > input and > > output modules. It is relatively easy to write such a parser (roughly > a day, > > I expect) and custom parsers can be integrated into rsyslogd in a way > that > > they only affect messages received on specific port. So far, I have > not > > actually created a custom parser, but I hope that over time many of > them will > > become available. My hope here is that we actually can build a > directory, > > which other folks can browse so that they are able to find a solution > to the > > mess their vendor has provided ;) > > this sounds very interesting. however, reading the link I'm a little > confused. > > on the one hand it talks a lot about the priority of parsers, but then > it > talks about binding different parsers to different ports. It's not > clear > if these are two different ways to use parsers, or how these two would > interact. > > where can these parsers be used? > > obviously the imudp module can use them. can all input modules use > them? > > what are the limits of the parser? > > a couple of extreme examples: > > could you implement relp as a parser for imtcp, or since relp sends > replies that can't be done (limiting it to different ways of processing > a > message that's already been received) > > could a parser detect that it had a piece of a multi-line messsage and > stash the piece until it receives the rest of the message so that it > could > submit the complete message? > > > a coouple questions from trying to look through the code at almost 3am > local time > > if it can easily be done, may I suggest changing the santization flag > from > a yes/no boolean to an enum so that there can be more than one > sanitation > routine > > do you have the default parser broken out as a seperate file tha canbe > used as an example? > > David Lang > > > This release also introduces the capability to run rulesets off their > own > > queue (also already mentioned on the mailing list). Plus, it contains > the > > usual set of bug fixes. > > > > We plan to make this version the basis for the next v5-stable. > > > > > > [1] http://www.rsyslog.com/doc-omruleset.html > > [2] http://www.rsyslog.com/doc-rsconf1_rulesetparser.html > > > > > > > > ChangeLog: > > > > http://www.rsyslog.com/Article423.phtml > > > > Download: > > > > http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid- > 185.phtml > > > > As always, feedback is appreciated. > > > > Best regards, > > Tom Bergfeld > > -- > > Support > > ======= > > > > Improving rsyslog is costly, but you can help! We are looking for > > organizations that find rsyslog useful and wish to contribute back. > You can > > contribute by reporting bugs, improve the software, or donate money > or > > equipment. > > > > Commercial support contracts for rsyslog are available, and they help > finance > > continued maintenance. Adiscon GmbH, a privately held German > company, is > > currently funding rsyslog development. We are always looking for > interesting > > development projects. For details on how to help, please see > > http://www.rsyslog.com/doc-how2help.html . > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From correo at miguelangelnieto.net Fri Nov 6 12:31:04 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Fri, 6 Nov 2009 12:31:04 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: I made rpm packages with the stable v3 of rsyslog for suse 10.3. If anyone need them, tell me. > in other cases you are willing to loose logs rather than freezing the > machine and can configure rsyslog to accept messages, even when it can't > do anything with them to avoid this sort of lockup. How can I do to tell rsyslog to accept all messages and not use any queue? -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From rgerhards at hq.adiscon.com Fri Nov 6 14:27:20 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 6 Nov 2009 14:27:20 +0100 Subject: [rsyslog] potential server problems on rsyslog.com Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103373@GRFEXC.intern.adiscon.com> Hi all, I was just told that http://www.rsyslog.com, associated services and some other sites (like the knowledge base) will probably experience some temporary outage within the next couple of hours. Looks like the sysadmins need to fix something urgently. So if you got a connection problem, please retry a little bit later. Thanks, Rainer From jbondc at openmv.com Fri Nov 6 16:18:55 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Fri, 6 Nov 2009 10:18:55 -0500 Subject: [rsyslog] Postgresql module: standard_conforming_strings Message-ID: <002801ca5ef4$74e7cd20$5eb76760$@com> I have two questions: 1) How does rsyslog espace SQL commands in the plugin ompgsql.c? I've been staring at the code but I can't figure out where the backslash escaping happens. My problem is I've set: standard_conforming_strings=On This means that the backslash espace ' gfgfdg \' ' is ignored and causes errors. There are many ways to fix this. a) Have rsyslog issue (SET standard_conforming_strings = off;) for postgresql 8.2+ (quick fix) b) Change default sql espacing to use doubles quotes '' --- so 'test \' ' becomes 'test '' ' http://www.postgresql.org/docs/8.1/static/libpq-exec.html#LIBPQ-EXEC-ESCAPE- STRING 2) Is there an SVN repository of rsyslog? Or does anyone know of a good way or using GIT on windows (turtoisegit is currently buggy) ? From rgerhards at hq.adiscon.com Fri Nov 6 16:45:33 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 6 Nov 2009 16:45:33 +0100 Subject: [rsyslog] Postgresql module: standard_conforming_strings References: <002801ca5ef4$74e7cd20$5eb76760$@com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103377@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Jonathan Bond-Caron > Sent: Friday, November 06, 2009 4:19 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Postgresql module: standard_conforming_strings > > I have two questions: > > > > 1) How does rsyslog espace SQL commands in the plugin ompgsql.c? > > > > I've been staring at the code but I can't figure out where the > backslash > escaping happens. > > > > My problem is I've set: standard_conforming_strings=On > > This means that the backslash espace ' gfgfdg \' ' is ignored and > causes > errors. There are many ways to fix this. > > > > a) Have rsyslog issue (SET standard_conforming_strings = off;) for > postgresql 8.2+ (quick fix) > > b) Change default sql espacing to use doubles quotes '' --- so > 'test \' > ' becomes 'test '' ' > > http://www.postgresql.org/docs/8.1/static/libpq-exec.html#LIBPQ-EXEC- > ESCAPE- > STRING > This is controlled via the template. For postgre SQL, I think it needs the STDSQL option. I will check if ompgsql requests that option. $template name,"insert ...",STDSQL The string must be sanitized before it is passed down to the module, because the module does not know any longer what a field is. > > > 2) Is there an SVN repository of rsyslog? No > Or does anyone know of a > good > way or using GIT on windows (turtoisegit is currently buggy) ? > My co-workers use http://code.google.com/p/msysgit/ and don't complain (too much ;)) about it. HTH Rainer > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From epiphani at gmail.com Fri Nov 6 19:18:37 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Fri, 6 Nov 2009 13:18:37 -0500 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103338@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103338@GRFEXC.intern.adiscon.com> Message-ID: Wanted to point out that I've observed this now in a second environment - so we can eliminate the possibility that this is specific to that situation. I'll be working to reproduce the issue in a separate system next week, and hopefully I can get something reproducable so I can test the updated version. Thanks! -Aaron On Wed, Nov 4, 2009 at 10:06 AM, Rainer Gerhards wrote: > >> that can work, however XML output may not be too bad, people entering >> commands manually are probably not going to be asking for that much >> data. > > If I look at time constraints, I will probably not able to define a big CLI > with a real language. So I think you might send something like "queuestatus" > and it will reply with all status information on all queues it has. > >> the biggest problem I see with XML is the need to send requests and >> responses via e-mail for debugging. if one end uses a HTML enabled e- >> mail >> client it may not work well to paste XML text into it. > > That's a very good point and I really should begin to think about this right > now. Some kind of "debug package" that's easily mailable would not be a bad > thing ;) > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Nov 6 19:36:03 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 6 Nov 2009 19:36:03 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103338@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710337D@GRFEXC.intern.adiscon.com> excellent news. I have reviewed the debug log handling. It looks like we can use the current version, mabe just slightly modified, to generate debug output when a problem occurs by sending SIGUSR1 to it. I hope to be able to complete testing on monday. If it works out like I expect, I'll do a write-up of how it can be done. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Friday, November 06, 2009 7:19 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) / Problem in production > > Wanted to point out that I've observed this now in a second > environment - so we can eliminate the possibility that this is > specific to that situation. > > I'll be working to reproduce the issue in a separate system next week, > and hopefully I can get something reproducable so I can test the > updated version. > > Thanks! > -Aaron > > On Wed, Nov 4, 2009 at 10:06 AM, Rainer Gerhards > wrote: > > > >> that can work, however XML output may not be too bad, people > entering > >> commands manually are probably not going to be asking for that much > >> data. > > > > If I look at time constraints, I will probably not able to define a > big CLI > > with a real language. So I think you might send something like > "queuestatus" > > and it will reply with all status information on all queues it has. > > > >> the biggest problem I see with XML is the need to send requests and > >> responses via e-mail for debugging. if one end uses a HTML enabled > e- > >> mail > >> client it may not work well to paste XML text into it. > > > > That's a very good point and I really should begin to think about > this right > > now. Some kind of "debug package" that's easily mailable would not be > a bad > > thing ;) > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Nov 6 19:47:41 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 6 Nov 2009 10:47:41 -0800 (PST) Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: On Fri, 6 Nov 2009, Miguel Angel Nieto wrote: >> in other cases you are willing to loose logs rather than freezing the >> machine and can configure rsyslog to accept messages, even when it can't >> do anything with them to avoid this sort of lockup. > > How can I do to tell rsyslog to accept all messages and not use any queue? you cannot tell rsyslog to not use a queue. the fundamental architecture of rsyslog is that the input modules receive messages, parse minimal info out of them, and put them in a queue.worker threads then pull messages from this queue and send them to their destination. this queue can be in ram, ram + disk file, or disk-only classic syslog doesn't use any queue and must fully process each message before receiving the next message. syslog-ng has the ability to delay writing the logs to disk a bit to do them in batches, but otherwise is the same. in rsyslog, significantly more time is spent in the output side of things than in the input side. historicly, syslog was intended to be a reliable logging mechanism, so applications do not complete their write until the syslog daemon has the log safe on disk. the performance of doing this is so horrible that everybody has some way of relaxing this requirement (the - option in sysklogd, the batch option in syslog-ng, the memory queue in rsyslog), but all of these options only allow a finite number of items to be buffered before the syslog daemon will stop to wait for them to really get to the destination. this is why your systems locked up, they were trying to write to rsyslog on the client, rsyslog on the client was configured to not consider the logs processed until they were acknowledged by the TCP stack of the server. so when the server is not available the clients keep accepting messages and queue them up, but when the queue filled up they stopped accepting new messages. the reason to use TCP for syslog instead of UDP is so that you can have the log server push back on the sender and tell the sender that it can't process the log message right now, the sender should hang on to it and tray again to send it later. if you are really willing to loose logs if the log server is down or backed up, why not just use UDP for the logs? look at the MainMsgQueue* options for setting high watermark, low watermark, discard options, etc if you want to change how rsyslog acts when the queue fills up. I haven't used these, so all I can do is to point you in the right direction. David Lang From mrdemeanour at jackpot.uk.net Fri Nov 6 20:10:15 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 06 Nov 2009 19:10:15 +0000 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: <4AF47497.4070407@jackpot.uk.net> david at lang.hm wrote: > On Fri, 6 Nov 2009, Miguel Angel Nieto wrote: > >>> in other cases you are willing to loose logs rather than freezing >>> the machine and can configure rsyslog to accept messages, even >>> when it can't do anything with them to avoid this sort of lockup. >>> >> How can I do to tell rsyslog to accept all messages and not use any >> queue? > > you cannot tell rsyslog to not use a queue. Is that really true? "Direct queues are non-queuing queues. A queue in direct mode does neither queue nor buffer any of the queue elements but rather passes the element directly (and immediately) from the producer to the consumer. This sounds strange, but there is a good reason for this queue type." [...] "To create a direct queue, use the "$QueueType Direct" config directive." e.g. (I suppose): $MainQueueType Direct http://www.rsyslog.com/doc-queues.html -- Jack. From ryan.b.lynch at gmail.com Fri Nov 6 22:02:40 2009 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Fri, 6 Nov 2009 16:02:40 -0500 Subject: [rsyslog] What backslash-escape sequences are supported in templates? Message-ID: <115906d10911061302h56d1b56cs50247fd010de57a5@mail.gmail.com> The man page gives '\7' (bell) as an example (in the section 'Templates'), and then it says "The set in rsyslog is a bit restricted currently." I couldn't find any more info in the online documentation, in either of the 'Templates' or 'Property Replacer' sections. I just tried to use '\t' (tab) in a log format template, though, and it didn't work--it prints as just 't'. Is there a doc that lists what escapes are and aren't accepted? If not, can somebody who knows the answer enlighten me? Ryan B. Lynch ryan.b.lynch at gmail.com From philr at jaspers.co.nz Sat Nov 7 21:37:59 2009 From: philr at jaspers.co.nz (Phil Reilly) Date: Sun, 08 Nov 2009 09:37:59 +1300 Subject: [rsyslog] Alerting rules via a database Message-ID: <4AF5DAA7.5010001@jaspers.co.nz> I attempting to allow for flexible rule matches on Syslogs from a web front end (rather than entires into the rsyslog config files) I want to get regexp filters from a db to alert upon messages. Not sure the best way to achieve this. I've so far though of. * Outputting to a pipe and runing it via an alerting script. * Having file watch the messages. * Recieving the messages then passing them to rsyslog (yuck) Can the rule engine allow for match rules outside of the config? is there an elegant way of doing this? From david at lang.hm Sun Nov 8 08:44:36 2009 From: david at lang.hm (david at lang.hm) Date: Sat, 7 Nov 2009 23:44:36 -0800 (PST) Subject: [rsyslog] Alerting rules via a database In-Reply-To: <4AF5DAA7.5010001@jaspers.co.nz> References: <4AF5DAA7.5010001@jaspers.co.nz> Message-ID: On Sun, 8 Nov 2009, Phil Reilly wrote: > I attempting to allow for flexible rule matches on Syslogs from a web > front end (rather than entires into the rsyslog config files) > > I want to get regexp filters from a db to alert upon messages. Not sure > the best way to achieve this. I've so far though of. > > * Outputting to a pipe and runing it via an alerting script. > * Having file watch the messages. > * Recieving the messages then passing them to rsyslog (yuck) > > Can the rule engine allow for match rules outside of the config? is > there an elegant way of doing this? rsyslog doesn't give you this ability, but it's not really the best approach to use for alerting anyway. what are you trying to achieve by having the alert definitions in a database? there are several tools out there to do alerting (SEC, Simple Event Correlator) is one of the leading ones, but I'm not aware of any of them that use a database for their rulesets. I'm also scratching my head trying to figure out what the advantage of doing so would be. David Lang From philr at jaspers.co.nz Sun Nov 8 09:29:30 2009 From: philr at jaspers.co.nz (Phil Reilly) Date: Sun, 08 Nov 2009 21:29:30 +1300 Subject: [rsyslog] Alerting rules via a database In-Reply-To: References: <4AF5DAA7.5010001@jaspers.co.nz> Message-ID: <4AF6816A.6050901@jaspers.co.nz> david at lang.hm wrote: > On Sun, 8 Nov 2009, Phil Reilly wrote: > > >> I attempting to allow for flexible rule matches on Syslogs from a web >> front end (rather than entires into the rsyslog config files) >> >> I want to get regexp filters from a db to alert upon messages. Not sure >> the best way to achieve this. I've so far though of. >> >> * Outputting to a pipe and runing it via an alerting script. >> * Having file watch the messages. >> * Recieving the messages then passing them to rsyslog (yuck) >> >> Can the rule engine allow for match rules outside of the config? is >> there an elegant way of doing this? >> > > rsyslog doesn't give you this ability, but it's not really the best > approach to use for alerting anyway. > > what are you trying to achieve by having the alert definitions in a > database? there are several tools out there to do alerting (SEC, Simple > Event Correlator) is one of the leading ones, but I'm not aware of any of > them that use a database for their rulesets. > > I'm also scratching my head trying to figure out what the advantage of > doing so would be. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > Thanks David, We have a networked environment. We also have a web page that allows you to configure regexp to match certain syslog messages. These patterns are compiled and kept in a table. The current syslog process we use listens for udp. When it gets a syslog message, we examine the patterns (which are re-read upon addition or change) and pass them to an alertering process before writing the logs to disk. The existing system works well, but we now want to scale it over a few machines and I'm examining what syslog products out there cater for alerting. So a database will make configuring alerts far more dynamic than statically entering them into a config file. It will also allow for grouped views so different groups have the ability to have custom alerts based upon their own interpretation of syslog messages. From rgerhards at hq.adiscon.com Sun Nov 8 11:48:19 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 8 Nov 2009 11:48:19 +0100 Subject: [rsyslog] Alerting rules via a database References: <4AF5DAA7.5010001@jaspers.co.nz> <4AF6816A.6050901@jaspers.co.nz> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> So what you are actually looking for is a system that can work with dynamically changable alert definitions? As David said, there is no such thing currently, but the best road to approach is is to write a custom output plugin, that you pass each message to. That plugin can even decide if messages should be discarded and not further processed. I envisioned such a plugin, but had not yet time to write, for a similar use case. If you intend to write one AND contribute it to the project, I can help you get started with the interface, would even be willing to create you a custom skeleton that you can fill in your logic ;) HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phil Reilly > Sent: Sunday, November 08, 2009 9:30 AM > To: rsyslog-users > Subject: Re: [rsyslog] Alerting rules via a database > > david at lang.hm wrote: > > On Sun, 8 Nov 2009, Phil Reilly wrote: > > > > > >> I attempting to allow for flexible rule matches on Syslogs > from a web > >> front end (rather than entires into the rsyslog config files) > >> > >> I want to get regexp filters from a db to alert upon > messages. Not sure > >> the best way to achieve this. I've so far though of. > >> > >> * Outputting to a pipe and runing it via an alerting script. > >> * Having file watch the messages. > >> * Recieving the messages then passing them to rsyslog (yuck) > >> > >> Can the rule engine allow for match rules outside of the config? is > >> there an elegant way of doing this? > >> > > > > rsyslog doesn't give you this ability, but it's not really the best > > approach to use for alerting anyway. > > > > what are you trying to achieve by having the alert definitions in a > > database? there are several tools out there to do alerting > (SEC, Simple > > Event Correlator) is one of the leading ones, but I'm not > aware of any of > > them that use a database for their rulesets. > > > > I'm also scratching my head trying to figure out what the > advantage of > > doing so would be. > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > Thanks David, > > We have a networked environment. We also have a web page that > allows you > to configure regexp to match certain syslog messages. These > patterns are > compiled and kept in a table. The current syslog process we > use listens > for udp. When it gets a syslog message, we examine the > patterns (which > are re-read upon addition or change) and pass them to an alertering > process before writing the logs to disk. The existing system > works well, > but we now want to scale it over a few machines and I'm > examining what > syslog products out there cater for alerting. > > So a database will make configuring alerts far more dynamic than > statically entering them into a config file. It will also allow for > grouped views so different groups have the ability to have > custom alerts > based upon their own interpretation of syslog messages. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Sun Nov 8 16:40:50 2009 From: david at lang.hm (david at lang.hm) Date: Sun, 8 Nov 2009 07:40:50 -0800 (PST) Subject: [rsyslog] Alerting rules via a database In-Reply-To: <4AF6816A.6050901@jaspers.co.nz> References: <4AF5DAA7.5010001@jaspers.co.nz> <4AF6816A.6050901@jaspers.co.nz> Message-ID: On Sun, 8 Nov 2009, Phil Reilly wrote: > > david at lang.hm wrote: >> On Sun, 8 Nov 2009, Phil Reilly wrote: >> >> >>> I attempting to allow for flexible rule matches on Syslogs from a web >>> front end (rather than entires into the rsyslog config files) >>> >>> I want to get regexp filters from a db to alert upon messages. Not sure >>> the best way to achieve this. I've so far though of. >>> >>> * Outputting to a pipe and runing it via an alerting script. >>> * Having file watch the messages. >>> * Recieving the messages then passing them to rsyslog (yuck) >>> >>> Can the rule engine allow for match rules outside of the config? is >>> there an elegant way of doing this? >>> >> >> rsyslog doesn't give you this ability, but it's not really the best >> approach to use for alerting anyway. >> >> what are you trying to achieve by having the alert definitions in a >> database? there are several tools out there to do alerting (SEC, Simple >> Event Correlator) is one of the leading ones, but I'm not aware of any of >> them that use a database for their rulesets. >> >> I'm also scratching my head trying to figure out what the advantage of >> doing so would be. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > Thanks David, > > We have a networked environment. We also have a web page that allows you > to configure regexp to match certain syslog messages. These patterns are > compiled and kept in a table. The current syslog process we use listens > for udp. When it gets a syslog message, we examine the patterns (which > are re-read upon addition or change) and pass them to an alertering > process before writing the logs to disk. The existing system works well, > but we now want to scale it over a few machines and I'm examining what > syslog products out there cater for alerting. > > So a database will make configuring alerts far more dynamic than > statically entering them into a config file. It will also allow for > grouped views so different groups have the ability to have custom alerts > based upon their own interpretation of syslog messages. I don't know anything that will read a database like you are lookng for. I think you would be better off having your web gui create SEC rules or something like that (you can still store the basic info in a database) David Lang From jbondc at openmv.com Sun Nov 8 17:42:34 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Sun, 8 Nov 2009 11:42:34 -0500 Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing Message-ID: <003001ca6092$792833d0$6b789b70$@com> Hi all, Attached is a patch for 3 things: a) Check that TCP connection is still alive when using TLS b) Improve TAG parsing so that it ends when it sees a tab or escape control character. c) Added configuration directive: escapecontrolcharactertab In rsyslog.conf, you can set: $escapeControlCharactersOnReceive on $escapeControlCharacterTab off This avoids having a tab being escaped since it is after a viewable character J I'd prefer this being the default behavior but I've left $escapeControlCharacterTab on as default in case people expect tabs to be escaped. This is my first GIT patch, hope it works well ;) From mrdemeanour at jackpot.uk.net Sun Nov 8 18:23:31 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Sun, 08 Nov 2009 17:23:31 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> Message-ID: <4AF6FE93.6080805@jackpot.uk.net> Rainer Gerhards wrote: > Thanks, but I wasn't specific enough. For TLS, I also need to client > config, because I need two machines to reproduce any issues (these > two instances are also the challenge for the current testbench, what > requires hopefully fewer than I expect changes ;)). Perhaps someone could advise me on an alternative to resolving this problem by means of bug-hunting. While 4.4.2 is the version of rsyslog currently shipping in Debian squeeze, it's obviously a bit out-of-date from the point of view of the developers. Well, the reason I'm using that version is precisely that it ships with Debian. Or to be more precise, I upgraded to squeeze because it ships with a version of rsyslog that includes encryption support. So perhaps I should just cut free, and install 5.x stable? -- Jack. From jbondc at openmv.com Sun Nov 8 20:18:43 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Sun, 8 Nov 2009 14:18:43 -0500 Subject: [rsyslog] Postgresql module: standard_conforming_strings In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103377@GRFEXC.intern.adiscon.com> References: <002801ca5ef4$74e7cd20$5eb76760$@com> <9B6E2A8877C38245BFB15CC491A11DA7103377@GRFEXC.intern.adiscon.com> Message-ID: <000601ca60a8$49f44d40$dddce7c0$@com> On Fri Nov 6 10:45 AM, Rainer Gerhards wrote: > > > > > > My problem is I've set: standard_conforming_strings=On > > > > This means that the backslash espace ' gfgfdg \' ' is ignored and > > causes errors. There are many ways to fix this. > > > > This is controlled via the template. For postgre SQL, I think it needs > the STDSQL option. I will check if ompgsql requests that option. > > $template name,"insert ...",STDSQL > Thanks that fixed it, I was using $template name,"insert ...",SQL From philr at jaspers.co.nz Sun Nov 8 20:36:07 2009 From: philr at jaspers.co.nz (Phil Reilly) Date: Mon, 09 Nov 2009 08:36:07 +1300 Subject: [rsyslog] Alerting rules via a database In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> References: <4AF5DAA7.5010001@jaspers.co.nz> <4AF6816A.6050901@jaspers.co.nz> <9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> Message-ID: <4AF71DA7.2090107@jaspers.co.nz> More than happy to Rainer. A Skeleton will be welcome. Rainer Gerhards wrote: > So what you are actually looking for is a system that can work with > dynamically changable alert definitions? As David said, there is no such > thing currently, but the best road to approach is is to write a custom output > plugin, that you pass each message to. That plugin can even decide if > messages should be discarded and not further processed. I envisioned such a > plugin, but had not yet time to write, for a similar use case. > > If you intend to write one AND contribute it to the project, I can help you > get started with the interface, would even be willing to create you a custom > skeleton that you can fill in your logic ;) > > HTH > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phil Reilly >> Sent: Sunday, November 08, 2009 9:30 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Alerting rules via a database >> >> david at lang.hm wrote: >> >>> On Sun, 8 Nov 2009, Phil Reilly wrote: >>> >>> >>> >>>> I attempting to allow for flexible rule matches on Syslogs >>>> >> from a web >> >>>> front end (rather than entires into the rsyslog config files) >>>> >>>> I want to get regexp filters from a db to alert upon >>>> >> messages. Not sure >> >>>> the best way to achieve this. I've so far though of. >>>> >>>> * Outputting to a pipe and runing it via an alerting script. >>>> * Having file watch the messages. >>>> * Recieving the messages then passing them to rsyslog (yuck) >>>> >>>> Can the rule engine allow for match rules outside of the config? is >>>> there an elegant way of doing this? >>>> >>>> >>> rsyslog doesn't give you this ability, but it's not really the best >>> approach to use for alerting anyway. >>> >>> what are you trying to achieve by having the alert definitions in a >>> database? there are several tools out there to do alerting >>> >> (SEC, Simple >> >>> Event Correlator) is one of the leading ones, but I'm not >>> >> aware of any of >> >>> them that use a database for their rulesets. >>> >>> I'm also scratching my head trying to figure out what the >>> >> advantage of >> >>> doing so would be. >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> Thanks David, >> >> We have a networked environment. We also have a web page that >> allows you >> to configure regexp to match certain syslog messages. These >> patterns are >> compiled and kept in a table. The current syslog process we >> use listens >> for udp. When it gets a syslog message, we examine the >> patterns (which >> are re-read upon addition or change) and pass them to an alertering >> process before writing the logs to disk. The existing system >> works well, >> but we now want to scale it over a few machines and I'm >> examining what >> syslog products out there cater for alerting. >> >> So a database will make configuring alerts far more dynamic than >> statically entering them into a config file. It will also allow for >> grouped views so different groups have the ability to have >> custom alerts >> based upon their own interpretation of syslog messages. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Mon Nov 9 05:48:27 2009 From: david at lang.hm (david at lang.hm) Date: Sun, 8 Nov 2009 20:48:27 -0800 (PST) Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <4AF6FE93.6080805@jackpot.uk.net> References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net> Message-ID: On Sun, 8 Nov 2009, Mr. Demeanour wrote: > Rainer Gerhards wrote: >> Thanks, but I wasn't specific enough. For TLS, I also need to client >> config, because I need two machines to reproduce any issues (these >> two instances are also the challenge for the current testbench, what >> requires hopefully fewer than I expect changes ;)). > > Perhaps someone could advise me on an alternative to resolving this > problem by means of bug-hunting. While 4.4.2 is the version of rsyslog > currently shipping in Debian squeeze, it's obviously a bit out-of-date > from the point of view of the developers. the rate of change in rsyslog over the last year is dizzying. This makes rsyslog a much better progam, but it also means that it's very unlikly that the debian maintainers will be able to backport all the fixes. > Well, the reason I'm using that version is precisely that it ships with > Debian. Or to be more precise, I upgraded to squeeze because it ships > with a version of rsyslog that includes encryption support. So perhaps I > should just cut free, and install 5.x stable? unfortunantly the current 5.2 stable release has some known problems. the 5.x development is looking pretty good right now, but it is very much the bleeding edge of things. if you need a newer stable version _now_ go with the latest 4.x if you can affort to test the 5.x development branch, it will probably work (I've found it stable over the last couple of weeks, running at loads of ~10,000 logs/sec), but as always you may run into unexpected problems (like you did with 4.4.2). I don't use encryption so I can't vouch for that part of things. David Lang From rgerhards at hq.adiscon.com Mon Nov 9 06:38:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 9 Nov 2009 06:38:29 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls Message-ID: <000501ca60fe$fac9ae3a$100013ac@intern.adiscon.com> More info follows, but v5 has exactly the same tls subsystem as v4. So we need to track that bug down. ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 09.11.09 05:49 Betreff: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls On Sun, 8 Nov 2009, Mr. Demeanour wrote: > Rainer Gerhards wrote: >> Thanks, but I wasn't specific enough. For TLS, I also need to client >> config, because I need two machines to reproduce any issues (these >> two instances are also the challenge for the current testbench, what >> requires hopefully fewer than I expect changes ;)). > > Perhaps someone could advise me on an alternative to resolving this > problem by means of bug-hunting. While 4.4.2 is the version of rsyslog > currently shipping in Debian squeeze, it's obviously a bit out-of-date > from the point of view of the developers. the rate of change in rsyslog over the last year is dizzying. This makes rsyslog a much better progam, but it also means that it's very unlikly that the debian maintainers will be able to backport all the fixes. > Well, the reason I'm using that version is precisely that it ships with > Debian. Or to be more precise, I upgraded to squeeze because it ships > with a version of rsyslog that includes encryption support. So perhaps I > should just cut free, and install 5.x stable? unfortunantly the current 5.2 stable release has some known problems. the 5.x development is looking pretty good right now, but it is very much the bleeding edge of things. if you need a newer stable version _now_ go with the latest 4.x if you can affort to test the 5.x development branch, it will probably work (I've found it stable over the last couple of weeks, running at loads of ~10,000 logs/sec), but as always you may run into unexpected problems (like you did with 4.4.2). I don't use encryption so I can't vouch for that part of things. David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Nov 9 10:02:40 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 9 Nov 2009 10:02:40 +0100 Subject: [rsyslog] rsyslog priorities - was: RE: Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710338D@GRFEXC.intern.adiscon.com> >From my POV the real issue is that there is very much going on at the moment (I don't want to complain about that). But that also means I need to prioritize the things I do. While I would like to have bugfixing always on top of the list, it is plain truth that there are also some other factors (like how to obtain bread for breakfast ;)), so priorities are also dictated by commercial activities that need to run alongside. As a reminder, rsyslog development is still very largely depending on Adiscon funding and that also means things like EventReporter actually pay for it (some people may think well about that fact and its implications ;)). If I worked on rsyslog just as a byline hobby activity, we were nowhere close to where we are right now. But, as I said: the bottom line is that priorities come with a twist of "as time permits". Anyhow, I am quite concerned about this bug and hope to be able to look at it as soon as possible. As another side-node running development versions and providing good bug reports is also an excellent way to contribute to the project... If you followed the mailing list, my priority scheme also slightly involves that bug reports/feature whishes from those that contribute frequently have a somewhat higher priority than from those who do not. I find this is only fair. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Sunday, November 08, 2009 6:24 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Rainer Gerhards wrote: > > Thanks, but I wasn't specific enough. For TLS, I also need to client > > config, because I need two machines to reproduce any issues (these > > two instances are also the challenge for the current testbench, what > > requires hopefully fewer than I expect changes ;)). > > Perhaps someone could advise me on an alternative to resolving this > problem by means of bug-hunting. While 4.4.2 is the version of rsyslog > currently shipping in Debian squeeze, it's obviously a bit out-of-date > from the point of view of the developers. > > Well, the reason I'm using that version is precisely that it ships with > Debian. Or to be more precise, I upgraded to squeeze because it ships > with a version of rsyslog that includes encryption support. So perhaps > I > should just cut free, and install 5.x stable? > > -- > Jack. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mrdemeanour at jackpot.uk.net Mon Nov 9 10:13:56 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Mon, 09 Nov 2009 09:13:56 +0000 Subject: [rsyslog] rsyslog priorities - was: RE: Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710338D@GRFEXC.intern.adiscon.com> References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA710338D@GRFEXC.intern.adiscon.com> Message-ID: <4AF7DD54.6080001@jackpot.uk.net> Rainer Gerhards wrote: > While I would like to have bugfixing always on top of the list, it is > plain truth that there are also some other factors (like how to > obtain bread for breakfast ;)), so priorities are also dictated by > commercial activities that need to run alongside. I wouldn't dream of questioning your priorities. I don't know how you manage to do so much development, while at the same time attending to this list, participating in general syslog standards-related actvities and so on. Frankly, I'm rather surprised to learn that you actually have time for breakfast! I think I shall try 5.2 on the server; I'll let you know if the trouble recurs. -- Jack. From martinmie at PartyGaming.com Mon Nov 9 10:15:13 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Mon, 9 Nov 2009 10:15:13 +0100 Subject: [rsyslog] rsyslog 5.2.0 is a zombie Message-ID: <0B1B9163138571439EAF804646F3F6AA08A58AA7@GIBSVWIN004X.partygaming.local> Hi, Just for the records... uur rsyslog servers, both running on the stable 5.2.0, died over the weekend. Well, to be more precise, they became zombies: # ps -efl | grep rsyslog 5 Z root 2565 1 73 77 0 - 0 exit Nov05 ? 2-20:44:14 [rsyslogd] ...so we sent from an "uninterruptible sleep" on 4.2.0 to a more "eternal sleep" LOL. I think it's time to revert to the latest v4 branch... suggestions? Cheers, Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From rgerhards at hq.adiscon.com Mon Nov 9 11:35:26 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 9 Nov 2009 11:35:26 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory References: <4AF47497.4070407@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103392@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Friday, November 06, 2009 8:10 PM > To: rsyslog-users > Subject: Re: [rsyslog] computer hang-up and WorkDirectory > > david at lang.hm wrote: > > On Fri, 6 Nov 2009, Miguel Angel Nieto wrote: > > > >>> in other cases you are willing to loose logs rather than freezing > >>> the machine and can configure rsyslog to accept messages, even > >>> when it can't do anything with them to avoid this sort of lockup. > >>> > >> How can I do to tell rsyslog to accept all messages and not use any > >> queue? > > > > you cannot tell rsyslog to not use a queue. > > Is that really true? > > "Direct queues are non-queuing queues. A queue in direct mode does > neither queue nor buffer any of the queue elements but rather passes > the > element directly (and immediately) from the producer to the consumer. > This sounds strange, but there is a good reason for this queue type." > > [...] > > "To create a direct queue, use the "$QueueType Direct" config > directive." > > e.g. (I suppose): > $MainQueueType Direct > > > http://www.rsyslog.com/doc-queues.html Sorry, I overlooked that message first. You are right. While rsyslog needs queues (more precisely: queue objects), one can configure queue objects not to queue. This is the default case for actions. However, you most often do not want to set the main queue to direct. But you can and there are use cases for it. The bottom line than is that sender and action processing are tightly couple, so with inputs like UDP you will most probably lose a lot of messages, especially when using slow backends like databases. One cure is to run those slow backends then on async action queues. HTH Rainer From rgerhards at hq.adiscon.com Mon Nov 9 12:24:04 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 9 Nov 2009 12:24:04 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103395@GRFEXC.intern.adiscon.com> Jack, Quick update: I was able to run a (relatively quick) test on a configuration that hopefully later becomes part of the testbench (I am working towards that goal and thought doing a manual check at that stage would not hurt). It is not based your config, and it is relatively simple and straightforward. Still, it uses TLS, and uses it in anon mode like you did. I used 4.4.2 on the server. I processed around 500,000 message (not kept track of the actual number). I did three such runs, all runing under the excellent valgrind[1] memory debugger. For none of the runs, valgrind reported any memory leaks. While this may not be an ultimate indication, valgrind is *very* effective in finding leaks and based on what you wrote I would have expected a small chunk of memory to be lost per message. So the bottom line is that I currently cannot reproduce the bug. This may change when I finally import your config. However, it would be useful for me if you could run valgrind on rsyslogd in your environment and let me know if valgrind reports any memory leaks. Doing so considerably slows down rsyslogd, but given your load, I'd expect that it would be acceptable. To run under valgrind is relatively simply. Valgrind is available as a package on almost all distros. All you need to do is run valgrind and specify your usual rsyslogd command line as the parameter. It is recommended to do this in the foreground (see rsyslog troubleshooting doc). So, for example, if you start rsyslogd usually by /sbin/rsyslogd -c4 -... you simply do valgrind /sbin/rsyslogd -c4 -... instead. By default, valgrind message go to stdout. At the end of the run (kill `rsyslog.pid`) valgrind prints out memory leaks it detects. During the run, it prints out any anomalies it sees. Would be great if you could give that a try. Rainer [1] http://www.valgrind.org > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Sunday, November 08, 2009 6:24 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Rainer Gerhards wrote: > > Thanks, but I wasn't specific enough. For TLS, I also need to client > > config, because I need two machines to reproduce any issues (these > > two instances are also the challenge for the current testbench, what > > requires hopefully fewer than I expect changes ;)). > > Perhaps someone could advise me on an alternative to resolving this > problem by means of bug-hunting. While 4.4.2 is the version of rsyslog > currently shipping in Debian squeeze, it's obviously a bit out-of-date > from the point of view of the developers. > > Well, the reason I'm using that version is precisely that it ships with > Debian. Or to be more precise, I upgraded to squeeze because it ships > with a version of rsyslog that includes encryption support. So perhaps > I > should just cut free, and install 5.x stable? > > -- > Jack. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mrdemeanour at jackpot.uk.net Mon Nov 9 17:24:45 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Mon, 09 Nov 2009 16:24:45 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103395@GRFEXC.intern.adiscon.com> References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA7103395@GRFEXC.intern.adiscon.com> Message-ID: <4AF8424D.30306@jackpot.uk.net> Rainer Gerhards wrote: > Jack, > > Quick update: I was able to run a (relatively quick) test on a > configuration that hopefully later becomes part of the testbench (I > am working towards that goal and thought doing a manual check at that > stage would not hurt). It is not based your config, and it is > relatively simple and straightforward. Still, it uses TLS, and uses > it in anon mode like you did. I used 4.4.2 on the server. I processed > around 500,000 message (not kept track of the actual number). I did > three such runs, all runing under the excellent valgrind[1] memory > debugger. > > For none of the runs, valgrind reported any memory leaks. While this > may not be an ultimate indication, valgrind is *very* effective in > finding leaks and based on what you wrote I would have expected a > small chunk of memory to be lost per message. OK (thanks). I have formed the impression that the problems are occurring after some period of running time; free memory decreases quite slowly, until some unknown event causes a rather rapid degeneration. That is: I suspect that any leak is not likely to be observable on a per-message basis, until this unknown event has occured. > > So the bottom line is that I currently cannot reproduce the bug. This > may change when I finally import your config. However, it would be > useful for me if you could run valgrind on rsyslogd in your > environment and let me know if valgrind reports any memory leaks. > Doing so considerably slows down rsyslogd, but given your load, I'd > expect that it would be acceptable. > > To run under valgrind is relatively simply. Valgrind is available as > a package on almost all distros. All you need to do is run valgrind > and specify your usual rsyslogd command line as the parameter. It is > recommended to do this in the foreground (see rsyslog troubleshooting > doc). > > So, for example, if you start rsyslogd usually by > > /sbin/rsyslogd -c4 -... So "valgrind /usr/sbin/rsyslogd -c4" results in rsyslogd backgrounding promptly, at which point valgrind prints its report (which shows no leaks - unsurprisingly). "valgrind /usr/sbin/rsyslogd -c4 -n" results in a hang. CTRL-C fails to kill the foreground task. "kill -9 " kills the task, but no valgrind report is produced. The same command without valgrind also results in a hung foreground task. If run under valgrind, memcheck-x86-li goes to 99% CPU on CTRL-C. I currently suspect problems with mySQL as the origin of these problems. I was this morning getting messages of the form "Lost connection to MySQL server at 'reading authorization packet'". I was also observing MySQL aborted clients and connects. I have increased the MySQL connect timeout, and can no longer reproduce these reports. For now, I assume that problem is fixed, but I can't yet say if the rsyslog hangs have stopped. I wonder if what was happening was that MySQL was "going away" in some sense, and that rsyslog was not reconnecting to it successfully, *and* not retrying? I noticed that although the ActionQueue for the mysql output module is not disk-assisted, the debug log records: action 4 queue: save on shutdown 1, max disk space allowed 0 So I've set $ActionQueueSaveOnShutdown off. However this hasn't changed the hang behaviour with -n. Since I can't get rsyslog to run in the foreground under valgrind, I am now running daemonized without valgrind (but with encryption); perhaps these changes have fixed the problem. I should know by late this evening - when the problem is observed, the server never lives for more than a few hours. -- Jack. From rgerhards at hq.adiscon.com Tue Nov 10 09:16:32 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 09:16:32 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls (valgrind log) References: <4AF8677B.40808@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033A3@GRFEXC.intern.adiscon.com> Thanks for the log file. Unfortunately, it is not a full run. I gues I needed to make some more things clear. The command line you need to use here $ valgrind /usr/sbin/rsyslogd -c4 -d >rsyslog.log 2>&1 Is the same that you usually use, except that -n must be specified. I unfortunately do no know what you (or better: your distro) usually uses, but you may see it in a process list. Before you start it that way, you must stop the regular syslogd and you must be sure to su to root. The -d switch, while generally useful, takes a lot of time and will generate an immense log. So it is best to not specify it for now. What I am after are valgrind violations (these are printed during the run) and the memory stats at the end of the run. The later will show any memory leaks. The log I received did not receive any message via TLS but two via UDP. It looks like a very short run. Thanks again for your help, I hope I have now explained better ;) Rainer > -----Original Message----- > From: Mr. Demeanour [mailto:mrdemeanour at jackpot.uk.net] > Sent: Monday, November 09, 2009 8:03 PM > To: Rainer Gerhards > Subject: Re: Rsyslog 4.4.2: server out-of-memory with gnutls > (valgrind log) > > Hi, > > I finally got a decent valgrind log using this command: > $ valgrind /usr/sbin/rsyslogd -c4 -d >rsyslog.log 2>&1 > > The log is attached. Free memory was declining steadily > throughout this run. > > Thanks, > -- > Jack. > From rgerhards at hq.adiscon.com Tue Nov 10 09:44:25 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 09:44:25 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103395@GRFEXC.intern.adiscon.com> <4AF8424D.30306@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033A8@GRFEXC.intern.adiscon.com> And finally a reply to "the meat" ;) > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Monday, November 09, 2009 5:25 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Rainer Gerhards wrote: > > Jack, > > > > Quick update: I was able to run a (relatively quick) test on a > > configuration that hopefully later becomes part of the testbench (I > > am working towards that goal and thought doing a manual > check at that > > stage would not hurt). It is not based your config, and it is > > relatively simple and straightforward. Still, it uses TLS, and uses > > it in anon mode like you did. I used 4.4.2 on the server. I > processed > > around 500,000 message (not kept track of the actual number). I did > > three such runs, all runing under the excellent valgrind[1] memory > > debugger. > > > > For none of the runs, valgrind reported any memory leaks. While this > > may not be an ultimate indication, valgrind is *very* effective in > > finding leaks and based on what you wrote I would have expected a > > small chunk of memory to be lost per message. > > OK (thanks). > > I have formed the impression that the problems are occurring > after some > period of running time; free memory decreases quite slowly, until some > unknown event causes a rather rapid degeneration. That is: I suspect > that any leak is not likely to be observable on a per-message basis, > until this unknown event has occured. This would explain what we currently see. It seems to be TLS-related, as you said. So it is unlikely to be a message at all, but rather a TLS event that causes the mem leak. With that said, I should probbably see if a connection abort can trigger one... > > So the bottom line is that I currently cannot reproduce the > bug. This > > may change when I finally import your config. However, it would be > > useful for me if you could run valgrind on rsyslogd in your > > environment and let me know if valgrind reports any memory leaks. > > Doing so considerably slows down rsyslogd, but given your load, I'd > > expect that it would be acceptable. > > > > To run under valgrind is relatively simply. Valgrind is available as > > a package on almost all distros. All you need to do is run valgrind > > and specify your usual rsyslogd command line as the parameter. It is > > recommended to do this in the foreground (see rsyslog > troubleshooting > > doc). > > > > So, for example, if you start rsyslogd usually by > > > > /sbin/rsyslogd -c4 -... > > So "valgrind /usr/sbin/rsyslogd -c4" results in rsyslogd backgrounding > promptly, at which point valgrind prints its report (which shows no > leaks - unsurprisingly). > > "valgrind /usr/sbin/rsyslogd -c4 -n" results in a hang. > CTRL-C fails to > kill the foreground task. That's intended behavior, ctrl-c is only enabled in debug mode (I took this over from sysklogd and never question it". > "kill -9 " kills the task, but no > valgrind report is produced. You must not use -9, which is untrappable. Just send the default SIGTERM: $ kill # no signal specified at all! >The same command without valgrind also > results in a hung foreground task. If run under valgrind, > memcheck-x86-li goes to 99% CPU on CTRL-C. > > I currently suspect problems with mySQL as the origin of > these problems. > I was this morning getting messages of the form > "Lost connection to MySQL server at 'reading authorization packet'". > I was also observing MySQL aborted clients and connects. I have > increased the MySQL connect timeout, and can no longer reproduce these > reports. For now, I assume that problem is fixed, but I can't > yet say if > the rsyslog hangs have stopped. > > I wonder if what was happening was that MySQL was "going away" in some > sense, and that rsyslog was not reconnecting to it successfully, *and* > not retrying? Hard to guess... > > I noticed that although the ActionQueue for the mysql output module is > not disk-assisted, the debug log records: > action 4 queue: save on shutdown 1, max disk space allowed 0 The debug output just spits out the values of the variables. Saveonshutdown is true by default, but if there is no disk queue, that won't help anything at all ;) In short: that's OK. > So I've set $ActionQueueSaveOnShutdown off. However this > hasn't changed > the hang behaviour with -n. > > Since I can't get rsyslog to run in the foreground under > valgrind, I am > now running daemonized without valgrind (but with encryption); perhaps > these changes have fixed the problem. I should know by late > this evening > - when the problem is observed, the server never lives for more than a > few hours. I hope this -and the other- mail will help straighten out the issue. Any logs you can send to my private mail address as you have already done ;) Rainer > -- > Jack. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 10 10:28:55 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 10:28:55 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103395@GRFEXC.intern.adiscon.com><4AF8424D.30306@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033A8@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033AB@GRFEXC.intern.adiscon.com> > This would explain what we currently see. It seems to be TLS-related, > as you > said. So it is unlikely to be a message at all, but rather a TLS event > that > causes the mem leak. With that said, I should probbably see if a > connection > abort can trigger one... Just for the records, the connection aborts I triggered did NOT result in leaked memory :( Rainer From rgerhards at hq.adiscon.com Tue Nov 10 10:53:16 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 10:53:16 +0100 Subject: [rsyslog] 4.4.2 leaks: another log References: <4AF93722.8040101@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com> excellent! I also see some violations, what is a good thing. But I notice I unfortunately missed one thing: by default, the violations do not include the loadbale module addresses. Would it be possible that you create a custom build of rsyslog with the "--enable-valgrind" configure option? That does not cause any overhead, but permits to see the function names on exit (and thus is really useful). Thanks, Rainer > -----Original Message----- > From: Mr. Demeanour [mailto:mrdemeanour at jackpot.uk.net] > Sent: Tuesday, November 10, 2009 10:49 AM > To: Rainer Gerhards > Subject: 4.4.2 leaks: another log > > Hi. > > Thanks for all your help. I didn't realise that -n suppressed SIGKILL. > I > also didn't realise that it was taking up to five minutes with -n and > valgrind, between the successful opening of a UDP listener on 514 and a > TCP listener appearing on 10514! That (and my misunderstanding of > signal > handling) is why I thought it had hung. > > So perhaps something is strange about my certificate. I'll see if I can > test it somehow. > > Anyway, here is a log apparently showing leakage; it represents about > 400 TCP messages over 30 minutes. The command was: > valgrind --leak-check=full /usr/sbin/rsyslogd -c4 -n >rsyslog.log 2>&1 > > By the time I killed it, memory usage had climbed from about 70M to > 100M. With a non-encrypting rsyslog (and without valgrind), usage is > stable at around 70M. Here's a "free" while running a non-encrypting > service: > > # free > total used free shared buffers > cached > Mem: 774972 732564 42408 0 22544 > 641436 > -/+ buffers/cache: 68584 706388 > Swap: 1951856 5640 1946216 > > The picture remains like that more-or-less indefinitely. > > Just before I killed the valgrind run corresponding to that log, it > looked like this: > # free > total used free shared buffers > cached > Mem: 774972 766292 8680 0 22016 > 643940 > -/+ buffers/cache: 100336 674636 > Swap: 1951856 5640 1946216 > > > > Now I know how to do this, I should be able to do it on demand. Thanks > again. > > -- > Jack. From rgerhards at hq.adiscon.com Tue Nov 10 11:09:41 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 11:09:41 +0100 Subject: [rsyslog] 4.4.2 leaks: another log References: <4AF93722.8040101@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> It just occurred to me that you may be seeing an anomaly in clib's malloc subsystem. I fought with this in early summer, but somehow have not mentioned it in the change log. The change itself was done on June, 22nd 2009. I have just checked, 4.4.2 does not have this code. I noticed that while there were some reports in the valgrind log, they were not large enough to explain what you saw. The libc malloc issue was discussed on the mailing list, and this here is a link to a later wrap-up type of message: http://lists.adiscon.net/pipermail/rsyslog/2009-June/002372.html So my suggestion is to move to v4-stable and see if the problem persists there (obviously, contrary to what I said, v5 would have fixed it in that case as well - so never trust a dev... ;)). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, November 10, 2009 10:53 AM > To: rsyslog-users > Subject: Re: [rsyslog] 4.4.2 leaks: another log > > excellent! I also see some violations, what is a good thing. But I > notice I > unfortunately missed one thing: by default, the violations do not > include the > loadbale module addresses. Would it be possible that you create a > custom > build of rsyslog with the "--enable-valgrind" configure option? That > does not > cause any overhead, but permits to see the function names on exit (and > thus > is really useful). > > Thanks, > Rainer > > > -----Original Message----- > > From: Mr. Demeanour [mailto:mrdemeanour at jackpot.uk.net] > > Sent: Tuesday, November 10, 2009 10:49 AM > > To: Rainer Gerhards > > Subject: 4.4.2 leaks: another log > > > > Hi. > > > > Thanks for all your help. I didn't realise that -n suppressed > SIGKILL. > > I > > also didn't realise that it was taking up to five minutes with -n and > > valgrind, between the successful opening of a UDP listener on 514 and > a > > TCP listener appearing on 10514! That (and my misunderstanding of > > signal > > handling) is why I thought it had hung. > > > > So perhaps something is strange about my certificate. I'll see if I > can > > test it somehow. > > > > Anyway, here is a log apparently showing leakage; it represents about > > 400 TCP messages over 30 minutes. The command was: > > valgrind --leak-check=full /usr/sbin/rsyslogd -c4 -n >rsyslog.log > 2>&1 > > > > By the time I killed it, memory usage had climbed from about 70M to > > 100M. With a non-encrypting rsyslog (and without valgrind), usage is > > stable at around 70M. Here's a "free" while running a non-encrypting > > service: > > > > # free > > total used free shared buffers > > cached > > Mem: 774972 732564 42408 0 22544 > > 641436 > > -/+ buffers/cache: 68584 706388 > > Swap: 1951856 5640 1946216 > > > > The picture remains like that more-or-less indefinitely. > > > > Just before I killed the valgrind run corresponding to that log, it > > looked like this: > > # free > > total used free shared buffers > > cached > > Mem: 774972 766292 8680 0 22016 > > 643940 > > -/+ buffers/cache: 100336 674636 > > Swap: 1951856 5640 1946216 > > > > > > > > Now I know how to do this, I should be able to do it on demand. > Thanks > > again. > > > > -- > > Jack. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mrdemeanour at jackpot.uk.net Tue Nov 10 11:29:41 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Tue, 10 Nov 2009 10:29:41 +0000 Subject: [rsyslog] 4.4.2 leaks: another log In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> References: <4AF93722.8040101@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> Message-ID: <4AF94095.2040205@jackpot.uk.net> Rainer Gerhards wrote: > It just occurred to me that you may be seeing an anomaly in clib's > malloc subsystem. I fought with this in early summer, but somehow > have not mentioned it in the change log. The change itself was done > on June, 22nd 2009. I have just checked, 4.4.2 does not have this > code. I noticed that while there were some reports in the valgrind > log, they were not large enough to explain what you saw. > > The libc malloc issue was discussed on the mailing list, and this > here is a link to a later wrap-up type of message: > > http://lists.adiscon.net/pipermail/rsyslog/2009-June/002372.html > > So my suggestion is to move to v4-stable and see if the problem > persists there (obviously, contrary to what I said, v5 would have > fixed it in that case as well - so never trust a dev... ;)). OK. I have to be somewhere else in half an hour, so I'll return to this later. -- Jack. From rgerhards at hq.adiscon.com Tue Nov 10 11:30:59 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 11:30:59 +0100 Subject: [rsyslog] 4.4.2 leaks: another log References: <4AF93722.8040101@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> <4AF94095.2040205@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033AE@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Tuesday, November 10, 2009 11:30 AM > To: rsyslog-users > Subject: Re: [rsyslog] 4.4.2 leaks: another log > > Rainer Gerhards wrote: > > It just occurred to me that you may be seeing an anomaly in clib's > > malloc subsystem. I fought with this in early summer, but somehow > > have not mentioned it in the change log. The change itself was done > > on June, 22nd 2009. I have just checked, 4.4.2 does not have this > > code. I noticed that while there were some reports in the valgrind > > log, they were not large enough to explain what you saw. > > > > The libc malloc issue was discussed on the mailing list, and this > > here is a link to a later wrap-up type of message: > > > > http://lists.adiscon.net/pipermail/rsyslog/2009-June/002372.html > > > > So my suggestion is to move to v4-stable and see if the problem > > persists there (obviously, contrary to what I said, v5 would have > > fixed it in that case as well - so never trust a dev... ;)). args, important type: suggest to move to v4-beta NOT -stable... > > OK. > > I have to be somewhere else in half an hour, so I'll return to this > later. > > -- > Jack. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 10 14:03:00 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 14:03:00 +0100 Subject: [rsyslog] On-Demand Debugging Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033B1@GRFEXC.intern.adiscon.com> Hi all, I worked today on the capability to obtain debug information from a running instance. Unfortunately, a small code change was needed to make it work well. If you cannot upgrade or apply the patch, I can create instructions to do it even without the patch, but that is a rather inconvenient way of doing thigs. The patch is here, it should apply to all recent v4 and v5 builds: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=2b5e15d549940c7ace9017ea f40d523e3741c9a2 I have also updated the documentation of the debug system and hope that it is much more usable this time. It specifically explains how to work with this new type of on-demand debugging. Details are online: http://www.rsyslog.com/doc-debug.html If that capability is useful, I could extend it in some ways. Most importantly, the current version does NOT permit to send debug logs to another remote instance. However, I tried to do as few changes as possible in an effort to help our current debugging efforts. Any new functionality will probably be v5-exclusive. I hope this is a useful new feature. So far, it is available via the git heads only, but I'll see that I release as soon as it makes sense (I don't like to release just for this mini-feature...). Rainer From szybalski at gmail.com Wed Nov 11 02:02:47 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Tue, 10 Nov 2009 19:02:47 -0600 Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> Message-ID: <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> Hello, I have a web application deployed in multiprocess and multi-thread scenario. ( 3 processes and 10 threads each). I want to log search query string from users to a file ?called todaysdate.log ...20091109.log I'm using a python app but I can't get rsyslog in debian system to catch and write my messages. Would you know how can I set this up? Here is a sample test case....in python. Save below to a file and run it. python testfile.py #----testfile.py----- from logging.handlers import SysLogHandler import logging # create logger logger = logging.getLogger("myapp") logger.setLevel(logging.DEBUG) ch = SysLogHandler('/dev/log') ch.setLevel(logging.DEBUG) # create formatter formatter = logging.Formatter("%(asctime)s - %(name)s-%(levelname)s - %(message)s") # add formatter to ch ch.setFormatter(formatter) logger.addHandler(ch) logger.info('This is a message') How can I setup rsyslog to filter "myapp" and save the messages to a file in /home/lucas/myapp/20091109.txt Thanks, Lucas From mrdemeanour at jackpot.uk.net Wed Nov 11 18:38:21 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Wed, 11 Nov 2009 17:38:21 +0000 Subject: [rsyslog] 4.4.2 leaks: another log In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71033AE@GRFEXC.intern.adiscon.com> References: <4AF93722.8040101@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> <4AF94095.2040205@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AE@GRFEXC.intern.adiscon.com> Message-ID: <4AFAF68D.9060203@jackpot.uk.net> Rainer Gerhards wrote: >>> >>> So my suggestion is to move to v4-stable and see if the problem >>> persists there (obviously, contrary to what I said, v5 would have >>> fixed it in that case as well - so never trust a dev... ;)). > > > args, important type: suggest to move to v4-beta NOT -stable... OK, so it looks like I've still got the problem, with v4 beta (4.5.6). I'll have another go with valgrind tomorrow. -- Jack. From szybalski at gmail.com Thu Nov 12 15:19:26 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Thu, 12 Nov 2009 08:19:26 -0600 Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> Message-ID: <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> Anybody know what the filter or a config line would be that would filter my "myapp" messages to a file in /home/lucas/20091112.txt? Ideas? Thanks, Lucas On Tue, Nov 10, 2009 at 7:02 PM, Lukasz Szybalski wrote: > Hello, > I have a web application deployed in multiprocess and multi-thread > scenario. ( 3 processes and 10 threads each). > > I want to log search query string from users to a file ?called > todaysdate.log ...20091109.log > > > I'm using a python app but I can't get rsyslog in debian system to > catch and write my messages. Would you know how can I set this up? > > > Here is a sample test case....in python. > Save below to a file and run it. > python testfile.py > > > #----testfile.py----- > from logging.handlers import SysLogHandler > > import logging > > > # create logger > logger = logging.getLogger("myapp") > logger.setLevel(logging.DEBUG) > ch = SysLogHandler('/dev/log') > ch.setLevel(logging.DEBUG) > # create formatter > formatter = logging.Formatter("%(asctime)s - %(name)s-%(levelname)s - > %(message)s") > # add formatter to ch > ch.setFormatter(formatter) > logger.addHandler(ch) > logger.info('This is a message') > > > How can I setup rsyslog to filter "myapp" and save the messages to a > file in /home/lucas/myapp/20091109.txt > > Thanks, > Lucas > -- Setup CalendarServer for your company. http://lucasmanual.com/mywiki/CalendarServer Automotive Recall Database - See if you vehicle has a recall http://lucasmanual.com/recall From rgerhards at hq.adiscon.com Thu Nov 12 15:23:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 12 Nov 2009 15:23:44 +0100 Subject: [rsyslog] multiprocess/multithread web app to rsyslog References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com><804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> If "myapp" is the tag, you can use this: :syslogtag, contains, "maypp" /home/lucas/20091112.txt There are better comparisons than "contains", but I don't know them out of my head. They are in the filter doc. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Lukasz Szybalski > Sent: Thursday, November 12, 2009 3:19 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] multiprocess/multithread web app to rsyslog > > Anybody know what the filter or a config line would be that would > filter my "myapp" messages to a file in /home/lucas/20091112.txt? > > Ideas? > Thanks, > Lucas > > > On Tue, Nov 10, 2009 at 7:02 PM, Lukasz Szybalski > wrote: > > Hello, > > I have a web application deployed in multiprocess and multi-thread > > scenario. ( 3 processes and 10 threads each). > > > > I want to log search query string from users to a file ?called > > todaysdate.log ...20091109.log > > > > > > I'm using a python app but I can't get rsyslog in debian system to > > catch and write my messages. Would you know how can I set this up? > > > > > > Here is a sample test case....in python. > > Save below to a file and run it. > > python testfile.py > > > > > > #----testfile.py----- > > from logging.handlers import SysLogHandler > > > > import logging > > > > > > # create logger > > logger = logging.getLogger("myapp") > > logger.setLevel(logging.DEBUG) > > ch = SysLogHandler('/dev/log') > > ch.setLevel(logging.DEBUG) > > # create formatter > > formatter = logging.Formatter("%(asctime)s - %(name)s-%(levelname)s - > > %(message)s") > > # add formatter to ch > > ch.setFormatter(formatter) > > logger.addHandler(ch) > > logger.info('This is a message') > > > > > > How can I setup rsyslog to filter "myapp" and save the messages to a > > file in /home/lucas/myapp/20091109.txt > > > > Thanks, > > Lucas > > > > > > -- > Setup CalendarServer for your company. > http://lucasmanual.com/mywiki/CalendarServer > Automotive Recall Database - See if you vehicle has a recall > http://lucasmanual.com/recall > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From szybalski at gmail.com Thu Nov 12 15:59:59 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Thu, 12 Nov 2009 08:59:59 -0600 Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> Message-ID: <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> On Thu, Nov 12, 2009 at 8:23 AM, Rainer Gerhards wrote: > If "myapp" is the tag, you can use this: > > :syslogtag, contains, "maypp" /home/lucas/20091112.txt > > There are better comparisons than "contains", but I don't know them out of my > head. They are in the filter doc. > Getting closer. I put a file in /etc/rsyslog.d/myapp.conf Inside I have: :syslogtag, contains, "myapp" /home/lucas/20091112.txt the rsyslog creates the file but no messages are logged in, they only appear in /var/log/messages Nov 12 08:45:32 localhost 2009-11-12 08:45:32,452 - myapp - INFO - This is a message Ideas why its not logging it in? Also, Can I make this config file like this? It doesn't seem to work...? :syslogtag, contains, "myapp" /home/lucas/$year$month$day.txt http://www.rsyslog.com/doc-property_replacer.html Thanks, Lucas > HTH > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Lukasz Szybalski >> Sent: Thursday, November 12, 2009 3:19 PM >> To: rsyslog at lists.adiscon.com >> Subject: Re: [rsyslog] multiprocess/multithread web app to rsyslog >> >> Anybody know what the filter or a config line would be that would >> filter my "myapp" messages to a file in /home/lucas/20091112.txt? >> >> Ideas? >> Thanks, >> Lucas >> >> >> On Tue, Nov 10, 2009 at 7:02 PM, Lukasz Szybalski >> wrote: >> > Hello, >> > I have a web application deployed in multiprocess and multi-thread >> > scenario. ( 3 processes and 10 threads each). >> > >> > I want to log search query string from users to a file ?called >> > todaysdate.log ...20091109.log >> > >> > >> > I'm using a python app but I can't get rsyslog in debian system to >> > catch and write my messages. Would you know how can I set this up? >> > >> > >> > Here is a sample test case....in python. >> > Save below to a file and run it. >> > python testfile.py >> > >> > >> > #----testfile.py----- >> > from logging.handlers import SysLogHandler >> > >> > import logging >> > >> > >> > # create logger >> > logger = logging.getLogger("myapp") >> > logger.setLevel(logging.DEBUG) >> > ch = SysLogHandler('/dev/log') >> > ch.setLevel(logging.DEBUG) >> > # create formatter >> > formatter = logging.Formatter("%(asctime)s - %(name)s-%(levelname)s - >> > %(message)s") >> > # add formatter to ch >> > ch.setFormatter(formatter) >> > logger.addHandler(ch) >> > logger.info('This is a message') >> > >> > >> > How can I setup rsyslog to filter "myapp" and save the messages to a >> > file in /home/lucas/myapp/20091109.txt >> > >> > Thanks, >> > Lucas >> > >> >> From rgerhards at hq.adiscon.com Thu Nov 12 16:33:34 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 12 Nov 2009 16:33:34 +0100 Subject: [rsyslog] multiprocess/multithread web app to rsyslog References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com><804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com><804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Lukasz Szybalski > Sent: Thursday, November 12, 2009 4:00 PM > To: rsyslog-users > Subject: Re: [rsyslog] multiprocess/multithread web app to rsyslog > > On Thu, Nov 12, 2009 at 8:23 AM, Rainer Gerhards > wrote: > > If "myapp" is the tag, you can use this: > > > > :syslogtag, contains, "maypp" /home/lucas/20091112.txt > > > > There are better comparisons than "contains", but I don't know them > out of my > > head. They are in the filter doc. > > > > Getting closer. > I put a file in /etc/rsyslog.d/myapp.conf > > Inside I have: > :syslogtag, contains, "myapp" /home/lucas/20091112.txt > > the rsyslog creates the file but no messages are logged in, they only > appear in /var/log/messages > > Nov 12 08:45:32 localhost 2009-11-12 08:45:32,452 - myapp - INFO - aha! this message is not really well-formed, please also see http://www.rsyslog.com/doc-syslog_parsing.html So the tag here is "2009-11-12". You can either use a custom parser, or if that would be sufficent, check msg instead of syslogtag. Rainer > This is a message > > > Ideas why its not logging it in? > > Also, > Can I make this config file like this? It doesn't seem to work...? > :syslogtag, contains, "myapp" /home/lucas/$year$month$day.txt > > > http://www.rsyslog.com/doc-property_replacer.html > > Thanks, > Lucas > > > > > > HTH > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Lukasz Szybalski > >> Sent: Thursday, November 12, 2009 3:19 PM > >> To: rsyslog at lists.adiscon.com > >> Subject: Re: [rsyslog] multiprocess/multithread web app to rsyslog > >> > >> Anybody know what the filter or a config line would be that would > >> filter my "myapp" messages to a file in /home/lucas/20091112.txt? > >> > >> Ideas? > >> Thanks, > >> Lucas > >> > >> > >> On Tue, Nov 10, 2009 at 7:02 PM, Lukasz Szybalski > > >> wrote: > >> > Hello, > >> > I have a web application deployed in multiprocess and multi-thread > >> > scenario. ( 3 processes and 10 threads each). > >> > > >> > I want to log search query string from users to a file ?called > >> > todaysdate.log ...20091109.log > >> > > >> > > >> > I'm using a python app but I can't get rsyslog in debian system to > >> > catch and write my messages. Would you know how can I set this up? > >> > > >> > > >> > Here is a sample test case....in python. > >> > Save below to a file and run it. > >> > python testfile.py > >> > > >> > > >> > #----testfile.py----- > >> > from logging.handlers import SysLogHandler > >> > > >> > import logging > >> > > >> > > >> > # create logger > >> > logger = logging.getLogger("myapp") > >> > logger.setLevel(logging.DEBUG) > >> > ch = SysLogHandler('/dev/log') > >> > ch.setLevel(logging.DEBUG) > >> > # create formatter > >> > formatter = logging.Formatter("%(asctime)s - %(name)s- > %(levelname)s - > >> > %(message)s") > >> > # add formatter to ch > >> > ch.setFormatter(formatter) > >> > logger.addHandler(ch) > >> > logger.info('This is a message') > >> > > >> > > >> > How can I setup rsyslog to filter "myapp" and save the messages to > a > >> > file in /home/lucas/myapp/20091109.txt > >> > > >> > Thanks, > >> > Lucas > >> > > >> > >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From szybalski at gmail.com Thu Nov 12 18:18:02 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Thu, 12 Nov 2009 11:18:02 -0600 Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> Message-ID: <804e5c70911120918s935f756j602d1c0ce7f0798b@mail.gmail.com> >> >> > >> >> > >> >> > Here is a sample test case....in python. >> >> > Save below to a file and run it. >> >> > python testfile.py >> >> > >> >> > >> >> > #----testfile.py----- >> >> > from logging.handlers import SysLogHandler >> >> > >> >> > import logging >> >> > >> >> > >> >> > # create logger >> >> > logger = logging.getLogger("myapp") >> >> > logger.setLevel(logging.DEBUG) >> >> > ch = SysLogHandler('/dev/log') >> >> > ch.setLevel(logging.DEBUG) >> >> > # create formatter >> >> > formatter = logging.Formatter("%(asctime)s - %(name)s- >> %(levelname)s - >> >> > %(message)s") >> >> > # add formatter to ch >> >> > ch.setFormatter(formatter) >> >> > logger.addHandler(ch) >> >> > logger.info('This is a message') >> >> > >> >> > >> >> > How can I setup rsyslog to filter "myapp" and save the messages to 1. So in my code I've defined a formatter to be: formatter = logging.Formatter("%(asctime)s - %(name)s- %(levelname)s - %(message)s") I don't care how it looks as long as it has a date, time, name of the app, level, and message. Is there a standard formatting I should be using? I don't want to create a custom solution if I can use standard one? 2. What about saving the file with todays date. I don't want to rotate the logs, but I want to have a file for each day saved in /home/lucas/$year$month$date.txt Above does not work, unless I'm missing some escape characters/parentheses? Thanks, Lucas From david at lang.hm Thu Nov 12 20:25:40 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 12 Nov 2009 11:25:40 -0800 (PST) Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <804e5c70911120918s935f756j602d1c0ce7f0798b@mail.gmail.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> <804e5c70911120918s935f756j602d1c0ce7f0798b@mail.gmail.com> Message-ID: On Thu, 12 Nov 2009, Lukasz Szybalski wrote: >>>>>> >>>>>> >>>>>> Here is a sample test case....in python. >>>>>> Save below to a file and run it. >>>>>> python testfile.py >>>>>> >>>>>> >>>>>> #----testfile.py----- >>>>>> from logging.handlers import SysLogHandler >>>>>> >>>>>> import logging >>>>>> >>>>>> >>>>>> # create logger >>>>>> logger = logging.getLogger("myapp") >>>>>> logger.setLevel(logging.DEBUG) >>>>>> ch = SysLogHandler('/dev/log') >>>>>> ch.setLevel(logging.DEBUG) >>>>>> # create formatter >>>>>> formatter = logging.Formatter("%(asctime)s - %(name)s- >>> %(levelname)s - >>>>>> %(message)s") >>>>>> # add formatter to ch >>>>>> ch.setFormatter(formatter) >>>>>> logger.addHandler(ch) >>>>>> logger.info('This is a message') >>>>>> >>>>>> >>>>>> How can I setup rsyslog to filter "myapp" and save the messages to > > 1. > So in my code I've defined a formatter to be: > formatter = logging.Formatter("%(asctime)s - %(name)s- %(levelname)s - > %(message)s") > > I don't care how it looks as long as it has a date, time, name of the > app, level, and message. Is there a standard formatting I should be > using? I don't want to create a custom solution if I can use standard > one? the format over the wire should be date server tag message if the message does not have this format, rsyslog will add a priority, date and server to the message in addition, the date you are setting is not formatted appropriately the date should be MMM DD HH:MM:SS I am not familiar enough with the logger package in python to tell you how to format it this way. however, based on the message that you sent earlier, if you set your format to be formatter = logging.Formatter("%(name)s %(message)s") it may be close enough to the correct thing to work. > 2. What about saving the file with todays date. I don't want to rotate > the logs, but I want to have a file for each day saved in > /home/lucas/$year$month$date.txt > Above does not work, unless I'm missing some escape characters/parentheses? look in the section on dynafiles. I have not used this feature, so I can't help directly. David Lang From szybalski at gmail.com Thu Nov 12 23:32:43 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Thu, 12 Nov 2009 16:32:43 -0600 Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> <804e5c70911120918s935f756j602d1c0ce7f0798b@mail.gmail.com> Message-ID: <804e5c70911121432y404d0a00h2f080590140addf@mail.gmail.com> On Thu, Nov 12, 2009 at 1:25 PM, wrote: > On Thu, 12 Nov 2009, Lukasz Szybalski wrote: > >>>>>>> >>>>>>> >>>>>>> Here is a sample test case....in python. >>>>>>> Save below to a file and run it. >>>>>>> python testfile.py >>>>>>> >>>>>>> >>>>>>> #----testfile.py----- >>>>>>> from logging.handlers import SysLogHandler >>>>>>> >>>>>>> import logging >>>>>>> >>>>>>> >>>>>>> # create logger >>>>>>> logger = logging.getLogger("myapp") >>>>>>> logger.setLevel(logging.DEBUG) >>>>>>> ch = SysLogHandler('/dev/log') >>>>>>> ch.setLevel(logging.DEBUG) >>>>>>> # create formatter >>>>>>> formatter = logging.Formatter("%(asctime)s - %(name)s- >>>> %(levelname)s - >>>>>>> %(message)s") >>>>>>> # add formatter to ch >>>>>>> ch.setFormatter(formatter) >>>>>>> logger.addHandler(ch) >>>>>>> logger.info('This is a message') >>>>>>> >>>>>>> >>>>>>> How can I setup rsyslog to filter "myapp" and save the messages to >> >> 1. >> So in my code I've defined a formatter to be: >> formatter = logging.Formatter("%(asctime)s - %(name)s- %(levelname)s - >> %(message)s") >> >> I don't care how it looks as long as it has a date, time, name of the >> app, level, and message. Is there a standard formatting I should be >> using? I don't want to create a custom solution if I can use standard >> one? > > the format over the wire should be > > date server tag message > > if the message does not have this format, rsyslog will add a priority, > date and server to the message > > in addition, the date you are setting is not formatted appropriately > > the date should be > > MMM DD HH:MM:SS > > I am not familiar enough with the logger package in python to tell you how > to format it this way. > > however, based on the message that you sent earlier, if you set your > format to be > > formatter = logging.Formatter("%(name)s %(message)s") This works.... nice... > > it may be close enough to the correct thing to work. > >> 2. What about saving the file with todays date. I don't want to rotate >> the logs, but I want to have a file for each day saved in >> /home/lucas/$year$month$date.txt >> Above does not work, unless I'm missing some escape characters/parentheses? > > look in the section on dynafiles. I have not used this feature, so I can't > help directly. Now I just have to find the proper syntax to get the file to save to :syslogtag, contains, "myapp" /home/lucas/20091112.txt to /home/lucas/$year$month$date.txt Lucas From david at lang.hm Fri Nov 13 00:02:08 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 12 Nov 2009 15:02:08 -0800 (PST) Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <804e5c70911121432y404d0a00h2f080590140addf@mail.gmail.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> <804e5c70911120918s935f756j602d1c0ce7f0798b@mail.gmail.com> <804e5c70911121432y404d0a00h2f080590140addf@mail.gmail.com> Message-ID: On Thu, 12 Nov 2009, Lukasz Szybalski wrote: > On Thu, Nov 12, 2009 at 1:25 PM, wrote: >> On Thu, 12 Nov 2009, Lukasz Szybalski wrote: >> >> >> it may be close enough to the correct thing to work. >> >>> 2. What about saving the file with todays date. I don't want to rotate >>> the logs, but I want to have a file for each day saved in >>> /home/lucas/$year$month$date.txt >>> Above does not work, unless I'm missing some escape characters/parentheses? >> >> look in the section on dynafiles. I have not used this feature, so I can't >> help directly. > > Now I just have to find the proper syntax to get the file to save to > :syslogtag, contains, "myapp" /home/lucas/20091112.txt > to > /home/lucas/$year$month$date.txt look at DynFile in http://www.rsyslog.com/doc-rsyslog_conf_actions.html David Lang From tbergfeld at hq.adiscon.com Fri Nov 13 12:51:09 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 13 Nov 2009 12:51:09 +0100 Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com> Hi all, Today, we released a new v5-beta branch. It bases on the last v5-devel, plus has some bug fixes and some enhancements. The enhancements provide a slightly increased performance. This release is a very important milestone. Based on feedback from the v5-devel branches, it is in pretty good shape. Our aim is to have a quick beta phase this time (taking the problems with the current v5-stable) into account. To do so, we need your cooperation. Please try out this new version and report back any issues you may experience. It would also be good to know if you use it and do not run into any issues. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article427.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-187.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From david at lang.hm Sat Nov 14 03:48:06 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 13 Nov 2009 18:48:06 -0800 (PST) Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com> Message-ID: I've got this installed in production and it's working well so far. monday morning (pacific time) should give it a good stress test. David Lang On Fri, 13 Nov 2009, Tom Bergfeld wrote: > Date: Fri, 13 Nov 2009 12:51:09 +0100 > From: Tom Bergfeld > Reply-To: rsyslog-users > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released > > Hi all, > > Today, we released a new v5-beta branch. It bases on the last v5-devel, plus > has some bug fixes and some enhancements. The enhancements provide a slightly > increased performance. This release is a very important milestone. Based on > feedback from the v5-devel branches, it is in pretty good shape. Our aim is > to have a quick beta phase this time (taking the problems with the current > v5-stable) into account. To do so, we need your cooperation. Please try out > this new version and report back any issues you may experience. It would also > be good to know if you use it and do not run into any issues. > > See Changelog for more details. > > ChangeLog: > > http://www.rsyslog.com/Article427.phtml > > Download: > > http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-187.phtml > > > As always, feedback is appreciated. > > Best regards, > Tom Bergfeld > -- > Support > ======= > > Improving rsyslog is costly, but you can help! We are looking for > organizations that find rsyslog useful and wish to contribute back. You can > contribute by reporting bugs, improve the software, or donate money or > equipment. > > Commercial support contracts for rsyslog are available, and they help finance > continued maintenance. Adiscon GmbH, a privately held German company, is > currently funding rsyslog development. We are always looking for interesting > development projects. For details on how to help, please see > http://www.rsyslog.com/doc-how2help.html . > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mrdemeanour at jackpot.uk.net Sat Nov 14 14:59:43 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Sat, 14 Nov 2009 13:59:43 +0000 Subject: [rsyslog] 4.4.2 leaks: another log In-Reply-To: <4AFAF68D.9060203@jackpot.uk.net> References: <4AF93722.8040101@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> <4AF94095.2040205@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AE@GRFEXC.intern.adiscon.com> <4AFAF68D.9060203@jackpot.uk.net> Message-ID: <4AFEB7CF.3060309@jackpot.uk.net> Mr. Demeanour wrote: > Rainer Gerhards wrote: >>>> So my suggestion is to move to v4-stable and see if the problem >>>> persists there (obviously, contrary to what I said, v5 would >>>> have fixed it in that case as well - so never trust a dev... >>>> ;)). >> >> args, important type: suggest to move to v4-beta NOT -stable... > > OK, so it looks like I've still got the problem, with v4 beta > (4.5.6). I'll have another go with valgrind tomorrow. I have been running a setup with 4.5.6 collecting messages by GnuTLS only, and logging to a file; and the problem has failed to materialise. Perhaps I didn't run it for long enough; but a few hours is normally sufficient for serious memory depletion to occur, and it hasn't so far. The other variables are that only one (LAN) client is logging to this instance - all previous runs have involved multiple clients, some logging via the internet; and that only one input and one output method is configured. Previous tests have had both imudp and imtcp configured, and have used both omfile and ommysql. I've now changed this from logging to a file to log via mySQL. If that remains stable after several hours, I'll reconfigure with dual input and output methods; then with multiple clients. Eventually I should be able to describe a method of replication. -- Jack. From rgerhards at hq.adiscon.com Mon Nov 16 15:00:20 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 16 Nov 2009 15:00:20 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> Hi all, I am working again on moving the DNS name resolution outside of the input thread of those sources where this is potentially time-consuming and affecting message acceptance rates. As it turned out, currently imudp seems to be the only case. While this is potentially easy to do, a problem is ACLs ($AllowedSender) which use system names rather than ip addresses. In order to check these ACLs, we need to do a DNS lookup. Especially in the case of UDP, such a lookup may actually case message loss and thus may be abused by an attacker to cause a certain degree of denial of service (what also points out that these types of ACLs are not really a good idea, even though requested by practice). In the light of this, I will now do something that sounds strange at first: I will always accept messages that require DNS lookups and enqueue these into the main queue and do the name resolution AND the final name-based ACL check only on the queue consumer part. Please note that it will be done BEFORE message content is parsed, so there is no chance that buffer overlow attacks can be carried out from non-authenticated hosts. The core idea is to move the lengthy, potentially message-loss causing code, away from the input thread. The only questionable effect I can currently see is that queue space is potentially taken up by messages which will immediately be discarded and should not be there in the first place. At the extreme end, that could lead to loss of valid messages. But on the other hand valid messages are more likely to be lost by the DNS name query overhead if I do the ACL check directly in the input thread. As such, I think my intended move is correct. Does anyone have an argument against the approach I am now taking? Thanks, Rainer From theinric at redhat.com Mon Nov 16 17:24:30 2009 From: theinric at redhat.com (Tomas Heinrich) Date: Mon, 16 Nov 2009 17:24:30 +0100 Subject: [rsyslog] support for arbitrary number of open files Message-ID: <4B017CBE.80904@redhat.com> Hi all, currently the total number of files (and tcp connections) that can be open simultaneously is limited by the select() system call. Ideally this would be changed to *poll(), but that can take some time. Until that happens, this patch[1] tries to remove the limitations of select() by enlarging the bit mask that is used for storing file descriptor information and redefining the macros that process it. This modification is inspired by Bind's use of select(). It is rather a workaround and may not be entirely portable. The actual changes to the code aren't big, but they are in many places so sufficient testing is needed. Allocating and freeing fd_sets in some frequently called functions may decrease performance. This can be dealt with but would require more pervasive changes. Thoughts? Thanks, Tomas [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited-select.patch From david at lang.hm Mon Nov 16 17:51:40 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 16 Nov 2009 08:51:40 -0800 (PST) Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 16 Nov 2009, Rainer Gerhards wrote: > Hi all, > > I am working again on moving the DNS name resolution outside of the input > thread of those sources where this is potentially time-consuming and > affecting message acceptance rates. As it turned out, currently imudp seems > to be the only case. > > While this is potentially easy to do, a problem is ACLs ($AllowedSender) > which use system names rather than ip addresses. In order to check these > ACLs, we need to do a DNS lookup. Especially in the case of UDP, such a > lookup may actually case message loss and thus may be abused by an attacker > to cause a certain degree of denial of service (what also points out that > these types of ACLs are not really a good idea, even though requested by > practice). > > In the light of this, I will now do something that sounds strange at first: I > will always accept messages that require DNS lookups and enqueue these into > the main queue and do the name resolution AND the final name-based ACL check > only on the queue consumer part. Please note that it will be done BEFORE > message content is parsed, so there is no chance that buffer overlow attacks > can be carried out from non-authenticated hosts. The core idea is to move the > lengthy, potentially message-loss causing code, away from the input thread. > The only questionable effect I can currently see is that queue space is > potentially taken up by messages which will immediately be discarded and > should not be there in the first place. At the extreme end, that could lead > to loss of valid messages. But on the other hand valid messages are more > likely to be lost by the DNS name query overhead if I do the ACL check > directly in the input thread. > > As such, I think my intended move is correct. Does anyone have an argument > against the approach I am now taking? personally I don't think that this sort of filtering belongs in rsyslog, it can be done at the OS level (with things like iptables), or rsyslog could use the tcpwrappers library. both cases would filter (by IP) prior to it hitting rsyslog in the first place. in addition, with UDP the source IP can be forged easily (rsyslog now contains this capability), so as a security measure it's questionable anyway. I agree that fewer messages will probably be lost by accepting them and checking later than by pausing to do the check initially. David Lang From rgerhards at hq.adiscon.com Mon Nov 16 17:54:28 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 16 Nov 2009 17:54:28 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> > personally I don't think that this sort of filtering belongs in > rsyslog, > it can be done at the OS level (with things like iptables), or rsyslog > could use the tcpwrappers library. both cases would filter (by IP) > prior > to it hitting rsyslog in the first place. > > in addition, with UDP the source IP can be forged easily (rsyslog now > contains this capability), so as a security measure it's questionable > anyway. I agree, but many people seem to use this functionality, and it was introduced some years ago by request. I do not feel comfortable with the idea of removing support for it. The newer protocols do not support ACLs in any case. Are there some more voices in regard to removing that functionality? Would make the implementation (and probably the throughput) a bit simpler/faster. > I agree that fewer messages will probably be lost by accepting them and > checking later than by pausing to do the check initially. Thanks for voicing this. Rainer From rgerhards at hq.adiscon.com Mon Nov 16 18:08:32 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 16 Nov 2009 18:08:32 +0100 Subject: [rsyslog] support for arbitrary number of open files References: <4B017CBE.80904@redhat.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103412@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > Sent: Monday, November 16, 2009 5:25 PM > To: rsyslog-users > Subject: [rsyslog] support for arbitrary number of open files > > Hi all, > > currently the total number of files (and tcp connections) that can be > open simultaneously is limited by the select() system call. Ideally > this > would be changed to *poll(), but that can take some time. For imudp, this change already happened (in the current devel). The others are on my radar. TCP is probably the most problematic, I need to redo the driver level. I wonder if I should split out gssapi while doing so, because that would simplify a lot of the calling conventions (at the price of some code duplication). For real-world scenarios, I think tcp is probably the only real issue, it is hard to believe that e.g. imuxsock will have an issue with that ;) > Until that happens, this patch[1] tries to remove the limitations of > select() by enlarging the bit mask that is used for storing file > descriptor information and redefining the macros that process it. > This modification is inspired by Bind's use of select(). > It is rather a workaround and may not be entirely portable. > > The actual changes to the code aren't big, but they are in many places > so sufficient testing is needed. Allocating and freeing fd_sets in > some > frequently called functions may decrease performance. This can be dealt > with but would require more pervasive changes. I need to have a close look at the patch. I consider it useful, but I agree that it must be very carefully checked. I've had a quick look and my understanding is that it is enabled by conditional compilation. That would ease many of my concerns, as we could disable it by default and only those that actually need it are at risk. > > Thoughts? I'd also appreciate feedback from other list members. Thanks, Rainer > > Thanks, > Tomas > > [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited- > select.patch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From aoz.syn at gmail.com Mon Nov 16 18:45:57 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 16 Nov 2009 10:45:57 -0700 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> Message-ID: <4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com> On Mon, Nov 16, 2009 at 09:54, Rainer Gerhards wrote: > I agree, but many people seem to use this functionality, and it was > introduced some years ago by request. I do not feel comfortable with the idea > of removing support for it. The newer protocols do not support ACLs in any > case. > > Are there some more voices in regard to removing that functionality? Would > make the implementation (and probably the throughput) a bit simpler/faster. I agree with david's assessment that "security" by this type of ACL is minimally effective. However, the functionality is occasionally useful for situations where management of the software is easier than management of the firewall (typically for business/operational constraints). I'd love to see it reimplemented as a modular part of the ephemeral "middle layer" along with other filtering and modification functionality. I know RanierScript is supposed to fill a lot of that void, but until it's implemented and proven performant my wishes still lie with a filter layer API. >> I agree that fewer messages will probably be lost by accepting them and >> checking later than by pausing to do the check initially. Ditto - although conceptually pure, front-end checking is probably too expensive for such a performance-critical component as the receiver. From rgerhards at hq.adiscon.com Mon Nov 16 20:56:38 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 16 Nov 2009 20:56:38 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> <4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of RB > Sent: Monday, November 16, 2009 6:46 PM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback request: ACLs, imudp and > accepting messages > > On Mon, Nov 16, 2009 at 09:54, Rainer Gerhards > wrote: > > I agree, but many people seem to use this functionality, and it was > > introduced some years ago by request. I do not feel > comfortable with the idea > > of removing support for it. The newer protocols do not > support ACLs in any > > case. > > > > Are there some more voices in regard to removing that > functionality? Would > > make the implementation (and probably the throughput) a bit > simpler/faster. > > I agree with david's assessment that "security" by this type of ACL is > minimally effective. >From a security POV, it may be more effective to remove that option, at least for UDP (people than know it is not secure, but neither is a similar firewall setting). But I fear all those broken configs, then without any protection at all. Maybe a thing for a new v5 default, but even then... > However, the functionality is occasionally > useful for situations where management of the software is easier than > management of the firewall (typically for business/operational > constraints). I'd love to see it reimplemented as a modular part of > the ephemeral "middle layer" along with other filtering and > modification functionality. I fail to see the interface you envision. Could you elaborate a bit? That would be most useful. I guess that a lot of this can now be done by the custom parsers, it may just need to be documented as a valid use case. > I know RanierScript is supposed to fill a > lot of that void, but until it's implemented and proven performant My thinking is moving a bit away from this "once size fits all" approach, primarily for performance reasons. It is quite expensive to start up a full scripting engine (with its VM), so native code loadable modules require more work to be done upfront, but are much faster. Rainer > my > wishes still lie with a filter layer API. > > >> I agree that fewer messages will probably be lost by > accepting them and > >> checking later than by pausing to do the check initially. > > Ditto - although conceptually pure, front-end checking is probably too > expensive for such a performance-critical component as the receiver. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From aoz.syn at gmail.com Mon Nov 16 21:28:54 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 16 Nov 2009 13:28:54 -0700 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> <4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com> Message-ID: <4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> On Mon, Nov 16, 2009 at 12:56, Rainer Gerhards wrote: > I fail to see the interface you envision. Could you elaborate a bit? ?That > would be most useful. Certainly. Some time ago, I expressed interest in writing a 'running checksum' module but had run into issues with a lack of state-maintenance in the output modules that helped my lack of time drop the issue. At the time, you'd hinted at a 'filter' module API that would (I presumed) fit somewhere between input and output and eventually address that use case. Such a layer could perform message modification, rate limiting, ACL enforcement, and so on. Perhaps this has already been addressed, since I've not kept as close of tabs on the development head as I'd like. Ideally, I think it would be a very interesting step to be able to 'stack' syslog modules to define the path a given set of messages take through the collector. Something akin to the Linux iptables concept, where a basic high-speed framework is laid out with a well-defined order and an API that enables administrators to elect how specific run-time modules interact within the stack. That's less of a request and more of a pipe dream, and is definitely influenced by the half network security hat I wear. From rgerhards at hq.adiscon.com Mon Nov 16 21:52:15 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 16 Nov 2009 21:52:15 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com><4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com> <4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103415@GRFEXC.intern.adiscon.com> Let me think about this, but it really sounds like custom parser modules could be moved into that direction with relative ease (at least for part of the functionality). Some info on them: http://www.rsyslog.com/doc-messageparser.html Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of RB > Sent: Monday, November 16, 2009 9:29 PM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback request: ACLs, imudp and > accepting messages > > On Mon, Nov 16, 2009 at 12:56, Rainer Gerhards > wrote: > > I fail to see the interface you envision. Could you > elaborate a bit? ?That > > would be most useful. > > Certainly. Some time ago, I expressed interest in writing a 'running > checksum' module but had run into issues with a lack of > state-maintenance in the output modules that helped my lack of time > drop the issue. At the time, you'd hinted at a 'filter' module API > that would (I presumed) fit somewhere between input and output and > eventually address that use case. Such a layer could perform message > modification, rate limiting, ACL enforcement, and so on. Perhaps this > has already been addressed, since I've not kept as close of tabs on > the development head as I'd like. > > Ideally, I think it would be a very interesting step to be able to > 'stack' syslog modules to define the path a given set of messages take > through the collector. Something akin to the Linux iptables concept, > where a basic high-speed framework is laid out with a well-defined > order and an API that enables administrators to elect how specific > run-time modules interact within the stack. That's less of a request > and more of a pipe dream, and is definitely influenced by the half > network security hat I wear. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Tue Nov 17 03:58:14 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 16 Nov 2009 18:58:14 -0800 (PST) Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages In-Reply-To: <4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> <4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com> <4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> Message-ID: On Mon, 16 Nov 2009, RB wrote: > On Mon, Nov 16, 2009 at 12:56, Rainer Gerhards wrote: >> I fail to see the interface you envision. Could you elaborate a bit? ?That >> would be most useful. > > Certainly. Some time ago, I expressed interest in writing a 'running > checksum' module but had run into issues with a lack of > state-maintenance in the output modules that helped my lack of time > drop the issue. At the time, you'd hinted at a 'filter' module API > that would (I presumed) fit somewhere between input and output and > eventually address that use case. Such a layer could perform message > modification, rate limiting, ACL enforcement, and so on. Perhaps this > has already been addressed, since I've not kept as close of tabs on > the development head as I'd like. > > Ideally, I think it would be a very interesting step to be able to > 'stack' syslog modules to define the path a given set of messages take > through the collector. Something akin to the Linux iptables concept, > where a basic high-speed framework is laid out with a well-defined > order and an API that enables administrators to elect how specific > run-time modules interact within the stack. That's less of a request > and more of a pipe dream, and is definitely influenced by the half > network security hat I wear. hmm, the recent ability to have secondary queues would allow this to fit in very nicely as a filter for what to do when moving messages between the queues. David Lang From philr at jaspers.co.nz Tue Nov 17 04:10:13 2009 From: philr at jaspers.co.nz (Phil Reilly) Date: Tue, 17 Nov 2009 16:10:13 +1300 Subject: [rsyslog] Alerting rules via a database In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> References: <4AF5DAA7.5010001@jaspers.co.nz> <4AF6816A.6050901@jaspers.co.nz> <9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> Message-ID: <4B021415.7080906@jaspers.co.nz> Any luck with the template? Or should I just roll my own. Cheers, Phil Rainer Gerhards wrote: > So what you are actually looking for is a system that can work with > dynamically changable alert definitions? As David said, there is no such > thing currently, but the best road to approach is is to write a custom output > plugin, that you pass each message to. That plugin can even decide if > messages should be discarded and not further processed. I envisioned such a > plugin, but had not yet time to write, for a similar use case. > > If you intend to write one AND contribute it to the project, I can help you > get started with the interface, would even be willing to create you a custom > skeleton that you can fill in your logic ;) > > HTH > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phil Reilly >> Sent: Sunday, November 08, 2009 9:30 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Alerting rules via a database >> >> david at lang.hm wrote: >> >>> On Sun, 8 Nov 2009, Phil Reilly wrote: >>> >>> >>> >>>> I attempting to allow for flexible rule matches on Syslogs >>>> >> from a web >> >>>> front end (rather than entires into the rsyslog config files) >>>> >>>> I want to get regexp filters from a db to alert upon >>>> >> messages. Not sure >> >>>> the best way to achieve this. I've so far though of. >>>> >>>> * Outputting to a pipe and runing it via an alerting script. >>>> * Having file watch the messages. >>>> * Recieving the messages then passing them to rsyslog (yuck) >>>> >>>> Can the rule engine allow for match rules outside of the config? is >>>> there an elegant way of doing this? >>>> >>>> >>> rsyslog doesn't give you this ability, but it's not really the best >>> approach to use for alerting anyway. >>> >>> what are you trying to achieve by having the alert definitions in a >>> database? there are several tools out there to do alerting >>> >> (SEC, Simple >> >>> Event Correlator) is one of the leading ones, but I'm not >>> >> aware of any of >> >>> them that use a database for their rulesets. >>> >>> I'm also scratching my head trying to figure out what the >>> >> advantage of >> >>> doing so would be. >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> Thanks David, >> >> We have a networked environment. We also have a web page that >> allows you >> to configure regexp to match certain syslog messages. These >> patterns are >> compiled and kept in a table. The current syslog process we >> use listens >> for udp. When it gets a syslog message, we examine the >> patterns (which >> are re-read upon addition or change) and pass them to an alertering >> process before writing the logs to disk. The existing system >> works well, >> but we now want to scale it over a few machines and I'm >> examining what >> syslog products out there cater for alerting. >> >> So a database will make configuring alerts far more dynamic than >> statically entering them into a config file. It will also allow for >> grouped views so different groups have the ability to have >> custom alerts >> based upon their own interpretation of syslog messages. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 17 07:28:25 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 07:28:25 +0100 Subject: [rsyslog] Alerting rules via a database Message-ID: <000901ca674f$38fd00a1$100013ac@intern.adiscon.com> Sorry, i have to admit it slipped my mind. Will create one this morning. rainer ----- Urspr?ngliche Nachricht ----- Von: "Phil Reilly" An: "rsyslog-users" Gesendet: 17.11.09 04:10 Betreff: Re: [rsyslog] Alerting rules via a database Any luck with the template? Or should I just roll my own. Cheers, Phil Rainer Gerhards wrote: > So what you are actually looking for is a system that can work with > dynamically changable alert definitions? As David said, there is no such > thing currently, but the best road to approach is is to write a custom output > plugin, that you pass each message to. That plugin can even decide if > messages should be discarded and not further processed. I envisioned such a > plugin, but had not yet time to write, for a similar use case. > > If you intend to write one AND contribute it to the project, I can help you > get started with the interface, would even be willing to create you a custom > skeleton that you can fill in your logic ;) > > HTH > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phil Reilly >> Sent: Sunday, November 08, 2009 9:30 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Alerting rules via a database >> >> david at lang.hm wrote: >> >>> On Sun, 8 Nov 2009, Phil Reilly wrote: >>> >>> >>> >>>> I attempting to allow for flexible rule matches on Syslogs >>>> >> from a web >> >>>> front end (rather than entires into the rsyslog config files) >>>> >>>> I want to get regexp filters from a db to alert upon >>>> >> messages. Not sure >> >>>> the best way to achieve this. I've so far though of. >>>> >>>> * Outputting to a pipe and runing it via an alerting script. >>>> * Having file watch the messages. >>>> * Recieving the messages then passing them to rsyslog (yuck) >>>> >>>> Can the rule engine allow for match rules outside of the config? is >>>> there an elegant way of doing this? >>>> >>>> >>> rsyslog doesn't give you this ability, but it's not really the best >>> approach to use for alerting anyway. >>> >>> what are you trying to achieve by having the alert definitions in a >>> database? there are several tools out there to do alerting >>> >> (SEC, Simple >> >>> Event Correlator) is one of the leading ones, but I'm not >>> >> aware of any of >> >>> them that use a database for their rulesets. >>> >>> I'm also scratching my head trying to figure out what the >>> >> advantage of >> >>> doing so would be. >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> Thanks David, >> >> We have a networked environment. We also have a web page that >> allows you >> to configure regexp to match certain syslog messages. These >> patterns are >> compiled and kept in a table. The current syslog process we >> use listens >> for udp. When it gets a syslog message, we examine the >> patterns (which >> are re-read upon addition or change) and pass them to an alertering >> process before writing the logs to disk. The existing system >> works well, >> but we now want to scale it over a few machines and I'm >> examining what >> syslog products out there cater for alerting. >> >> So a database will make configuring alerts far more dynamic than >> statically entering them into a config file. It will also allow for >> grouped views so different groups have the ability to have >> custom alerts >> based upon their own interpretation of syslog messages. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 17 08:15:22 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 08:15:22 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com><4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com><4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103417@GRFEXC.intern.adiscon.com> > > Ideally, I think it would be a very interesting step to be able to > > 'stack' syslog modules to define the path a given set of messages > take > > through the collector. Something akin to the Linux iptables concept, > > where a basic high-speed framework is laid out with a well-defined > > order and an API that enables administrators to elect how specific > > run-time modules interact within the stack. That's less of a request > > and more of a pipe dream, and is definitely influenced by the half > > network security hat I wear. > > hmm, the recent ability to have secondary queues would allow this to > fit > in very nicely as a filter for what to do when moving messages between > the > queues. I have thought more about this over the night. I think most of the capabilities are already present, just need to be a bit "re-labled". However, what is missing is functionality to use loadable modules as a filter condition. However, especially for complex filters, this sounds very valuable, both from a performance as well as from a capability POV. It also looks like it is not hard to add. I will definitely look into that direction. Thanks all for bringing up these thoughts, any further comments are of course appreciated as well ;) Rainer From rgerhards at hq.adiscon.com Tue Nov 17 08:49:28 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 08:49:28 +0100 Subject: [rsyslog] Alerting rules via a database References: <4AF5DAA7.5010001@jaspers.co.nz><4AF6816A.6050901@jaspers.co.nz><9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> <4B021415.7080906@jaspers.co.nz> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103418@GRFEXC.intern.adiscon.com> It is now present in the master branch. It resides in ./plugins/omdbalerting to enable it, use $ ./configure --enable-omdbalerting Obviously, the code now needs to be populated. I suggest that you create a simple sample with fixed constants, submit it to me, and then we can talk about how to obtain parameters from the config file. I hope this is useful. If you have any question (I guess you will have), please ask. But be sure to review module-template.h before beginning with any work. Looking at omtemplate.c is also a good idea. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Phil Reilly > Sent: Tuesday, November 17, 2009 4:10 AM > To: rsyslog-users > Subject: Re: [rsyslog] Alerting rules via a database > > Any luck with the template? > > Or should I just roll my own. > > Cheers, > > Phil > > Rainer Gerhards wrote: > > So what you are actually looking for is a system that can work with > > dynamically changable alert definitions? As David said, there is no > such > > thing currently, but the best road to approach is is to write a > custom output > > plugin, that you pass each message to. That plugin can even decide if > > messages should be discarded and not further processed. I envisioned > such a > > plugin, but had not yet time to write, for a similar use case. > > > > If you intend to write one AND contribute it to the project, I can > help you > > get started with the interface, would even be willing to create you a > custom > > skeleton that you can fill in your logic ;) > > > > HTH > > Rainer > > > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phil Reilly > >> Sent: Sunday, November 08, 2009 9:30 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Alerting rules via a database > >> > >> david at lang.hm wrote: > >> > >>> On Sun, 8 Nov 2009, Phil Reilly wrote: > >>> > >>> > >>> > >>>> I attempting to allow for flexible rule matches on Syslogs > >>>> > >> from a web > >> > >>>> front end (rather than entires into the rsyslog config files) > >>>> > >>>> I want to get regexp filters from a db to alert upon > >>>> > >> messages. Not sure > >> > >>>> the best way to achieve this. I've so far though of. > >>>> > >>>> * Outputting to a pipe and runing it via an alerting script. > >>>> * Having file watch the messages. > >>>> * Recieving the messages then passing them to rsyslog (yuck) > >>>> > >>>> Can the rule engine allow for match rules outside of the config? > is > >>>> there an elegant way of doing this? > >>>> > >>>> > >>> rsyslog doesn't give you this ability, but it's not really the best > >>> approach to use for alerting anyway. > >>> > >>> what are you trying to achieve by having the alert definitions in a > >>> database? there are several tools out there to do alerting > >>> > >> (SEC, Simple > >> > >>> Event Correlator) is one of the leading ones, but I'm not > >>> > >> aware of any of > >> > >>> them that use a database for their rulesets. > >>> > >>> I'm also scratching my head trying to figure out what the > >>> > >> advantage of > >> > >>> doing so would be. > >>> > >>> David Lang > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >>> > >> Thanks David, > >> > >> We have a networked environment. We also have a web page that > >> allows you > >> to configure regexp to match certain syslog messages. These > >> patterns are > >> compiled and kept in a table. The current syslog process we > >> use listens > >> for udp. When it gets a syslog message, we examine the > >> patterns (which > >> are re-read upon addition or change) and pass them to an alertering > >> process before writing the logs to disk. The existing system > >> works well, > >> but we now want to scale it over a few machines and I'm > >> examining what > >> syslog products out there cater for alerting. > >> > >> So a database will make configuring alerts far more dynamic than > >> statically entering them into a config file. It will also allow for > >> grouped views so different groups have the ability to have > >> custom alerts > >> based upon their own interpretation of syslog messages. > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 17 10:00:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 10:00:44 +0100 Subject: [rsyslog] support for arbitrary number of open files References: <4B017CBE.80904@redhat.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710341A@GRFEXC.intern.adiscon.com> Tomas, I am currently working on integration of this patch. What puzzles me is the call to "howmany()". I don't find any doc on it, nor an implementation inside the patch. Could you elaborate what it does and where it stems from? Also, I think there is a segfault in gss-misc, because the glbl interface is never aquired (the will result in a NULL-pointer dereference). I also need to change the glbl interface definitions, FdSetSize must always be present, else the interface is no longer well-defined. I will post the completely integrated patch when I am done. But feedback on howmany() would be most appreciated, because I currently do not know exactly how to handle it. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > Sent: Monday, November 16, 2009 5:25 PM > To: rsyslog-users > Subject: [rsyslog] support for arbitrary number of open files > > Hi all, > > currently the total number of files (and tcp connections) that can be > open simultaneously is limited by the select() system call. Ideally > this > would be changed to *poll(), but that can take some time. > > Until that happens, this patch[1] tries to remove the limitations of > select() by enlarging the bit mask that is used for storing file > descriptor information and redefining the macros that process it. > This modification is inspired by Bind's use of select(). > It is rather a workaround and may not be entirely portable. > > The actual changes to the code aren't big, but they are in many places > so sufficient testing is needed. Allocating and freeing fd_sets in > some > frequently called functions may decrease performance. This can be dealt > with but would require more pervasive changes. > > Thoughts? > > Thanks, > Tomas > > [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited- > select.patch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 17 10:49:16 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 10:49:16 +0100 Subject: [rsyslog] support for arbitrary number of open files References: <4B017CBE.80904@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA710341A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710341B@GRFEXC.intern.adiscon.com> OK, I have finally integrated the patch, I could work around my missing knowledge of howmany() [but still would appreciate some more info on it]. As of rsyslog policy, new features are never integrated into stable or beta builds. As such, it is available in the v4-devel and master branches. Note that I made a number of changes, you will probably like to re-check for your use case. I ran the testbench successfully in both modes and did some manual tests with the new functionality disabled. The v5-branch does NOT include the patch for imudp. As you said, it is a (good ;)) work-around and imudp already supports epoll(), so there is no need for this workaround. I plan to gradually remove that feature in v5 a I work towards supporting epoll in all inputs. The master branch will probably be released tomorrow, v4-devel as need arises (but it is available from git right now). Thanks again, I think this is a very useful addition. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, November 17, 2009 10:01 AM > To: rsyslog-users > Subject: Re: [rsyslog] support for arbitrary number of open files > > Tomas, > > I am currently working on integration of this patch. What puzzles me is > the > call to "howmany()". I don't find any doc on it, nor an implementation > inside > the patch. Could you elaborate what it does and where it stems from? > > Also, I think there is a segfault in gss-misc, because the glbl > interface is > never aquired (the will result in a NULL-pointer dereference). I also > need to > change the glbl interface definitions, FdSetSize must always be > present, else > the interface is no longer well-defined. I will post the completely > integrated patch when I am done. > > But feedback on howmany() would be most appreciated, because I > currently do > not know exactly how to handle it. > > Thanks, > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > > Sent: Monday, November 16, 2009 5:25 PM > > To: rsyslog-users > > Subject: [rsyslog] support for arbitrary number of open files > > > > Hi all, > > > > currently the total number of files (and tcp connections) that can be > > open simultaneously is limited by the select() system call. Ideally > > this > > would be changed to *poll(), but that can take some time. > > > > Until that happens, this patch[1] tries to remove the limitations of > > select() by enlarging the bit mask that is used for storing file > > descriptor information and redefining the macros that process it. > > This modification is inspired by Bind's use of select(). > > It is rather a workaround and may not be entirely portable. > > > > The actual changes to the code aren't big, but they are in many > places > > so sufficient testing is needed. Allocating and freeing fd_sets in > > some > > frequently called functions may decrease performance. This can be > dealt > > with but would require more pervasive changes. > > > > Thoughts? > > > > Thanks, > > Tomas > > > > [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited- > > select.patch > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 17 11:29:57 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 11:29:57 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com><4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com><4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103417@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710341E@GRFEXC.intern.adiscon.com> I have added some doc about what modules can do today: http://www.rsyslog.com/doc-rsyslog_conf_modules.html Note that message modification is possible, it was just not spelled out. The actual code also currently creates the false impression it is not possible. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, November 17, 2009 8:15 AM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback request: ACLs, imudp and accepting > messages > > > > Ideally, I think it would be a very interesting step to be able to > > > 'stack' syslog modules to define the path a given set of messages > > take > > > through the collector. Something akin to the Linux iptables > concept, > > > where a basic high-speed framework is laid out with a well-defined > > > order and an API that enables administrators to elect how specific > > > run-time modules interact within the stack. That's less of a > request > > > and more of a pipe dream, and is definitely influenced by the half > > > network security hat I wear. > > > > hmm, the recent ability to have secondary queues would allow this to > > fit > > in very nicely as a filter for what to do when moving messages > between > > the > > queues. > > I have thought more about this over the night. I think most of the > capabilities are already present, just need to be a bit "re-labled". > However, > what is missing is functionality to use loadable modules as a filter > condition. However, especially for complex filters, this sounds very > valuable, both from a performance as well as from a capability POV. It > also > looks like it is not hard to add. > > I will definitely look into that direction. Thanks all for bringing up > these > thoughts, any further comments are of course appreciated as well ;) > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Nov 18 04:13:06 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 17 Nov 2009 19:13:06 -0800 (PST) Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 13 Nov 2009, david at lang.hm wrote: > I've got this installed in production and it's working well so far. monday > morning (pacific time) should give it a good stress test. it's survived since friday without a problem on a couple of dozen machines (all my perimiter relay boxes plus my core logging server) looking good. David Lang > David Lang > > On Fri, 13 Nov 2009, Tom Bergfeld wrote: > >> Date: Fri, 13 Nov 2009 12:51:09 +0100 >> From: Tom Bergfeld >> Reply-To: rsyslog-users >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released >> >> Hi all, >> >> Today, we released a new v5-beta branch. It bases on the last v5-devel, >> plus >> has some bug fixes and some enhancements. The enhancements provide a >> slightly >> increased performance. This release is a very important milestone. Based on >> feedback from the v5-devel branches, it is in pretty good shape. Our aim is >> to have a quick beta phase this time (taking the problems with the current >> v5-stable) into account. To do so, we need your cooperation. Please try out >> this new version and report back any issues you may experience. It would >> also >> be good to know if you use it and do not run into any issues. >> >> See Changelog for more details. >> >> ChangeLog: >> >> http://www.rsyslog.com/Article427.phtml >> >> Download: >> >> http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-187.phtml >> >> >> As always, feedback is appreciated. >> >> Best regards, >> Tom Bergfeld >> -- >> Support >> ======= >> >> Improving rsyslog is costly, but you can help! We are looking for >> organizations that find rsyslog useful and wish to contribute back. You >> can >> contribute by reporting bugs, improve the software, or donate money or >> equipment. >> >> Commercial support contracts for rsyslog are available, and they help >> finance >> continued maintenance. Adiscon GmbH, a privately held German company, is >> currently funding rsyslog development. We are always looking for >> interesting >> development projects. For details on how to help, please see >> http://www.rsyslog.com/doc-how2help.html . >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > From rgerhards at hq.adiscon.com Wed Nov 18 09:23:24 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 18 Nov 2009 09:23:24 +0100 Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released References: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710342D@GRFEXC.intern.adiscon.com> This is excellent news, thanks for sharing it. Did anybody else also try it? Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, November 18, 2009 4:13 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 5.3.5 (v5-beta) released > > On Fri, 13 Nov 2009, david at lang.hm wrote: > > > I've got this installed in production and it's working well so far. > monday > > morning (pacific time) should give it a good stress test. > > it's survived since friday without a problem on a couple of dozen > machines > (all my perimiter relay boxes plus my core logging server) > > looking good. > > David Lang > > > David Lang > > > > On Fri, 13 Nov 2009, Tom Bergfeld wrote: > > > >> Date: Fri, 13 Nov 2009 12:51:09 +0100 > >> From: Tom Bergfeld > >> Reply-To: rsyslog-users > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released > >> > >> Hi all, > >> > >> Today, we released a new v5-beta branch. It bases on the last v5- > devel, > >> plus > >> has some bug fixes and some enhancements. The enhancements provide a > >> slightly > >> increased performance. This release is a very important milestone. > Based on > >> feedback from the v5-devel branches, it is in pretty good shape. Our > aim is > >> to have a quick beta phase this time (taking the problems with the > current > >> v5-stable) into account. To do so, we need your cooperation. Please > try out > >> this new version and report back any issues you may experience. It > would > >> also > >> be good to know if you use it and do not run into any issues. > >> > >> See Changelog for more details. > >> > >> ChangeLog: > >> > >> http://www.rsyslog.com/Article427.phtml > >> > >> Download: > >> > >> http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid- > 187.phtml > >> > >> > >> As always, feedback is appreciated. > >> > >> Best regards, > >> Tom Bergfeld > >> -- > >> Support > >> ======= > >> > >> Improving rsyslog is costly, but you can help! We are looking for > >> organizations that find rsyslog useful and wish to contribute back. > You > >> can > >> contribute by reporting bugs, improve the software, or donate money > or > >> equipment. > >> > >> Commercial support contracts for rsyslog are available, and they > help > >> finance > >> continued maintenance. Adiscon GmbH, a privately held German > company, is > >> currently funding rsyslog development. We are always looking for > >> interesting > >> development projects. For details on how to help, please see > >> http://www.rsyslog.com/doc-how2help.html . > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From martinmie at PartyGaming.com Wed Nov 18 10:52:21 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Wed, 18 Nov 2009 10:52:21 +0100 Subject: [rsyslog] rsyslog in uninterruptable sleep In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032D3@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA0892EE19@GIBSVWIN004X.partygaming.local> <9B6E2A8877C38245BFB15CC491A11DA71032D3@GRFEXC.intern.adiscon.com> Message-ID: <0B1B9163138571439EAF804646F3F6AA08B398B6@GIBSVWIN004X.partygaming.local> Hi Rainer, Since some weeks ago we run rsyslog 4.4.2 and unfortunately yesterday I stumbled on rsyslog with "D"-state process again... so I guess that the fix is somewhere else. Can you please confirm which rsyslog version provides the solution for this problem? Thanks, Martin > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: 27 October 2009 16:02 > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog in uninterruptable sleep > > I think we just recently had this on the mailing list or bug tracker and I > made a fix for it. I guess the fix is included in 4.4.2. > > HTH > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Martin Mielke > > Sent: Tuesday, October 27, 2009 11:57 AM > > To: rsyslog-users > > Subject: [rsyslog] rsyslog in uninterruptable sleep > > > > > > Hi, > > > > I just did some small changes to the rsyslog.conf and wanted to restart > > the process: > > > > # /etc/init.d/rsyslog restart > > Shutting down system logger: [FAILED] > > Starting system logger: Already running. > > [FAILED] > > > > # ps -elf | grep rsyslog > > 5 D root 2037 1 1 76 0 - 12040 sync_b Oct18 ? > > 03:32:10 rsyslogd -c 4 -x -f /etc/rsyslog.conf -i /var/run/syslogd.pid > > > > > > This is rsyslog 4.2.0 running on RHEL5 and it's not the first time it > > happens. > > > > > > Any ideas on why this might happened and how to solve this w/o > > rebooting > > the system?? > > > > > > Thanks, > > Martin > > > > > > This email and any attachments are confidential, and may be legally > > privileged and protected by copyright. If you are not the intended > > recipient dissemination or copying of this email is prohibited. If you > > have received this in error, please notify the sender by replying by > > email and then delete the email completely from your system. > > > > Any views or opinions are solely those of the sender. This > > communication is not intended to form a binding contract unless > > expressly indicated to the contrary and properly authorised. Any > > actions taken on the basis of this email are at the recipient's own > > risk. > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > yslog > http://www.rsyslog.com From tbergfeld at hq.adiscon.com Wed Nov 18 11:05:28 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 18 Nov 2009 11:05:28 +0100 Subject: [rsyslog] rsyslog 4.5.7 (v4-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103435@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.5.7, a member of the v4-beta branch. Most importantly, a so-called "On Demand Debug" mode was added, in which debug output can be generated only after the process has started, but not right from the beginning. This is assumed to be useful for hard-to-find bugs. Also improved the doc on the debug system. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article429.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-188.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From tbergfeld at hq.adiscon.com Wed Nov 18 11:14:30 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 18 Nov 2009 11:14:30 +0100 Subject: [rsyslog] rsyslog 5.5.0 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103437@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.5.0, a member of the devel branch. The 5.5.0 version begins a new development branch. The primary new functionality in that release is moving away reverse DNS resolution from the udp receiver's input thread to backend processing, what should reduce UDP message loss. ChangeLog: http://www.rsyslog.com/Article431.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-189.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From theinric at redhat.com Wed Nov 18 14:18:48 2009 From: theinric at redhat.com (Tomas Heinrich) Date: Wed, 18 Nov 2009 14:18:48 +0100 Subject: [rsyslog] support for arbitrary number of open files In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710341B@GRFEXC.intern.adiscon.com> References: <4B017CBE.80904@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA710341A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710341B@GRFEXC.intern.adiscon.com> Message-ID: <4B03F438.5030303@redhat.com> Rainer, thanks for integrating it and thanks for fixing the things I've overlooked. I've skimmed through the commit logs and the changes look good, I'll run some more tests with the new sources. Regarding howmany(), it is defined here in my environment: /usr/include/sys/param.h: # define howmany(x, y) (((x) + ((y) - 1)) / (y)) It computes the number of 'y's needed to store 'x'. Tomas On 11/17/2009 10:49 AM, Rainer Gerhards wrote: > OK, I have finally integrated the patch, I could work around my missing > knowledge of howmany() [but still would appreciate some more info on it]. > > As of rsyslog policy, new features are never integrated into stable or beta > builds. As such, it is available in the v4-devel and master branches. Note > that I made a number of changes, you will probably like to re-check for your > use case. I ran the testbench successfully in both modes and did some manual > tests with the new functionality disabled. > > The v5-branch does NOT include the patch for imudp. As you said, it is a > (good ;)) work-around and imudp already supports epoll(), so there is no need > for this workaround. I plan to gradually remove that feature in v5 a I work > towards supporting epoll in all inputs. > > The master branch will probably be released tomorrow, v4-devel as need arises > (but it is available from git right now). > > Thanks again, I think this is a very useful addition. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Tuesday, November 17, 2009 10:01 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] support for arbitrary number of open files >> >> Tomas, >> >> I am currently working on integration of this patch. What puzzles me is >> the >> call to "howmany()". I don't find any doc on it, nor an implementation >> inside >> the patch. Could you elaborate what it does and where it stems from? >> >> Also, I think there is a segfault in gss-misc, because the glbl >> interface is >> never aquired (the will result in a NULL-pointer dereference). I also >> need to >> change the glbl interface definitions, FdSetSize must always be >> present, else >> the interface is no longer well-defined. I will post the completely >> integrated patch when I am done. >> >> But feedback on howmany() would be most appreciated, because I >> currently do >> not know exactly how to handle it. >> >> Thanks, >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich >>> Sent: Monday, November 16, 2009 5:25 PM >>> To: rsyslog-users >>> Subject: [rsyslog] support for arbitrary number of open files >>> >>> Hi all, >>> >>> currently the total number of files (and tcp connections) that can be >>> open simultaneously is limited by the select() system call. Ideally >>> this >>> would be changed to *poll(), but that can take some time. >>> >>> Until that happens, this patch[1] tries to remove the limitations of >>> select() by enlarging the bit mask that is used for storing file >>> descriptor information and redefining the macros that process it. >>> This modification is inspired by Bind's use of select(). >>> It is rather a workaround and may not be entirely portable. >>> >>> The actual changes to the code aren't big, but they are in many >> places >>> so sufficient testing is needed. Allocating and freeing fd_sets in >>> some >>> frequently called functions may decrease performance. This can be >> dealt >>> with but would require more pervasive changes. >>> >>> Thoughts? >>> >>> Thanks, >>> Tomas >>> >>> [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited- >>> select.patch >>> From rgerhards at hq.adiscon.com Wed Nov 18 14:35:33 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 18 Nov 2009 14:35:33 +0100 Subject: [rsyslog] support for arbitrary number of open files References: <4B017CBE.80904@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA710341A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710341B@GRFEXC.intern.adiscon.com> <4B03F438.5030303@redhat.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710343E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > Sent: Wednesday, November 18, 2009 2:19 PM > To: rsyslog-users > Subject: Re: [rsyslog] support for arbitrary number of open files > > Rainer, > > thanks for integrating it and thanks for fixing the things I've > overlooked. > > I've skimmed through the commit logs and the changes look good, I'll > run > some more tests with the new sources. excellent - pls let me know if you see some issues. As a side-note, I've also used this as a reminder to finally look at epoll() for the plain tcp driver, probably the most prominent case, also performance wise. I hope I am able continue work on it, so we should have it in the not so distant future (in v5). > > Regarding howmany(), it is defined here in my environment: > /usr/include/sys/param.h: > # define howmany(x, y) (((x) + ((y) - 1)) / (y)) > It computes the number of 'y's needed to store 'x'. > Ahhh! I should have done a grep ;) But somehow did not expect it there... Thanks, Rainer > Tomas > > On 11/17/2009 10:49 AM, Rainer Gerhards wrote: > > OK, I have finally integrated the patch, I could work around my > missing > > knowledge of howmany() [but still would appreciate some more info on > it]. > > > > As of rsyslog policy, new features are never integrated into stable > or beta > > builds. As such, it is available in the v4-devel and master branches. > Note > > that I made a number of changes, you will probably like to re-check > for your > > use case. I ran the testbench successfully in both modes and did some > manual > > tests with the new functionality disabled. > > > > The v5-branch does NOT include the patch for imudp. As you said, it > is a > > (good ;)) work-around and imudp already supports epoll(), so there is > no need > > for this workaround. I plan to gradually remove that feature in v5 a > I work > > towards supporting epoll in all inputs. > > > > The master branch will probably be released tomorrow, v4-devel as > need arises > > (but it is available from git right now). > > > > Thanks again, I think this is a very useful addition. > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >> Sent: Tuesday, November 17, 2009 10:01 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] support for arbitrary number of open files > >> > >> Tomas, > >> > >> I am currently working on integration of this patch. What puzzles me > is > >> the > >> call to "howmany()". I don't find any doc on it, nor an > implementation > >> inside > >> the patch. Could you elaborate what it does and where it stems from? > >> > >> Also, I think there is a segfault in gss-misc, because the glbl > >> interface is > >> never aquired (the will result in a NULL-pointer dereference). I > also > >> need to > >> change the glbl interface definitions, FdSetSize must always be > >> present, else > >> the interface is no longer well-defined. I will post the completely > >> integrated patch when I am done. > >> > >> But feedback on howmany() would be most appreciated, because I > >> currently do > >> not know exactly how to handle it. > >> > >> Thanks, > >> Rainer > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > >>> Sent: Monday, November 16, 2009 5:25 PM > >>> To: rsyslog-users > >>> Subject: [rsyslog] support for arbitrary number of open files > >>> > >>> Hi all, > >>> > >>> currently the total number of files (and tcp connections) that can > be > >>> open simultaneously is limited by the select() system call. Ideally > >>> this > >>> would be changed to *poll(), but that can take some time. > >>> > >>> Until that happens, this patch[1] tries to remove the limitations > of > >>> select() by enlarging the bit mask that is used for storing file > >>> descriptor information and redefining the macros that process it. > >>> This modification is inspired by Bind's use of select(). > >>> It is rather a workaround and may not be entirely portable. > >>> > >>> The actual changes to the code aren't big, but they are in many > >> places > >>> so sufficient testing is needed. Allocating and freeing fd_sets in > >>> some > >>> frequently called functions may decrease performance. This can be > >> dealt > >>> with but would require more pervasive changes. > >>> > >>> Thoughts? > >>> > >>> Thanks, > >>> Tomas > >>> > >>> [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited- > >>> select.patch > >>> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Nov 19 10:58:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 19 Nov 2009 10:58:53 +0100 Subject: [rsyslog] clarification on priorities for rsyslog work Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710344B@GRFEXC.intern.adiscon.com> Hi all, as we currently have a lot of requests (both bugs and features), I would like to make *my* decision process a bit more transparent. So I have blogged about how I usually assign priorities: http://blog.gerhards.net/2009/11/priorities-for-rsyslog-work.html I hope this is useful. Rainer From mrdemeanour at jackpot.uk.net Thu Nov 19 19:19:02 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Thu, 19 Nov 2009 18:19:02 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <4AF2F9CD.9000402@jackpot.uk.net> References: <4AF2F9CD.9000402@jackpot.uk.net> Message-ID: <4B058C16.7010008@jackpot.uk.net> Mr. Demeanour wrote: > Hi, > > I'm running a central rsyslog server with a couple of remote WAN > (internet) clients and several remote LAN clients. Traffic is low - > of the order of 10,000 messages per day. Internet clients communicate > with the server using gnutls. LAN clients are currently using UDP. > The server writes client logs to mysql, and also writes messages of > local origin to disk. Further to this: I have been running 4.5.6 for about a week now, *without* gnutls enabled. No leaks. This evening I re-enabled gnutls, and almost immediately noted excessive memory usage, *and* 99% cpu. It seems that the high CPU usage occurs with hosts outside my local network; it may be that there is some misconfiguration of NAT that is behind that problem. I note that leaks are possible with the versions of gnutls shipping with Debian: http://permalink.gmane.org/gmane.network.gnutls.general/1465 That document describes a leak that would be expected to arise during connection setup, but not per message. I guess a dubious connection (e.g. resulting from misconfigured NAT) might result in repeated setup attempts, and so in leaks *and* cpu spiking. -- Jack. From mrdemeanour at jackpot.uk.net Thu Nov 19 19:50:16 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Thu, 19 Nov 2009 18:50:16 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <4B058C16.7010008@jackpot.uk.net> References: <4AF2F9CD.9000402@jackpot.uk.net> <4B058C16.7010008@jackpot.uk.net> Message-ID: <4B059368.2070701@jackpot.uk.net> Mr. Demeanour wrote: > > It seems that the high CPU usage occurs with hosts outside my local > network; it may be that there is some misconfiguration of NAT that is > behind that problem. Hmmm. What I meant is that the problem was only observed with gnutls *clients* outside my local network; the rsyslog server is inside. Sorry for being unclear. -- Jack. From rgerhards at hq.adiscon.com Thu Nov 19 21:30:22 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 19 Nov 2009 21:30:22 +0100 Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing References: <003001ca6092$792833d0$6b789b70$@com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> Hi, Sorry, I seem to have overlooked that message. But the mailing list processor has stripped the attachment. Could you please re-send to my personal mail address. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Jonathan Bond-Caron > Sent: Sunday, November 08, 2009 5:43 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing > > Hi all, > > > > Attached is a patch for 3 things: > > > > a) Check that TCP connection is still alive when using TLS > > > > b) Improve TAG parsing so that it ends when it sees a > tab or escape > control character. > > > > c) Added configuration directive: escapecontrolcharactertab > > In rsyslog.conf, you can set: > > $escapeControlCharactersOnReceive on > > $escapeControlCharacterTab off > > > > This avoids having a tab being escaped since it is after a viewable > character J > > I'd prefer this being the default behavior but I've left > $escapeControlCharacterTab on as default in case people > expect tabs to be > escaped. > > > > This is my first GIT patch, hope it works well ;) > > > > From david at lang.hm Thu Nov 19 21:37:32 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 19 Nov 2009 12:37:32 -0800 (PST) Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> References: <003001ca6092$792833d0$6b789b70$@com> <9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> Message-ID: for what it's worth, I think that b and c are very useful when dealing with strange data sources, and a is probably a bugfix. David Lang On Thu, 19 Nov 2009, Rainer Gerhards wrote: > > Sorry, I seem to have overlooked that message. But the mailing list processor > has stripped the attachment. Could you please re-send to my personal mail > address. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >> Jonathan Bond-Caron >> Sent: Sunday, November 08, 2009 5:43 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing >> >> Hi all, >> >> >> >> Attached is a patch for 3 things: >> >> >> >> a) Check that TCP connection is still alive when using TLS >> >> >> >> b) Improve TAG parsing so that it ends when it sees a >> tab or escape >> control character. >> >> >> >> c) Added configuration directive: escapecontrolcharactertab >> >> In rsyslog.conf, you can set: >> >> $escapeControlCharactersOnReceive on >> >> $escapeControlCharacterTab off >> >> >> >> This avoids having a tab being escaped since it is after a viewable >> character J >> >> I'd prefer this being the default behavior but I've left >> $escapeControlCharacterTab on as default in case people >> expect tabs to be >> escaped. >> >> >> >> This is my first GIT patch, hope it works well ;) >> >> >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Nov 20 11:03:59 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 20 Nov 2009 11:03:59 +0100 Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing References: <003001ca6092$792833d0$6b789b70$@com><9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710346E@GRFEXC.intern.adiscon.com> I, too, think this is useful. Will look at it as soon as I get the patch. The b) and c) parts may be a bit harder to integrate if they are not for the newest devel branch, as I rewrote the parser part of rsyslog. But in any case, it is worth trying to merge it in. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, November 19, 2009 9:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] [PATCH] TLS connection check + syslog TAG > parsing > > for what it's worth, I think that b and c are very useful when dealing > with strange data sources, and a is probably a bugfix. > > David Lang > > On Thu, 19 Nov 2009, Rainer Gerhards wrote: > > > > > Sorry, I seem to have overlooked that message. But the mailing list > processor > > has stripped the attachment. Could you please re-send to my personal > mail > > address. > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >> Jonathan Bond-Caron > >> Sent: Sunday, November 08, 2009 5:43 PM > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing > >> > >> Hi all, > >> > >> > >> > >> Attached is a patch for 3 things: > >> > >> > >> > >> a) Check that TCP connection is still alive when using TLS > >> > >> > >> > >> b) Improve TAG parsing so that it ends when it sees a > >> tab or escape > >> control character. > >> > >> > >> > >> c) Added configuration directive: escapecontrolcharactertab > >> > >> In rsyslog.conf, you can set: > >> > >> $escapeControlCharactersOnReceive on > >> > >> $escapeControlCharacterTab off > >> > >> > >> > >> This avoids having a tab being escaped since it is after a viewable > >> character J > >> > >> I'd prefer this being the default behavior but I've left > >> $escapeControlCharacterTab on as default in case people > >> expect tabs to be > >> escaped. > >> > >> > >> > >> This is my first GIT patch, hope it works well ;) > >> > >> > >> > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mrdemeanour at jackpot.uk.net Fri Nov 20 17:10:24 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 20 Nov 2009 16:10:24 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <4B058C16.7010008@jackpot.uk.net> References: <4AF2F9CD.9000402@jackpot.uk.net> <4B058C16.7010008@jackpot.uk.net> Message-ID: <4B06BF70.6070900@jackpot.uk.net> Mr. Demeanour wrote: > Mr. Demeanour wrote: >> Hi, >> >> I'm running a central rsyslog server with a couple of remote WAN >> (internet) clients and several remote LAN clients. Traffic is low - >> of the order of 10,000 messages per day. Internet clients >> communicate with the server using gnutls. LAN clients are currently >> using UDP. The server writes client logs to mysql, and also writes >> messages of local origin to disk. > > Further to this: > > I have been running 4.5.6 for about a week now, *without* gnutls > enabled. No leaks. > > This evening I re-enabled gnutls, and almost immediately noted > excessive memory usage, *and* 99% cpu. > > It seems that the high CPU usage occurs with hosts outside my local > network; it may be that there is some misconfiguration of NAT that is > behind that problem. Not NAT. It seems that I had set up the server certificate with an incorrect CN. I guess the client was trying repeatedly to make a connection that was doomed to fail every time. That would explain the CPU spike. If there is also a memory leak in the gnutls server code concerning connection setup, that would explain the memory consumption also. Perhaps rsyslog should give up trying to connect to a remote server, or at least back off, if the error it encounters is of a kind that most likely requires human intervention? Such would generally be the case if a certificate is invalid. -- Jack. From mrdemeanour at jackpot.uk.net Fri Nov 20 17:11:37 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 20 Nov 2009 16:11:37 +0000 Subject: [rsyslog] Possible bug: REs in templates Message-ID: <4B06BFB9.8080100@jackpot.uk.net> The following template produces one character fewer than expected. %HOSTNAME:R,ERE,1,FIELD,0:([^\.]+)--end% If, for example, the HOSTNAME string is 192.168.1.1, then the output is "19". -- Jack. From rgerhards at hq.adiscon.com Sun Nov 22 12:42:20 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 22 Nov 2009 12:42:20 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net> <4B058C16.7010008@jackpot.uk.net> <4B06BF70.6070900@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710348E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Friday, November 20, 2009 5:10 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Mr. Demeanour wrote: > > Mr. Demeanour wrote: > >> Hi, > >> > >> I'm running a central rsyslog server with a couple of remote WAN > >> (internet) clients and several remote LAN clients. Traffic is low - > >> of the order of 10,000 messages per day. Internet clients > >> communicate with the server using gnutls. LAN clients are currently > >> using UDP. The server writes client logs to mysql, and also writes > >> messages of local origin to disk. > > > > Further to this: > > > > I have been running 4.5.6 for about a week now, *without* gnutls > > enabled. No leaks. > > > > This evening I re-enabled gnutls, and almost immediately noted > > excessive memory usage, *and* 99% cpu. > > > > It seems that the high CPU usage occurs with hosts outside my local > > network; it may be that there is some misconfiguration of > NAT that is > > behind that problem. > > Not NAT. It seems that I had set up the server certificate with an > incorrect CN. > > I guess the client was trying repeatedly to make a connection that was > doomed to fail every time. That would explain the CPU spike. > If there is > also a memory leak in the gnutls server code concerning connection > setup, that would explain the memory consumption also. > > Perhaps rsyslog should give up trying to connect to a remote > server, or > at least back off, if the error it encounters is of a kind that most > likely requires human intervention? Such would generally be > the case if > a certificate is invalid. The situation is a bit complex in such cases. There, a shortcoming in RFC5425 "works together" with a shortcoming of GnuTLS. The end result is that I am unable to detect a problem occurs. This happens: RFC5425 does not specifiy app-level acknowledgements. Consequently, there is no way to convey an error condition back to the client. GnuTLS does not permit to check the certificate *BEFORE* accepting a connection. Consequently, the connection must be accepted, then the certificate checked and then terminated if the cert if invalid. However, to the client this looks like a successfully established connection (beause it was connected) that just quickly went down (but the client does not know an error state as there is no app-layer vehicle to provide it back). This also disables the usual rsyslog logic to pause failed connection attempts - because the connection succeeds in the first place. The only solution I currently see is either not to use RFC5425 or use something else than GnuTLS (for example, openssl permits to check the certificate BEFORE accepting the connection). An other approach is to modify GnuTLS so that it provides an ability to do the check before accepting the certificate. I looked into that option, but it requires far too much effort to me. So the right thing to do would probably write an additional TLS driver for e.g. openssl. While this is actually on the todo list, I have to admit that I do not have time to do so at the moment, nor do I see a spot in the near future where I could tackle that beast. I still have the hope that we will receive a contributed driver for NSS, which would most probably also solve this issue. Or wait for GnuTLS to evolve... I guess you get the picture ;) Rainer From xkubina at fi.muni.cz Mon Nov 23 16:56:14 2009 From: xkubina at fi.muni.cz (Tomas Kubina) Date: Mon, 23 Nov 2009 16:56:14 +0100 Subject: [rsyslog] imgssapi configuration Message-ID: <4B0AB09E.9030005@fi.muni.cz> Hi, I am trying to set up rsyslog server with GSSAPI auth. I have used the doc that is here http://www.rsyslog.com/doc-gssapi.html but without successful end. The issue is that when I want to start rsyslog server it hangs-up during starting procedure. Here are some details: I have version 5.2.0 and this is server's conf file: === $ModLoad immark # provides --MARK-- message capability $ModLoad imuxsock # provides support for local system logging (e.g. via logger) $ModLoad imklog # kernel logging (formerly provided by rklogd) #Global directives $ActionFileDefaultTemplate RSYSLOG_FileFormat <(rules for storing syslog messages)> $ModLoad imgssapi $InputGSSServerPermitPlainTCP off $InputGSSServerRun 10514 === client config is simple: === $ModLoad omgssapi *.info : omgssapi:server.example.com:10514 === Here is a part of debug output: ============================ 9766.638881000:2b51af4b1ac0: cfline: '$ModLoad imgssapi' 9766.638893000:2b51af4b1ac0: Requested to load module 'imgssapi' 9766.638905000:2b51af4b1ac0: loading module '/usr/local/lib/rsyslog/imgssapi.so' 9766.640185000:2b51af4b1ac0: caller requested object 'tcps_sess', not found (iRet -3003) 9766.640207000:2b51af4b1ac0: Requested to load module 'lmtcpsrv' 9766.640219000:2b51af4b1ac0: loading module '/usr/local/lib/rsyslog/lmtcpsrv.so' 9766.640292000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 80: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[1/2b51af4b1ac0] 9766.640309000:2b51af4b1ac0: caller requested object 'netstrm', not found (iRet -3003) 9766.640322000:2b51af4b1ac0: Requested to load module 'lmnetstrms' 9766.640333000:2b51af4b1ac0: loading module '/usr/local/lib/rsyslog/lmnetstrms.so' 9766.640394000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 80: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[1/2b51af4b1ac0] 9766.640409000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640424000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 80: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[1/2b51af4b1ac0] 9766.640436000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640450000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 80: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[1/2b51af4b1ac0] 9766.640461000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640477000:2b51af4b1ac0: module of type 2 being loaded. 9766.640488000:2b51af4b1ac0: entry point 'isCompatibleWithFeature' not present in module 9766.640500000:2b51af4b1ac0: source file tcps_sess.c requested reference for module 'lmnetstrms', reference count now 1 9766.640512000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640528000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640541000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640553000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640573000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640587000:2b51af4b1ac0: source file tcpsrv.c requested reference for module 'lmnet', reference count now 4 9766.640599000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640611000:2b51af4b1ac0: source file tcpsrv.c requested reference for module 'lmnetstrms', reference count now 2 9766.640624000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640637000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640651000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640665000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640678000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640691000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640705000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640717000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640737000:2b51af4b1ac0: module of type 2 being loaded. 9766.640748000:2b51af4b1ac0: entry point 'isCompatibleWithFeature' not present in module 9766.640759000:2b51af4b1ac0: source file imgssapi.c requested reference for module 'lmtcpsrv', reference count now 1 9766.640771000:2b51af4b1ac0: source file imgssapi.c requested reference for module 'lmtcpsrv', reference count now 2 9766.640784000:2b51af4b1ac0: caller requested object 'gssutil', not found (iRet -3003) 9766.640794000:2b51af4b1ac0: Requested to load module 'lmgssutil' 9766.640805000:2b51af4b1ac0: loading module '/usr/local/lib/rsyslog/lmgssutil.so' 9766.640880000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 100: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[1/2b51af4b1ac0] 9766.640897000:2b51af4b1ac0: module of type 2 being loaded. 9766.640908000:2b51af4b1ac0: entry point 'isCompatibleWithFeature' not present in module 9766.640919000:2b51af4b1ac0: source file imgssapi.c requested reference for module 'lmgssutil', reference count now 1 9766.640933000:2b51af4b1ac0: source file imgssapi.c requested reference for module 'lmnetstrms', reference count now 3 9766.640946000:2b51af4b1ac0: source file imgssapi.c requested reference for module 'lmnet', reference count now 5 9766.640969000:2b51af4b1ac0: module of type 0 being loaded. 9766.640985000:2b51af4b1ac0: cfline: '$InputGSSServerPermitPlainTCP off' 9766.641000000:2b51af4b1ac0: cfline: '$InputGSSServerRun 10514' 9766.641027000:2b51af4b1ac0: Signal 11 (SIGSEGV) occured, execution must be terminated. 9766.641147000:2b51af4b1ac0: 9766.641175000:2b51af4b1ac0: Recorded Call Order for Thread '2b51af4b1ac0': 9766.641186000:2b51af4b1ac0: 0: syslogd.c:3097:realMain: 9766.641196000:2b51af4b1ac0: 1: syslogd.c:2749:mainThread: 9766.641206000:2b51af4b1ac0: 2: syslogd.c:2185:init: 9766.641219000:2b51af4b1ac0: 3: conf.c:410:processConfFile: 9766.641229000:2b51af4b1ac0: 4: conf.c:1179:cfline: 9766.641238000:2b51af4b1ac0: 5: conf.c:359:cfsysline: 9766.641249000:2b51af4b1ac0: 6: cfsysline.c:902:processCfSysLineCommand: 9766.641259000:2b51af4b1ac0: 7: cfsysline.c:673:cslchCallHdlr: 9766.641269000:2b51af4b1ac0: 8: cfsysline.c:501:doGetWord: 9766.641281000:2b51af4b1ac0: 9: imgssapi.c:310:addGSSListener: 9766.641292000:2b51af4b1ac0: 10: tcpsrv.c:135:configureTCPListen: 9766.641301000:2b51af4b1ac0: 11: tcpsrv.c:102:addNewLstnPort: 9766.641311000:2b51af4b1ac0: maximum number of nested calls for this thread: 26. 9766.641324000:2b51af4b1ac0: NOTE: not all calls may have been recorded, code does not currently guarantee that! 9766.641333000:2b51af4b1ac0: Mutex log for all known mutex operations: 9766.641343000:2b51af4b1ac0: If the call trace is empty, you may want to ./configure --enable-rtinst 9766.641353000:2b51af4b1ac0: To submit bug reports, visit http://www.rsyslog.com/bugs 9766.641363000:2b51af4b1ac0: To submit bug reports, visit http://www.rsyslog.com/bugs Aborted === I am a newbie, any idea? Thanks, Tomas Kubina From josesan311 at yahoo.com Thu Nov 26 05:23:11 2009 From: josesan311 at yahoo.com (Jose Sanchez) Date: Wed, 25 Nov 2009 20:23:11 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog Message-ID: <948839.16588.qm@web35605.mail.mud.yahoo.com> Hello, I've been using classic syslog for centralizing apache access logs from one server to a remote syslog server, the thing is syslog adds some nasty tags before the lines in the access logs and I cant get them off, ie: "Nov 25 21:25:37 server1 logger:" I would like to know if rsyslog has the option to filter this kind of stuff, I just want to have the logs sent to the syslog server exactly like I was saving them in a local access.log file. Thanks in advance. From rgerhards at hq.adiscon.com Thu Nov 26 09:11:01 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 26 Nov 2009 09:11:01 +0100 Subject: [rsyslog] filter logger tags from syslog References: <948839.16588.qm@web35605.mail.mud.yahoo.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034AC@GRFEXC.intern.adiscon.com> I don't have the specifics at hand, but you can surely do that. I would even assume that if you just write the %msg% property to file, the infomation should look good. If not, you need to fiddle a bit with the property replacer. The basic idea is to use $template line,"%msg%\n" *.* /path/to/log;line HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jose Sanchez > Sent: Thursday, November 26, 2009 5:23 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] filter logger tags from syslog > > Hello, > > I've been using classic syslog for centralizing apache access > logs from one server to a remote syslog server, the thing is > syslog adds some nasty tags before the lines in the access > logs and I cant get them off, ie: > > "Nov 25 21:25:37 server1 logger:" > > I would like to know if rsyslog has the option to filter this > kind of stuff, I just want to have the logs sent to the > syslog server exactly like I was saving them in a local > access.log file. > > Thanks in advance. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu Nov 26 09:21:44 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 26 Nov 2009 00:21:44 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <948839.16588.qm@web35605.mail.mud.yahoo.com> References: <948839.16588.qm@web35605.mail.mud.yahoo.com> Message-ID: On Wed, 25 Nov 2009, Jose Sanchez wrote: > Hello, > > I've been using classic syslog for centralizing apache access logs from one server to a remote syslog server, the thing is syslog adds some nasty tags before the lines in the access logs and I cant get them off, ie: > > "Nov 25 21:25:37 server1 logger:" > > I would like to know if rsyslog has the option to filter this kind of stuff, I just want to have the logs sent to the syslog server exactly like I was saving them in a local access.log file. > > Thanks in advance. 'logger:' is added by the logger program that apache is using to send the logs to syslog. a properly formatted syslog message will include a timestamp and what server it came from (note that the apache logs do _not_ tell you what virtual server the log comes from, it usually uses a different file for each log, so when you mix them into syslog you won't be able to tell them apart) so you have three basic options 1. let logger do it's default thing and then use a formatting command to strip off the 'syslogie' parts to get back to the apache default in the file 2. leave the 'syslogie' parts in when you write it to a file and have your analysis tool strip them out 3. reformat the apache log message so that you put useful information in the 'syslogie' parts of the message. you can move the timestamp to the beginning (you can do this with or without the timezone, the format obviously differs slightly) you can put the name of the virtual host in the server field you can replace 'logger:' with something like apache[80]: or apache[443]: I am going to be setting up something along the lines of #3 in the next few weeks. I figure I will also want to tinker with other things in the log message. there are items that apache can log, but does not log by default (I believe that how long it took to process the request is one of these), and also since syslog defaults to limiting log messages to 1-2K (depending on your impementation), there are some fields that I would want to move late in the message so that if they get very long they don't cause other fields to be lost due to truncation (URL and referrer fields can be several K long by themselves) David Lang From vanderleeden at logicunited.com Thu Nov 26 13:57:54 2009 From: vanderleeden at logicunited.com (Rudolf van der Leeden) Date: Thu, 26 Nov 2009 13:57:54 +0100 Subject: [rsyslog] Indefinite number of retries with ompgsql Message-ID: Hi, I'm currently testing ompgsql and running into a problem with the number of pgsql retries if the INSERT statement is wrong. My rsyslog.conf setting: $ModLoad ompgsql # load the output driver for PostgreSQL $ModLoad imudp # network reception $MaxMessageSize 8192 # default 2048; use 32768 to support large message sizes for IHE $UDPServerRun 514 # start a udp server at port 514 $WorkDirectory /var/spool/rsyslog/work # default location for work (spool) files $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName dbq # set file name, also enables disk mode $ActionResumeRetryCount 0 $template applogFormat,"INSERT INTO applog (message, host, priority, priotext, tag, logtime) VALUES \ (E'%msg%', '%hostname%', '%syslogpriority%', '%syslogpriority-text %', '%programname%', '%timegenerated:::date-pgsql%')",stdsql if $programname contains 'production' then :ompgsql:loghost,syslog,syslog,syslog;applogFormat I changed $ActionResumeRetryCount from -1 to 1 and then to 0, but always the same result: indefinite retries of rsyslogd to get the INSERT done. This setup is basically running OK. Only if a binary 0 (\000) is contained in the message Postgres returns: ERROR: invalid byte sequence for encoding "UTF8": 0x00 What am I doing wrong? Any hints are welcome. Thanks for your help, Rudolf. Rudolf VanderLeeden Scoreloop GmbH vanderleeden at scoreloop.com From rgerhards at hq.adiscon.com Thu Nov 26 17:47:26 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 26 Nov 2009 17:47:26 +0100 Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710346E@GRFEXC.intern.adiscon.com> References: <003001ca6092$792833d0$6b789b70$@com> <9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710346E@GRFEXC.intern.adiscon.com> Message-ID: <1259254046.16407.1.camel@rgf11> Hi all, I have now integrated the TLS part of the patch into all relevant versions. It is available in public git now. I will try to look into the parsing part tomorrow. Thanks again to Jonathan for sharing it! Rainer On Fri, 2009-11-20 at 11:03 +0100, Rainer Gerhards wrote: > I, too, think this is useful. Will look at it as soon as I get the patch. The > b) and c) parts may be a bit harder to integrate if they are not for the > newest devel branch, as I rewrote the parser part of rsyslog. But in any > case, it is worth trying to merge it in. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Thursday, November 19, 2009 9:38 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] [PATCH] TLS connection check + syslog TAG > > parsing > > > > for what it's worth, I think that b and c are very useful when dealing > > with strange data sources, and a is probably a bugfix. > > > > David Lang > > > > On Thu, 19 Nov 2009, Rainer Gerhards wrote: > > > > > > > > Sorry, I seem to have overlooked that message. But the mailing list > > processor > > > has stripped the attachment. Could you please re-send to my personal > > mail > > > address. > > > > > > Rainer > > > > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com > > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > >> Jonathan Bond-Caron > > >> Sent: Sunday, November 08, 2009 5:43 PM > > >> To: rsyslog at lists.adiscon.com > > >> Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing > > >> > > >> Hi all, > > >> > > >> > > >> > > >> Attached is a patch for 3 things: > > >> > > >> > > >> > > >> a) Check that TCP connection is still alive when using TLS > > >> > > >> > > >> > > >> b) Improve TAG parsing so that it ends when it sees a > > >> tab or escape > > >> control character. > > >> > > >> > > >> > > >> c) Added configuration directive: escapecontrolcharactertab > > >> > > >> In rsyslog.conf, you can set: > > >> > > >> $escapeControlCharactersOnReceive on > > >> > > >> $escapeControlCharacterTab off > > >> > > >> > > >> > > >> This avoids having a tab being escaped since it is after a viewable > > >> character J > > >> > > >> I'd prefer this being the default behavior but I've left > > >> $escapeControlCharacterTab on as default in case people > > >> expect tabs to be > > >> escaped. > > >> > > >> > > >> > > >> This is my first GIT patch, hope it works well ;) > > >> > > >> > > >> > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From josesan311 at yahoo.com Fri Nov 27 01:25:22 2009 From: josesan311 at yahoo.com (Jose Sanchez) Date: Thu, 26 Nov 2009 16:25:22 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: Message-ID: <886471.94508.qm@web35606.mail.mud.yahoo.com> Hello, I appreciate all the responses. Im not sure how can I can acconplish options 1) and 2) automatically. For option 3) the thing is I need "combined" log type so I cannot reform this. Im trying to centralize an access_log file from one server to the rsyslog server and I need to completely remove the tags I mentioned on my previous post. I have also tried using a perl script mentioned at the botton of this email, but it salso arriving with a tag, "apache_syslog:" as showed below, "apache_syslog: XXX.XXX.XXX.XXX - - [26/Nov/2009:18:23:02 -0600] \"GET /.." Basically, this log will be parsed by awstats which is pretty much stricted with the log format so that's why I need a clean log sent from the apache server to the rsyslog server. Thank you very much for all the help. Below is the Perl script: #!/usr/local/bin/perl # script: apache-access-logger use Sys::Syslog; $SERVER_NAME = shift || ''; $PRIORITY = 'info'; $FACILITY = 'local1'; Sys::Syslog::setlogsock('unix'); openlog ($SERVER_NAME,'ndelay', $FACILITY); while (<>) { chomp; syslog($PRIORITY,$_); } closelog; --- On Thu, 11/26/09, david at lang.hm wrote: > From: david at lang.hm > Subject: Re: [rsyslog] filter logger tags from syslog > To: "rsyslog-users" > Date: Thursday, November 26, 2009, 2:21 AM > On Wed, 25 Nov 2009, Jose Sanchez > wrote: > > > Hello, > > > > I've been using classic syslog for centralizing apache > access logs from one server to a remote syslog server, the > thing is syslog adds some nasty tags before the lines in the > access logs and I cant get them off, ie: > > > > "Nov 25 21:25:37 server1 logger:" > > > > I would like to know if rsyslog has the option to > filter this kind of stuff, I just want to have the logs sent > to the syslog server exactly like I was saving them in a > local access.log file. > > > > Thanks in advance. > > 'logger:' is added by the logger program that apache is > using to send the > logs to syslog. > > a properly formatted syslog message will include a > timestamp and what > server it came from (note that the apache logs do _not_ > tell you what > virtual server the log comes from, it usually uses a > different file for > each log, so when you mix them into syslog you won't be > able to tell them > apart) > > so you have three basic options > > 1. let logger do it's default thing and then use a > formatting command to > strip off the 'syslogie' parts to get back to the apache > default in the > file > > 2. leave the 'syslogie' parts in when you write it to a > file and have your > analysis tool strip them out > > 3. reformat the apache log message so that you put useful > information in > the 'syslogie' parts of the message. > > you can move the timestamp to the beginning (you can do > this with or > without the timezone, the format obviously differs > slightly) > > you can put the name of the virtual host in the server > field > > you can replace 'logger:' with something like apache[80]: > or apache[443]: > > I am going to be setting up something along the lines of #3 > in the next > few weeks. I figure I will also want to tinker with other > things in the > log message. there are items that apache can log, but does > not log by > default (I believe that how long it took to process the > request is one of > these), and also since syslog defaults to limiting log > messages to 1-2K > (depending on your impementation), there are some fields > that I would want > to move late in the message so that if they get very long > they don't cause > other fields to be lost due to truncation (URL and referrer > fields can be > several K long by themselves) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From josesan311 at yahoo.com Fri Nov 27 01:26:24 2009 From: josesan311 at yahoo.com (Jose Sanchez) Date: Thu, 26 Nov 2009 16:26:24 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71034AC@GRFEXC.intern.adiscon.com> Message-ID: <840944.77123.qm@web35608.mail.mud.yahoo.com> Hello, Will this work if Im using rsyslog just at the server side? (client is using syslog) Many thanks. --- On Thu, 11/26/09, Rainer Gerhards wrote: > From: Rainer Gerhards > Subject: Re: [rsyslog] filter logger tags from syslog > To: "rsyslog-users" > Date: Thursday, November 26, 2009, 2:11 AM > I don't have the specifics at hand, > but you can surely do that. I would even > assume that if you just write the %msg% property to file, > the infomation > should look good. If not, you need to fiddle a bit with the > property > replacer. > > The basic idea is to use > > $template line,"%msg%\n" > *.* /path/to/log;line > > HTH > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com > > > [mailto:rsyslog-bounces at lists.adiscon.com] > On Behalf Of Jose Sanchez > > Sent: Thursday, November 26, 2009 5:23 AM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] filter logger tags from syslog > > > > Hello, > > > > I've been using classic syslog for centralizing apache > access > > logs from one server to a remote syslog server, the > thing is > > syslog adds some nasty tags before the lines in the > access > > logs and I cant get them off, ie: > > > > "Nov 25 21:25:37 server1 logger:" > > > > I would like to know if rsyslog has the option to > filter this > > kind of stuff, I just want to have the logs sent to > the > > syslog server exactly like I was saving them in a > local > > access.log file. > > > > Thanks in advance. > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Nov 27 01:30:52 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 26 Nov 2009 16:30:52 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <840944.77123.qm@web35608.mail.mud.yahoo.com> References: <840944.77123.qm@web35608.mail.mud.yahoo.com> Message-ID: On Thu, 26 Nov 2009, Jose Sanchez wrote: > Will this work if Im using rsyslog just at the server side? (client is using syslog) yes, although depending on exactly which syslog you may need to tinker with the template. try what Rainer suggested below and see what you get out. David Lang > Many thanks. > > --- On Thu, 11/26/09, Rainer Gerhards wrote: > >> From: Rainer Gerhards >> Subject: Re: [rsyslog] filter logger tags from syslog >> To: "rsyslog-users" >> Date: Thursday, November 26, 2009, 2:11 AM >> I don't have the specifics at hand, >> but you can surely do that. I would even >> assume that if you just write the %msg% property to file, >> the infomation >> should look good. If not, you need to fiddle a bit with the >> property >> replacer. >> >> The basic idea is to use >> >> $template line,"%msg%\n" >> *.* /path/to/log;line >> >> HTH >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com >> >>> [mailto:rsyslog-bounces at lists.adiscon.com] >> On Behalf Of Jose Sanchez >>> Sent: Thursday, November 26, 2009 5:23 AM >>> To: rsyslog at lists.adiscon.com >>> Subject: [rsyslog] filter logger tags from syslog >>> >>> Hello, >>> >>> I've been using classic syslog for centralizing apache >> access >>> logs from one server to a remote syslog server, the >> thing is >>> syslog adds some nasty tags before the lines in the >> access >>> logs and I cant get them off, ie: >>> >>> "Nov 25 21:25:37 server1 logger:" >>> >>> I would like to know if rsyslog has the option to >> filter this >>> kind of stuff, I just want to have the logs sent to >> the >>> syslog server exactly like I was saving them in a >> local >>> access.log file. >>> >>> Thanks in advance. >>> >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Nov 27 01:38:03 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 26 Nov 2009 16:38:03 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <886471.94508.qm@web35606.mail.mud.yahoo.com> References: <886471.94508.qm@web35606.mail.mud.yahoo.com> Message-ID: On Thu, 26 Nov 2009, Jose Sanchez wrote: > Hello, > > I appreciate all the responses. > Im not sure how can I can acconplish options 1) and 2) automatically. > For option 3) the thing is I need "combined" log type so I cannot reform this. > > Im trying to centralize an access_log file from one server to the rsyslog server and I need to completely remove the tags I mentioned on my previous post. > I have also tried using a perl script mentioned at the botton of this email, but it salso arriving with a tag, "apache_syslog:" as showed below, > > "apache_syslog: XXX.XXX.XXX.XXX - - [26/Nov/2009:18:23:02 -0600] \"GET /.." > > Basically, this log will be parsed by awstats which is pretty much stricted with the log format so that's why I need a clean log sent from the apache server to the rsyslog server. don't forget that you need to filter these messages into a seperate file, otherwise you will have your apache combined log messages mixed with other syslog messages (which will really confuse awstats) option 1 is what Rainer suggested option 2 is to run the log through another step before awstats runs, something along the lines of cut -c 16- file |cut -f 3- -d ' ' |awstats the first cut removes the timestamp (always 15 characters, but with a variable number of spaces in it), the second cut removes the servername and the syslog tag ('logger:' in your first example) David Lang > Thank you very much for all the help. > > Below is the Perl script: > > #!/usr/local/bin/perl > # script: apache-access-logger > > use Sys::Syslog; > $SERVER_NAME = shift || ''; > > $PRIORITY = 'info'; > $FACILITY = 'local1'; > > Sys::Syslog::setlogsock('unix'); > openlog ($SERVER_NAME,'ndelay', $FACILITY); > > while (<>) { > chomp; > syslog($PRIORITY,$_); > } > closelog; > > --- On Thu, 11/26/09, david at lang.hm wrote: > >> From: david at lang.hm >> Subject: Re: [rsyslog] filter logger tags from syslog >> To: "rsyslog-users" >> Date: Thursday, November 26, 2009, 2:21 AM >> On Wed, 25 Nov 2009, Jose Sanchez >> wrote: >> >>> Hello, >>> >>> I've been using classic syslog for centralizing apache >> access logs from one server to a remote syslog server, the >> thing is syslog adds some nasty tags before the lines in the >> access logs and I cant get them off, ie: >>> >>> "Nov 25 21:25:37 server1 logger:" >>> >>> I would like to know if rsyslog has the option to >> filter this kind of stuff, I just want to have the logs sent >> to the syslog server exactly like I was saving them in a >> local access.log file. >>> >>> Thanks in advance. >> >> 'logger:' is added by the logger program that apache is >> using to send the >> logs to syslog. >> >> a properly formatted syslog message will include a >> timestamp and what >> server it came from (note that the apache logs do _not_ >> tell you what >> virtual server the log comes from, it usually uses a >> different file for >> each log, so when you mix them into syslog you won't be >> able to tell them >> apart) >> >> so you have three basic options >> >> 1. let logger do it's default thing and then use a >> formatting command to >> strip off the 'syslogie' parts to get back to the apache >> default in the >> file >> >> 2. leave the 'syslogie' parts in when you write it to a >> file and have your >> analysis tool strip them out >> >> 3. reformat the apache log message so that you put useful >> information in >> the 'syslogie' parts of the message. >> >> you can move the timestamp to the beginning (you can do >> this with or >> without the timezone, the format obviously differs >> slightly) >> >> you can put the name of the virtual host in the server >> field >> >> you can replace 'logger:' with something like apache[80]: >> or apache[443]: >> >> I am going to be setting up something along the lines of #3 >> in the next >> few weeks. I figure I will also want to tinker with other >> things in the >> log message. there are items that apache can log, but does >> not log by >> default (I believe that how long it took to process the >> request is one of >> these), and also since syslog defaults to limiting log >> messages to 1-2K >> (depending on your impementation), there are some fields >> that I would want >> to move late in the message so that if they get very long >> they don't cause >> other fields to be lost due to truncation (URL and referrer >> fields can be >> several K long by themselves) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From josesan311 at yahoo.com Fri Nov 27 05:13:00 2009 From: josesan311 at yahoo.com (Jose Sanchez) Date: Thu, 26 Nov 2009 20:13:00 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: Message-ID: <925071.2214.qm@web35604.mail.mud.yahoo.com> Hello, Thanks again for the great response. It's actually working! rsyslog is removing the "logger:" thing and all the nasty stuff from it automatically, how come? Is it because we are not adding any tag in the template? Im still not understanding how rsyslog removes the logger thing. Ok, Im now getting the proper output and like David said Im now getting issues with filtering the apache logs with all the rsyslog messages. I've tried to use the following filter but for some reason is not working and Im not 100% if this is the best solution to use, This is how I had set it up, $template line,"%msg%\n" if $msg contains 'GET' then /var/log/apache.test.log;line *.* /var/log/test.log;line Not sure if Im on the right path, any help will be appreciated. I have also tried the "if" sentence without specifying the template name. Many Thanks. --- On Thu, 11/26/09, david at lang.hm wrote: > From: david at lang.hm > Subject: Re: [rsyslog] filter logger tags from syslog > To: "rsyslog-users" > Date: Thursday, November 26, 2009, 6:38 PM > On Thu, 26 Nov 2009, Jose Sanchez > wrote: > > > Hello, > > > > I appreciate all the responses. > > Im not sure how can I can acconplish options 1) and 2) > automatically. > > For option 3) the thing is I need "combined" log type > so I cannot reform this. > > > > Im trying to centralize an access_log file from one > server to the rsyslog server and I need to completely remove > the tags I mentioned on my previous post. > > I have also tried using a perl script mentioned at the > botton of this email, but it salso arriving with a tag, > "apache_syslog:" as showed below, > > > > "apache_syslog: XXX.XXX.XXX.XXX - - > [26/Nov/2009:18:23:02 -0600] \"GET /.." > > > > Basically, this log will be parsed by awstats which is > pretty much stricted with the log format so that's why I > need a clean log sent from the apache server to the rsyslog > server. > > don't forget that you need to filter these messages into a > seperate file, > otherwise you will have your apache combined log messages > mixed with other > syslog messages (which will really confuse awstats) > > option 1 is what Rainer suggested > > option 2 is to run the log through another step before > awstats runs, > something along the lines of > > cut -c 16- file |cut -f 3- -d ' ' |awstats > > the first cut removes the timestamp (always 15 characters, > but with a > variable number of spaces in it), the second cut removes > the servername > and the syslog tag ('logger:' in your first example) > > David Lang > > > Thank you very much for all the help. > > > > Below is the Perl script: > > > > #!/usr/local/bin/perl > > # script: apache-access-logger > > > > use Sys::Syslog; > > $SERVER_NAME = shift || ''; > > > > $PRIORITY = 'info'; > > $FACILITY = 'local1'; > > > > Sys::Syslog::setlogsock('unix'); > > openlog ($SERVER_NAME,'ndelay', $FACILITY); > > > > while (<>) { > >? chomp; > >? syslog($PRIORITY,$_); > > } > > closelog; > > > > --- On Thu, 11/26/09, david at lang.hm > wrote: > > > >> From: david at lang.hm > >> Subject: Re: [rsyslog] filter logger tags from > syslog > >> To: "rsyslog-users" > >> Date: Thursday, November 26, 2009, 2:21 AM > >> On Wed, 25 Nov 2009, Jose Sanchez > >> wrote: > >> > >>> Hello, > >>> > >>> I've been using classic syslog for > centralizing apache > >> access logs from one server to a remote syslog > server, the > >> thing is syslog adds some nasty tags before the > lines in the > >> access logs and I cant get them off, ie: > >>> > >>> "Nov 25 21:25:37 server1 logger:" > >>> > >>> I would like to know if rsyslog has the option > to > >> filter this kind of stuff, I just want to have the > logs sent > >> to the syslog server exactly like I was saving > them in a > >> local access.log file. > >>> > >>> Thanks in advance. > >> > >> 'logger:' is added by the logger program that > apache is > >> using to send the > >> logs to syslog. > >> > >> a properly formatted syslog message will include > a > >> timestamp and what > >> server it came from (note that the apache logs do > _not_ > >> tell you what > >> virtual server the log comes from, it usually uses > a > >> different file for > >> each log, so when you mix them into syslog you > won't be > >> able to tell them > >> apart) > >> > >> so you have three basic options > >> > >> 1. let logger do it's default thing and then use > a > >> formatting command to > >> strip off the 'syslogie' parts to get back to the > apache > >> default in the > >> file > >> > >> 2. leave the 'syslogie' parts in when you write it > to a > >> file and have your > >> analysis tool strip them out > >> > >> 3. reformat the apache log message so that you put > useful > >> information in > >> the 'syslogie' parts of the message. > >> > >> you can move the timestamp to the beginning (you > can do > >> this with or > >> without the timezone, the format obviously > differs > >> slightly) > >> > >> you can put the name of the virtual host in the > server > >> field > >> > >> you can replace 'logger:' with something like > apache[80]: > >> or apache[443]: > >> > >> I am going to be setting up something along the > lines of #3 > >> in the next > >> few weeks. I figure I will also want to tinker > with other > >> things in the > >> log message. there are items that apache can log, > but does > >> not log by > >> default (I believe that how long it took to > process the > >> request is one of > >> these), and also since syslog defaults to limiting > log > >> messages to 1-2K > >> (depending on your impementation), there are some > fields > >> that I would want > >> to move late in the message so that if they get > very long > >> they don't cause > >> other fields to be lost due to truncation (URL and > referrer > >> fields can be > >> several K long by themselves) > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Nov 27 09:41:13 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 27 Nov 2009 09:41:13 +0100 Subject: [rsyslog] filter logger tags from syslog References: <925071.2214.qm@web35604.mail.mud.yahoo.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034C3@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Jose Sanchez > Sent: Friday, November 27, 2009 5:13 AM > To: rsyslog-users > Subject: Re: [rsyslog] filter logger tags from syslog > > Hello, > > Thanks again for the great response. > It's actually working! rsyslog is removing the "logger:" thing and all > the nasty stuff from it automatically, how come? you really need to read through the property doc. I know the doc is not great, but with some persistence you'll find anything you need (and doc contributions are always very welcome!). Understanding properties is key to getting complex configurations right. > Is it because we are > not adding any tag in the template? Yes, because the template specifies which properties are used (and how). > Im still not understanding how > rsyslog removes the logger thing. > > Ok, Im now getting the proper output and like David said Im now getting > issues with filtering the apache logs with all the rsyslog messages. > I've tried to use the following filter but for some reason is not > working and Im not 100% if this is the best solution to use, > > This is how I had set it up, > > $template line,"%msg%\n" > if $msg contains 'GET' then /var/log/apache.test.log;line > *.* /var/log/test.log;line If you want to split the messages, you need to discard the one that you just wrote: if $msg contains 'GET' then /var/log/apache.test.log;line & ~ Helpful read is http://www.rsyslog.com/doc-queues_analogy.html HTH Rainer > > Not sure if Im on the right path, any help will be appreciated. > I have also tried the "if" sentence without specifying the template > name. > > Many Thanks. > > --- On Thu, 11/26/09, david at lang.hm wrote: > > > From: david at lang.hm > > Subject: Re: [rsyslog] filter logger tags from syslog > > To: "rsyslog-users" > > Date: Thursday, November 26, 2009, 6:38 PM > > On Thu, 26 Nov 2009, Jose Sanchez > > wrote: > > > > > Hello, > > > > > > I appreciate all the responses. > > > Im not sure how can I can acconplish options 1) and 2) > > automatically. > > > For option 3) the thing is I need "combined" log type > > so I cannot reform this. > > > > > > Im trying to centralize an access_log file from one > > server to the rsyslog server and I need to completely remove > > the tags I mentioned on my previous post. > > > I have also tried using a perl script mentioned at the > > botton of this email, but it salso arriving with a tag, > > "apache_syslog:" as showed below, > > > > > > "apache_syslog: XXX.XXX.XXX.XXX - - > > [26/Nov/2009:18:23:02 -0600] \"GET /.." > > > > > > Basically, this log will be parsed by awstats which is > > pretty much stricted with the log format so that's why I > > need a clean log sent from the apache server to the rsyslog > > server. > > > > don't forget that you need to filter these messages into a > > seperate file, > > otherwise you will have your apache combined log messages > > mixed with other > > syslog messages (which will really confuse awstats) > > > > option 1 is what Rainer suggested > > > > option 2 is to run the log through another step before > > awstats runs, > > something along the lines of > > > > cut -c 16- file |cut -f 3- -d ' ' |awstats > > > > the first cut removes the timestamp (always 15 characters, > > but with a > > variable number of spaces in it), the second cut removes > > the servername > > and the syslog tag ('logger:' in your first example) > > > > David Lang > > > > > Thank you very much for all the help. > > > > > > Below is the Perl script: > > > > > > #!/usr/local/bin/perl > > > # script: apache-access-logger > > > > > > use Sys::Syslog; > > > $SERVER_NAME = shift || ''; > > > > > > $PRIORITY = 'info'; > > > $FACILITY = 'local1'; > > > > > > Sys::Syslog::setlogsock('unix'); > > > openlog ($SERVER_NAME,'ndelay', $FACILITY); > > > > > > while (<>) { > > >? chomp; > > >? syslog($PRIORITY,$_); > > > } > > > closelog; > > > > > > --- On Thu, 11/26/09, david at lang.hm > > wrote: > > > > > >> From: david at lang.hm > > >> Subject: Re: [rsyslog] filter logger tags from > > syslog > > >> To: "rsyslog-users" > > >> Date: Thursday, November 26, 2009, 2:21 AM > > >> On Wed, 25 Nov 2009, Jose Sanchez > > >> wrote: > > >> > > >>> Hello, > > >>> > > >>> I've been using classic syslog for > > centralizing apache > > >> access logs from one server to a remote syslog > > server, the > > >> thing is syslog adds some nasty tags before the > > lines in the > > >> access logs and I cant get them off, ie: > > >>> > > >>> "Nov 25 21:25:37 server1 logger:" > > >>> > > >>> I would like to know if rsyslog has the option > > to > > >> filter this kind of stuff, I just want to have the > > logs sent > > >> to the syslog server exactly like I was saving > > them in a > > >> local access.log file. > > >>> > > >>> Thanks in advance. > > >> > > >> 'logger:' is added by the logger program that > > apache is > > >> using to send the > > >> logs to syslog. > > >> > > >> a properly formatted syslog message will include > > a > > >> timestamp and what > > >> server it came from (note that the apache logs do > > _not_ > > >> tell you what > > >> virtual server the log comes from, it usually uses > > a > > >> different file for > > >> each log, so when you mix them into syslog you > > won't be > > >> able to tell them > > >> apart) > > >> > > >> so you have three basic options > > >> > > >> 1. let logger do it's default thing and then use > > a > > >> formatting command to > > >> strip off the 'syslogie' parts to get back to the > > apache > > >> default in the > > >> file > > >> > > >> 2. leave the 'syslogie' parts in when you write it > > to a > > >> file and have your > > >> analysis tool strip them out > > >> > > >> 3. reformat the apache log message so that you put > > useful > > >> information in > > >> the 'syslogie' parts of the message. > > >> > > >> you can move the timestamp to the beginning (you > > can do > > >> this with or > > >> without the timezone, the format obviously > > differs > > >> slightly) > > >> > > >> you can put the name of the virtual host in the > > server > > >> field > > >> > > >> you can replace 'logger:' with something like > > apache[80]: > > >> or apache[443]: > > >> > > >> I am going to be setting up something along the > > lines of #3 > > >> in the next > > >> few weeks. I figure I will also want to tinker > > with other > > >> things in the > > >> log message. there are items that apache can log, > > but does > > >> not log by > > >> default (I believe that how long it took to > > process the > > >> request is one of > > >> these), and also since syslog defaults to limiting > > log > > >> messages to 1-2K > > >> (depending on your impementation), there are some > > fields > > >> that I would want > > >> to move late in the message so that if they get > > very long > > >> they don't cause > > >> other fields to be lost due to truncation (URL and > > referrer > > >> fields can be > > >> several K long by themselves) > > >> > > >> David Lang > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Nov 27 09:49:35 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 27 Nov 2009 00:49:35 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <925071.2214.qm@web35604.mail.mud.yahoo.com> References: <925071.2214.qm@web35604.mail.mud.yahoo.com> Message-ID: On Thu, 26 Nov 2009, Jose Sanchez wrote: > Hello, > > Thanks again for the great response. > It's actually working! rsyslog is removing the "logger:" thing and all > the nasty stuff from it automatically, how come? Is it because we are > not adding any tag in the template? Im still not understanding how > rsyslog removes the logger thing. > > Ok, Im now getting the proper output and like David said Im now getting > issues with filtering the apache logs with all the rsyslog messages. > I've tried to use the following filter but for some reason is not > working and Im not 100% if this is the best solution to use, > > This is how I had set it up, > > $template line,"%msg%\n" > if $msg contains 'GET' then /var/log/apache.test.log;line > *.* /var/log/test.log;line > > Not sure if Im on the right path, any help will be appreciated. > I have also tried the "if" sentence without specifying the template name. when rsyslog receives a message it parses it. the message over the wire hasn't changed (still has the timestamp, servername, logger: etc), but rsyslog puts those parts into the seperate variables and puts what is left of the message into the %msg% variable. so when you change the template from the default of %timestamp% %hostname% %syslogtag%%msg% to just %msg% the log file has just the part you care about. now for the filtering. you could do :%programname, isequal, "logger" /var/log/apache.test.log;line (as I understand it, this format is a bit more efficiant for rsyslog than the equivalent of if $programname eq "logger" then /var/log/apache.test.log;line ) I would actually suggest that you use the perl script that you posted, and filter for programname equal to "apache_syslog", filtering on just 'logger' means that you can't use logger for anything else. you don't want to just filter for 'GET' as there are a bunch of log files that won't have GET in them David Lang > Many Thanks. > > --- On Thu, 11/26/09, david at lang.hm wrote: > >> From: david at lang.hm >> Subject: Re: [rsyslog] filter logger tags from syslog >> To: "rsyslog-users" >> Date: Thursday, November 26, 2009, 6:38 PM >> On Thu, 26 Nov 2009, Jose Sanchez >> wrote: >> >>> Hello, >>> >>> I appreciate all the responses. >>> Im not sure how can I can acconplish options 1) and 2) >> automatically. >>> For option 3) the thing is I need "combined" log type >> so I cannot reform this. >>> >>> Im trying to centralize an access_log file from one >> server to the rsyslog server and I need to completely remove >> the tags I mentioned on my previous post. >>> I have also tried using a perl script mentioned at the >> botton of this email, but it salso arriving with a tag, >> "apache_syslog:" as showed below, >>> >>> "apache_syslog: XXX.XXX.XXX.XXX - - >> [26/Nov/2009:18:23:02 -0600] \"GET /.." >>> >>> Basically, this log will be parsed by awstats which is >> pretty much stricted with the log format so that's why I >> need a clean log sent from the apache server to the rsyslog >> server. >> >> don't forget that you need to filter these messages into a >> seperate file, >> otherwise you will have your apache combined log messages >> mixed with other >> syslog messages (which will really confuse awstats) >> >> option 1 is what Rainer suggested >> >> option 2 is to run the log through another step before >> awstats runs, >> something along the lines of >> >> cut -c 16- file |cut -f 3- -d ' ' |awstats >> >> the first cut removes the timestamp (always 15 characters, >> but with a >> variable number of spaces in it), the second cut removes >> the servername >> and the syslog tag ('logger:' in your first example) >> >> David Lang >> >>> Thank you very much for all the help. >>> >>> Below is the Perl script: >>> >>> #!/usr/local/bin/perl >>> # script: apache-access-logger >>> >>> use Sys::Syslog; >>> $SERVER_NAME = shift || ''; >>> >>> $PRIORITY = 'info'; >>> $FACILITY = 'local1'; >>> >>> Sys::Syslog::setlogsock('unix'); >>> openlog ($SERVER_NAME,'ndelay', $FACILITY); >>> >>> while (<>) { >>> ? chomp; >>> ? syslog($PRIORITY,$_); >>> } >>> closelog; >>> >>> --- On Thu, 11/26/09, david at lang.hm >> wrote: >>> >>>> From: david at lang.hm >>>> Subject: Re: [rsyslog] filter logger tags from >> syslog >>>> To: "rsyslog-users" >>>> Date: Thursday, November 26, 2009, 2:21 AM >>>> On Wed, 25 Nov 2009, Jose Sanchez >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> I've been using classic syslog for >> centralizing apache >>>> access logs from one server to a remote syslog >> server, the >>>> thing is syslog adds some nasty tags before the >> lines in the >>>> access logs and I cant get them off, ie: >>>>> >>>>> "Nov 25 21:25:37 server1 logger:" >>>>> >>>>> I would like to know if rsyslog has the option >> to >>>> filter this kind of stuff, I just want to have the >> logs sent >>>> to the syslog server exactly like I was saving >> them in a >>>> local access.log file. >>>>> >>>>> Thanks in advance. >>>> >>>> 'logger:' is added by the logger program that >> apache is >>>> using to send the >>>> logs to syslog. >>>> >>>> a properly formatted syslog message will include >> a >>>> timestamp and what >>>> server it came from (note that the apache logs do >> _not_ >>>> tell you what >>>> virtual server the log comes from, it usually uses >> a >>>> different file for >>>> each log, so when you mix them into syslog you >> won't be >>>> able to tell them >>>> apart) >>>> >>>> so you have three basic options >>>> >>>> 1. let logger do it's default thing and then use >> a >>>> formatting command to >>>> strip off the 'syslogie' parts to get back to the >> apache >>>> default in the >>>> file >>>> >>>> 2. leave the 'syslogie' parts in when you write it >> to a >>>> file and have your >>>> analysis tool strip them out >>>> >>>> 3. reformat the apache log message so that you put >> useful >>>> information in >>>> the 'syslogie' parts of the message. >>>> >>>> you can move the timestamp to the beginning (you >> can do >>>> this with or >>>> without the timezone, the format obviously >> differs >>>> slightly) >>>> >>>> you can put the name of the virtual host in the >> server >>>> field >>>> >>>> you can replace 'logger:' with something like >> apache[80]: >>>> or apache[443]: >>>> >>>> I am going to be setting up something along the >> lines of #3 >>>> in the next >>>> few weeks. I figure I will also want to tinker >> with other >>>> things in the >>>> log message. there are items that apache can log, >> but does >>>> not log by >>>> default (I believe that how long it took to >> process the >>>> request is one of >>>> these), and also since syslog defaults to limiting >> log >>>> messages to 1-2K >>>> (depending on your impementation), there are some >> fields >>>> that I would want >>>> to move late in the message so that if they get >> very long >>>> they don't cause >>>> other fields to be lost due to truncation (URL and >> referrer >>>> fields can be >>>> several K long by themselves) >>>> >>>> David Lang >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Nov 27 11:46:03 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 27 Nov 2009 11:46:03 +0100 Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710346E@GRFEXC.intern.adiscon.com> References: <003001ca6092$792833d0$6b789b70$@com> <9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710346E@GRFEXC.intern.adiscon.com> Message-ID: <1259318763.15106.1.camel@rgf11> I have now implemented the complete patch. For v5, I needed to rewrite it, but that wasn't much of an issue once I knew what was required ;) v5 now even has an automatted test for this functionality. Thanks again, Rainer On Fri, 2009-11-20 at 11:03 +0100, Rainer Gerhards wrote: > I, too, think this is useful. Will look at it as soon as I get the patch. The > b) and c) parts may be a bit harder to integrate if they are not for the > newest devel branch, as I rewrote the parser part of rsyslog. But in any > case, it is worth trying to merge it in. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Thursday, November 19, 2009 9:38 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] [PATCH] TLS connection check + syslog TAG > > parsing > > > > for what it's worth, I think that b and c are very useful when dealing > > with strange data sources, and a is probably a bugfix. > > > > David Lang > > > > On Thu, 19 Nov 2009, Rainer Gerhards wrote: > > > > > > > > Sorry, I seem to have overlooked that message. But the mailing list > > processor > > > has stripped the attachment. Could you please re-send to my personal > > mail > > > address. > > > > > > Rainer > > > > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com > > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > >> Jonathan Bond-Caron > > >> Sent: Sunday, November 08, 2009 5:43 PM > > >> To: rsyslog at lists.adiscon.com > > >> Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing > > >> > > >> Hi all, > > >> > > >> > > >> > > >> Attached is a patch for 3 things: > > >> > > >> > > >> > > >> a) Check that TCP connection is still alive when using TLS > > >> > > >> > > >> > > >> b) Improve TAG parsing so that it ends when it sees a > > >> tab or escape > > >> control character. > > >> > > >> > > >> > > >> c) Added configuration directive: escapecontrolcharactertab > > >> > > >> In rsyslog.conf, you can set: > > >> > > >> $escapeControlCharactersOnReceive on > > >> > > >> $escapeControlCharacterTab off > > >> > > >> > > >> > > >> This avoids having a tab being escaped since it is after a viewable > > >> character J > > >> > > >> I'd prefer this being the default behavior but I've left > > >> $escapeControlCharacterTab on as default in case people > > >> expect tabs to be > > >> escaped. > > >> > > >> > > >> > > >> This is my first GIT patch, hope it works well ;) > > >> > > >> > > >> > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From tbergfeld at hq.adiscon.com Fri Nov 27 12:08:17 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 27 Nov 2009 12:08:17 +0100 Subject: [rsyslog] rsyslog 5.5.1 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034CC@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.5.1, a member of the v5-devel branch. The primary new feature is that the epoll() interface is now supported for plain tcp syslog. This results in better performance and also removes the upper bound of connections in select() in a portable way. The select() interface is still supported for platforms that do not provide epoll(). There are also some important fixes. It is a recommended update for all users of the v5-beta. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article433.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-190.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From josesan311 at yahoo.com Sat Nov 28 01:42:02 2009 From: josesan311 at yahoo.com (Jose Sanchez) Date: Fri, 27 Nov 2009 16:42:02 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: Message-ID: <472628.6567.qm@web35606.mail.mud.yahoo.com> Hello David and Reiner, First I would like to thank you for all the help offered, I was able to setup almost everything because of you guys. I had some issues today, though. I found that rsyslog was removing the "logger" properly but it was adding an extra empty space not sure why so I had to cut if off (by watching how to do it on video tutorial first!) by modifying the template that David gave me, I currently have it like this, $template line,"%msg:2:1000%\n" The thing here is Im not sure if this is a reliable solution, I couldnt find if there is any setting that will tell rsyslog to simply remove the empty space or to get everything until the last letter so I configured a very long (1000) number in case rsyslog cuts some part of the text. Not sure if there is any negative impact on doing it this way, if there is any other better way, please let me know. Regarding David's reply, > I would actually suggest that you use the perl script that > you posted, and > filter for programname equal to "apache_syslog", filtering > on just > 'logger' means that you can't use logger for anything > else. I ended using logger because with logger I can specify tags when sending the logs and since I have multiple vhosts and two webserver nodes I could identify from which webserver is coming the logs from, ie: :programname, isequal, "node1.www.domain.com" /var/log/httpd/node1/www.domain.com.log;line :programname, isequal, "node1.blog.domain.com" /var/log/httpd/node1/blog.domain.com.log;line :programname, isequal, "node2.www.domain.com" /var/log/httpd/node2/www.domain.com.log;line :programname, isequal, "node2.blog.domain.com" /var/log/httpd/node2/blog.domain.com.log;line I spent like 3 hours adding those lines due the amount of vhosts I have anyway this created every log and started delivering every vhost for each node on their specific log file just fine. Same thing as above Im still not really sure if this is the most realiable way of doing this with rsyslog. I hope this is the right path for doing such a big thing. Thank you again for everything. --- On Fri, 11/27/09, david at lang.hm wrote: > From: david at lang.hm > Subject: Re: [rsyslog] filter logger tags from syslog > To: "rsyslog-users" > Date: Friday, November 27, 2009, 2:49 AM > On Thu, 26 Nov 2009, Jose Sanchez > wrote: > > > Hello, > > > > Thanks again for the great response. > > It's actually working! rsyslog is removing the > "logger:" thing and all > > the nasty stuff from it automatically, how come? Is it > because we are > > not adding any tag in the template? Im still not > understanding how > > rsyslog removes the logger thing. > > > > Ok, Im now getting the proper output and like David > said Im now getting > > issues with filtering the apache logs with all the > rsyslog messages. > > > I've tried to use the following filter but for some > reason is not > > working and Im not 100% if this is the best solution > to use, > > > > This is how I had set it up, > > > > $template line,"%msg%\n" > > if $msg contains 'GET' then > /var/log/apache.test.log;line > > *.* /var/log/test.log;line > > > > Not sure if Im on the right path, any help will be > appreciated. > > I have also tried the "if" sentence without specifying > the template name. > > > when rsyslog receives a message it parses it. the message > over the wire > hasn't changed (still has the timestamp, servername, > logger: etc), but > rsyslog puts those parts into the seperate variables and > puts what is left > of the message into the %msg% variable. > > so when you change the template from the default of > %timestamp% %hostname% %syslogtag%%msg% > to just > %msg% > > the log file has just the part you care about. > > now for the filtering. > > you could do > :%programname, isequal, "logger" > /var/log/apache.test.log;line > > (as I understand it, this format is a bit more efficiant > for rsyslog than > the equivalent of > if $programname eq "logger" then > /var/log/apache.test.log;line > ) > > I would actually suggest that you use the perl script that > you posted, and > filter for programname equal to "apache_syslog", filtering > on just > 'logger' means that you can't use logger for anything > else. > > you don't want to just filter for 'GET' as there are a > bunch of log files > that won't have GET in them > > David Lang > > > > Many Thanks. > > > > --- On Thu, 11/26/09, david at lang.hm > wrote: > > > >> From: david at lang.hm > >> Subject: Re: [rsyslog] filter logger tags from > syslog > >> To: "rsyslog-users" > >> Date: Thursday, November 26, 2009, 6:38 PM > >> On Thu, 26 Nov 2009, Jose Sanchez > >> wrote: > >> > >>> Hello, > >>> > >>> I appreciate all the responses. > >>> Im not sure how can I can acconplish options > 1) and 2) > >> automatically. > >>> For option 3) the thing is I need "combined" > log type > >> so I cannot reform this. > >>> > >>> Im trying to centralize an access_log file > from one > >> server to the rsyslog server and I need to > completely remove > >> the tags I mentioned on my previous post. > >>> I have also tried using a perl script > mentioned at the > >> botton of this email, but it salso arriving with a > tag, > >> "apache_syslog:" as showed below, > >>> > >>> "apache_syslog: XXX.XXX.XXX.XXX - - > >> [26/Nov/2009:18:23:02 -0600] \"GET /.." > >>> > >>> Basically, this log will be parsed by awstats > which is > >> pretty much stricted with the log format so that's > why I > >> need a clean log sent from the apache server to > the rsyslog > >> server. > >> > >> don't forget that you need to filter these > messages into a > >> seperate file, > >> otherwise you will have your apache combined log > messages > >> mixed with other > >> syslog messages (which will really confuse > awstats) > >> > >> option 1 is what Rainer suggested > >> > >> option 2 is to run the log through another step > before > >> awstats runs, > >> something along the lines of > >> > >> cut -c 16- file |cut -f 3- -d ' ' |awstats > >> > >> the first cut removes the timestamp (always 15 > characters, > >> but with a > >> variable number of spaces in it), the second cut > removes > >> the servername > >> and the syslog tag ('logger:' in your first > example) > >> > >> David Lang > >> > >>> Thank you very much for all the help. > >>> > >>> Below is the Perl script: > >>> > >>> #!/usr/local/bin/perl > >>> # script: apache-access-logger > >>> > >>> use Sys::Syslog; > >>> $SERVER_NAME = shift || ''; > >>> > >>> $PRIORITY = 'info'; > >>> $FACILITY = 'local1'; > >>> > >>> Sys::Syslog::setlogsock('unix'); > >>> openlog ($SERVER_NAME,'ndelay', $FACILITY); > >>> > >>> while (<>) { > >>> ? chomp; > >>> ? syslog($PRIORITY,$_); > >>> } > >>> closelog; > >>> > >>> --- On Thu, 11/26/09, david at lang.hm > >> wrote: > >>> > >>>> From: david at lang.hm > >>>> Subject: Re: [rsyslog] filter logger tags > from > >> syslog > >>>> To: "rsyslog-users" > >>>> Date: Thursday, November 26, 2009, 2:21 > AM > >>>> On Wed, 25 Nov 2009, Jose Sanchez > >>>> wrote: > >>>> > >>>>> Hello, > >>>>> > >>>>> I've been using classic syslog for > >> centralizing apache > >>>> access logs from one server to a remote > syslog > >> server, the > >>>> thing is syslog adds some nasty tags > before the > >> lines in the > >>>> access logs and I cant get them off, ie: > >>>>> > >>>>> "Nov 25 21:25:37 server1 logger:" > >>>>> > >>>>> I would like to know if rsyslog has > the option > >> to > >>>> filter this kind of stuff, I just want to > have the > >> logs sent > >>>> to the syslog server exactly like I was > saving > >> them in a > >>>> local access.log file. > >>>>> > >>>>> Thanks in advance. > >>>> > >>>> 'logger:' is added by the logger program > that > >> apache is > >>>> using to send the > >>>> logs to syslog. > >>>> > >>>> a properly formatted syslog message will > include > >> a > >>>> timestamp and what > >>>> server it came from (note that the apache > logs do > >> _not_ > >>>> tell you what > >>>> virtual server the log comes from, it > usually uses > >> a > >>>> different file for > >>>> each log, so when you mix them into syslog > you > >> won't be > >>>> able to tell them > >>>> apart) > >>>> > >>>> so you have three basic options > >>>> > >>>> 1. let logger do it's default thing and > then use > >> a > >>>> formatting command to > >>>> strip off the 'syslogie' parts to get back > to the > >> apache > >>>> default in the > >>>> file > >>>> > >>>> 2. leave the 'syslogie' parts in when you > write it > >> to a > >>>> file and have your > >>>> analysis tool strip them out > >>>> > >>>> 3. reformat the apache log message so that > you put > >> useful > >>>> information in > >>>> the 'syslogie' parts of the message. > >>>> > >>>> you can move the timestamp to the > beginning (you > >> can do > >>>> this with or > >>>> without the timezone, the format > obviously > >> differs > >>>> slightly) > >>>> > >>>> you can put the name of the virtual host > in the > >> server > >>>> field > >>>> > >>>> you can replace 'logger:' with something > like > >> apache[80]: > >>>> or apache[443]: > >>>> > >>>> I am going to be setting up something > along the > >> lines of #3 > >>>> in the next > >>>> few weeks. I figure I will also want to > tinker > >> with other > >>>> things in the > >>>> log message. there are items that apache > can log, > >> but does > >>>> not log by > >>>> default (I believe that how long it took > to > >> process the > >>>> request is one of > >>>> these), and also since syslog defaults to > limiting > >> log > >>>> messages to 1-2K > >>>> (depending on your impementation), there > are some > >> fields > >>>> that I would want > >>>> to move late in the message so that if > they get > >> very long > >>>> they don't cause > >>>> other fields to be lost due to truncation > (URL and > >> referrer > >>>> fields can be > >>>> several K long by themselves) > >>>> > >>>> David Lang > >>>> > _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> > _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > -----Inline Attachment Follows----- > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From ryan.b.lynch at gmail.com Sat Nov 28 05:45:42 2009 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Fri, 27 Nov 2009 23:45:42 -0500 Subject: [rsyslog] Auto-creating directories, when using templates and the property replacer. Message-ID: <115906d10911272045v237eada0n96b204fcaf1fa6d4@mail.gmail.com> I recently migrated a few Fedora and CentOS hosts from Rsyslog 4.x to 5.x. Several of my systems use output file path templates like this, with multiple dynamically-named directory components: `template DefaultOutputPath,"/var/log/rsyslog/%hostname:::secpath-replace%/%syslogfacility-text:::secpath-replace%/%programname:::secpath-replace%/%$year%/%$month%/%$day%/%$hour%/%$minute%/_.log"` Under 4.x, Rsyslog will automatically create any directories that don't already exist, as needed. But under 5.x, it just fails silently if any of the directories in the path don't exist. Is there a configuration setting that governs this in 5.x, or some way to get back the old automatic directory creation behavior? -Ryan Ryan B. Lynch ryan.b.lynch at gmail.com From rgerhards at hq.adiscon.com Sat Nov 28 09:52:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 28 Nov 2009 09:52:12 +0100 Subject: [rsyslog] Auto-creating directories, when using templates and the property replacer. Message-ID: <000101ca7008$230eb68e$100013ac@intern.adiscon.com> That sounds like a bug, will look into it hopefully early next week. Thanks for reporting! Oh: which version do you use? 5.2.0? Then you should try the recent beta first... rainer ----- Urspr?ngliche Nachricht ----- Von: "Ryan Lynch" An: "rsyslog at lists.adiscon.com" Gesendet: 28.11.09 05:46 Betreff: [rsyslog] Auto-creating directories,when using templates and the property replacer. I recently migrated a few Fedora and CentOS hosts from Rsyslog 4.x to 5.x. Several of my systems use output file path templates like this, with multiple dynamically-named directory components: `template DefaultOutputPath,"/var/log/rsyslog/%hostname:::secpath-replace%/%syslogfacility-text:::secpath-replace%/%programname:::secpath-replace%/%$year%/%$month%/%$day%/%$hour%/%$minute%/_.log"` Under 4.x, Rsyslog will automatically create any directories that don't already exist, as needed. But under 5.x, it just fails silently if any of the directories in the path don't exist. Is there a configuration setting that governs this in 5.x, or some way to get back the old automatic directory creation behavior? -Ryan Ryan B. Lynch ryan.b.lynch at gmail.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Sat Nov 28 10:43:03 2009 From: david at lang.hm (david at lang.hm) Date: Sat, 28 Nov 2009 01:43:03 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <472628.6567.qm@web35606.mail.mud.yahoo.com> References: <472628.6567.qm@web35606.mail.mud.yahoo.com> Message-ID: On Fri, 27 Nov 2009, Jose Sanchez wrote: > Hello David and Reiner, > > First I would like to thank you for all the help offered, I was able to setup almost everything because of you guys. > > I had some issues today, though. I found that rsyslog was removing the "logger" properly but it was adding an extra empty space not sure why so I had to cut if off (by watching how to do it on video tutorial first!) by modifying the template that David gave me, I currently have it like this, > > $template line,"%msg:2:1000%\n" > > The thing here is Im not sure if this is a reliable solution, I couldnt find if there is any setting that will tell rsyslog to simply remove the empty space or to get everything until the last letter so I configured a very long (1000) number in case rsyslog cuts some part of the text. Not sure if there is any negative impact on doing it this way, if there is any other better way, please let me know. if you use '$' instead of '1000' it will go to the end of the message, no matter how long it is (1000 is not long enough for some messages) I think that what you are doing is probably the best way to deal with this space. David Lang From ryan.b.lynch at gmail.com Sat Nov 28 11:08:17 2009 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Sat, 28 Nov 2009 05:08:17 -0500 Subject: [rsyslog] Auto-creating directories, when using templates and the property replacer. In-Reply-To: <000101ca7008$230eb68e$100013ac@intern.adiscon.com> References: <000101ca7008$230eb68e$100013ac@intern.adiscon.com> Message-ID: <115906d10911280208m3ee48754i1c755ebaee67a1d2@mail.gmail.com> On Sat, Nov 28, 2009 at 03:52, Rainer Gerhards wrote: > Oh: which version do you use? 5.2.0? Then you should try the recent beta first... I'm using Rsyslog v5.5.1, and Librelp v.0.1.3. -Ryan From rgerhards at hq.adiscon.com Mon Nov 30 11:38:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 30 Nov 2009 11:38:29 +0100 Subject: [rsyslog] Auto-creating directories, when using templates and the property replacer. References: <000101ca7008$230eb68e$100013ac@intern.adiscon.com> <115906d10911280208m3ee48754i1c755ebaee67a1d2@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034DB@GRFEXC.intern.adiscon.com> Thanks, I was able to reproduced the problem, will now look into it. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > Sent: Saturday, November 28, 2009 11:08 AM > To: rsyslog-users > Subject: Re: [rsyslog] Auto-creating directories,when using templates > and the property replacer. > > On Sat, Nov 28, 2009 at 03:52, Rainer Gerhards > wrote: > > Oh: which version do you use? 5.2.0? Then you should try the recent > beta first... > > I'm using Rsyslog v5.5.1, and Librelp v.0.1.3. > > -Ryan > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Nov 30 12:08:09 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 30 Nov 2009 12:08:09 +0100 Subject: [rsyslog] Auto-creating directories, when using templates and the property replacer. References: <000101ca7008$230eb68e$100013ac@intern.adiscon.com><115906d10911280208m3ee48754i1c755ebaee67a1d2@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71034DB@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034DC@GRFEXC.intern.adiscon.com> Actually, it looks like it was just an improperly initialized variable, so that the default for creating directories was most often "on" (as documented), but could randomly be "off" as well. This bug was hidden all the way back to v3 (I didn't bother to look at v2...). Simple solution, now merged into all branches. The patch for is http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=7b40604e9ae8a0948f17eafd 4299eeb7fb3356c2 Or probably better just pull the newest git version. I would appreciate if you let me know if it works for you. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Monday, November 30, 2009 11:38 AM > To: rsyslog-users > Subject: Re: [rsyslog] Auto-creating directories,when using templates > and the property replacer. > > Thanks, I was able to reproduced the problem, will now look into it. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > > Sent: Saturday, November 28, 2009 11:08 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] Auto-creating directories,when using templates > > and the property replacer. > > > > On Sat, Nov 28, 2009 at 03:52, Rainer Gerhards > > wrote: > > > Oh: which version do you use? 5.2.0? Then you should try the recent > > beta first... > > > > I'm using Rsyslog v5.5.1, and Librelp v.0.1.3. > > > > -Ryan > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From xkubina at fi.muni.cz Mon Nov 30 14:55:55 2009 From: xkubina at fi.muni.cz (Tomas Kubina) Date: Mon, 30 Nov 2009 14:55:55 +0100 Subject: [rsyslog] TLS/GSSAPI client message lost Message-ID: <4B13CEEB.4040504@fi.muni.cz> Hi, I want to use TLS or GSS for message delivering to central rsyslog server. The problem is that the first message logged after server's shutdown is lost, but when I use plain TCP this issue doesn't happen. Is it a feature or mistake in my config? This is config for client: ################# #### MODULES #### ################# $ModLoad imuxsock # provides support for local system logging $ModLoad imklog # provides kernel logging support (previously done by rklogd) $ModLoad immark # provides --MARK-- message capability ########################### #### GLOBAL DIRECTIVES #### ########################### # # Use traditional timestamp format. # To enable high precision timestamps, comment out the following line. # $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat ############### #### RULES #### ############### # # First some standard log files. Log by facility. # auth,authpriv.* /var/log/auth.log *.*;auth,authpriv.none -/var/log/syslog #cron.* /var/log/cron.log daemon.* -/var/log/daemon.log kern.* -/var/log/kern.log lpr.* -/var/log/lpr.log mail.* -/var/log/mail.log user.* -/var/log/user.log # # Logging for the mail system. Split it up so that # it is easy to write scripts to parse these files. # mail.info -/var/log/mail.info mail.warn -/var/log/mail.warn mail.err /var/log/mail.err # # Logging for INN news system. # news.crit /var/log/news/news.crit news.err /var/log/news/news.err news.notice -/var/log/news/news.notice # # Some "catch-all" log files. # *.=debug;\ auth,authpriv.none;\ news.none;mail.none -/var/log/debug *.=info;*.=notice;*.=warn;\ auth,authpriv.none;\ cron,daemon.none;\ mail,news.none -/var/log/messages # # Emergencies are sent to everybody logged in. # *.emerg * # # I like to have messages displayed on the console, but only on a virtual # console I usually leave idle. # #daemon,mail.*;\ # news.=crit;news.=err;news.=notice;\ # *.=debug;*.=info;\ # *.=notice;*.=warn /dev/tty8 # The named pipe /dev/xconsole is for the `xconsole' utility. To use it, # you must invoke `xconsole' with the `-file' option: # # $ xconsole -file /dev/xconsole [...] # # NOTE: adjust the list below, or you'll go crazy if you have a reasonably # busy site.. # daemon.*;mail.*;\ news.err;\ *.=debug;*.=info;\ *.=notice;*.=warn |/dev/xconsole # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. $WorkDirectory /var/tmp/rsyslog/spool # where to place spool files $ActionQueueFileName uniqName # unique name prefix for spool files $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) $ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously $ActionResumeRetryCount -1 # infinite retries if host is down #$DefaultNetstreamDriver gtls # use gtls netstream driver # certificate files - just CA for a client #$DefaultNetstreamDriverCAFile /root/ils/cacert.pem #$DefaultNetstreamDriverCertFile /root/ils/bobatko_cert.pem #$DefaultNetstreamDriverKeyFile /root/ils/bobatko_cert.pem # set up the action #$ActionSendStreamDriverMode 1 # require TLS for the connection #$ActionSendStreamDriverAuthMode anon # server is NOT authenticated #local7.info @@example.com:10514 $ModLoad omgssapi local7.info : omgssapi:example.com:10514 and the server side: $ModLoad immark # provides --MARK-- message capability $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imklog # kernel logging (formerly provided by rklogd) # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none -/var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* -/var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit -/var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) #$ModLoad imtcp # TCP listener # make gtls driver the default #$DefaultNetstreamDriver gtls # certificate files #$DefaultNetstreamDriverCAFile /root/rsyslog/cacert.pem #$DefaultNetstreamDriverCertFile /root/rsyslog/rsyslog_cert.pem #$DefaultNetstreamDriverKeyFile /root/rsyslog/rsyslog_key.pem #$InputTCPServerStreamDriverAuthMode x509/name #$InputTCPServerStreamDriverPermittedPeer bobatko #$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode #$InputTCPServerRun 10514 # start up listener at port 10514 $ModLoad imgssapi $InputGSSServerRun 10514 Thank you for the answer, Regards, Tomas Kubina From rgerhards at hq.adiscon.com Mon Nov 30 15:06:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 30 Nov 2009 15:06:53 +0100 Subject: [rsyslog] TLS/GSSAPI client message lost References: <4B13CEEB.4040504@fi.muni.cz> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034DF@GRFEXC.intern.adiscon.com> I unfortunately do not have enough insight into GSSAPI to provide a real answer. My guess is that this happens for the same reason it can happen with any ack-less transport. In the plain tcp driver, I have done quite some work to try to limit loss potential. I have also some ideas of how to further try to prevent message loss, but you cannot totally aovid it. some background: http://blog.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tomas Kubina > Sent: Monday, November 30, 2009 2:56 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] TLS/GSSAPI client message lost > > Hi, > > I want to use TLS or GSS for message delivering to central rsyslog > server. > The problem is that the first message logged after server's shutdown is > lost, > but when I use plain TCP this issue doesn't happen. Is it a feature or > mistake > in my config? > > This is config for client: > > ################# > #### MODULES #### > ################# > > $ModLoad imuxsock # provides support for local system logging > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > $ModLoad immark # provides --MARK-- message capability > > > ########################### > #### GLOBAL DIRECTIVES #### > ########################### > > # > # Use traditional timestamp format. > # To enable high precision timestamps, comment out the following line. > # > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > ############### > #### RULES #### > ############### > > # > # First some standard log files. Log by facility. > # > auth,authpriv.* /var/log/auth.log > *.*;auth,authpriv.none -/var/log/syslog > #cron.* /var/log/cron.log > daemon.* -/var/log/daemon.log > kern.* -/var/log/kern.log > lpr.* -/var/log/lpr.log > mail.* -/var/log/mail.log > user.* -/var/log/user.log > > # > # Logging for the mail system. Split it up so that > # it is easy to write scripts to parse these files. > # > mail.info -/var/log/mail.info > mail.warn -/var/log/mail.warn > mail.err /var/log/mail.err > > # > # Logging for INN news system. > # > news.crit /var/log/news/news.crit > news.err /var/log/news/news.err > news.notice -/var/log/news/news.notice > > # > # Some "catch-all" log files. > # > *.=debug;\ > auth,authpriv.none;\ > news.none;mail.none -/var/log/debug > *.=info;*.=notice;*.=warn;\ > auth,authpriv.none;\ > cron,daemon.none;\ > mail,news.none -/var/log/messages > > # > # Emergencies are sent to everybody logged in. > # > *.emerg * > > # > # I like to have messages displayed on the console, but only on a > virtual > # console I usually leave idle. > # > #daemon,mail.*;\ > # news.=crit;news.=err;news.=notice;\ > # *.=debug;*.=info;\ > # *.=notice;*.=warn /dev/tty8 > > # The named pipe /dev/xconsole is for the `xconsole' utility. To use > it, > # you must invoke `xconsole' with the `-file' option: > # > # $ xconsole -file /dev/xconsole [...] > # > # NOTE: adjust the list below, or you'll go crazy if you have a > reasonably > # busy site.. > # > daemon.*;mail.*;\ > news.err;\ > *.=debug;*.=info;\ > *.=notice;*.=warn |/dev/xconsole > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/tmp/rsyslog/spool # where to place spool files > $ActionQueueFileName uniqName # unique name prefix for spool files > $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) > $ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > $ActionResumeRetryCount -1 # infinite retries if host is down > > > #$DefaultNetstreamDriver gtls # use gtls netstream driver > > # certificate files - just CA for a client > #$DefaultNetstreamDriverCAFile /root/ils/cacert.pem > #$DefaultNetstreamDriverCertFile /root/ils/bobatko_cert.pem > #$DefaultNetstreamDriverKeyFile /root/ils/bobatko_cert.pem > > # set up the action > #$ActionSendStreamDriverMode 1 # require TLS for the connection > #$ActionSendStreamDriverAuthMode anon # server is NOT authenticated > > #local7.info @@example.com:10514 > > > $ModLoad omgssapi > local7.info : omgssapi:example.com:10514 > > > and the server side: > > $ModLoad immark # provides --MARK-- message capability > $ModLoad imuxsock # provides support for local system logging (e.g. via > logger command) > $ModLoad imklog # kernel logging (formerly provided by rklogd) > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none -/var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* -/var/log/maillog > > > # Log cron stuff > cron.* -/var/log/cron > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit -/var/log/spooler > > # Save boot messages also to boot.log > local7.* /var/log/boot.log > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > > #$ModLoad imtcp # TCP listener > > # make gtls driver the default > #$DefaultNetstreamDriver gtls > > # certificate files > #$DefaultNetstreamDriverCAFile /root/rsyslog/cacert.pem > #$DefaultNetstreamDriverCertFile /root/rsyslog/rsyslog_cert.pem > #$DefaultNetstreamDriverKeyFile /root/rsyslog/rsyslog_key.pem > > #$InputTCPServerStreamDriverAuthMode x509/name > #$InputTCPServerStreamDriverPermittedPeer bobatko > #$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode > > #$InputTCPServerRun 10514 # start up listener at port 10514 > > $ModLoad imgssapi > $InputGSSServerRun 10514 > > Thank you for the answer, > > Regards, > Tomas Kubina > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From iamsayan at gmail.com Mon Nov 30 17:20:16 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Mon, 30 Nov 2009 11:20:16 -0500 Subject: [rsyslog] using arbitrary facility id Message-ID: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com> Hello All, Is it possible to use a facility id other than local0-local7? I was using a facility id of 50 in some of the messages , and I had written a selector line in my rsyslog file as well to log messages with facility id of 50 into a seperate file. However, I see that sometimes the messages are being written into all the files(/var/log/messages, boot.log,/var/log/secure etc) instead of the one I specified in the rsyslog.conf. If I restart rsyslog the problem goes away. I am using rsyslog version 4.2.0. Here is the selector line in my config if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and $syslogseverity <= '6' then $log_rotation_50 where $log_rotation_50 is an outchannel which is configured to rotate the file when it reaches the size of 2MB. Regards, Sayan From rgerhards at hq.adiscon.com Mon Nov 30 17:40:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 30 Nov 2009 17:40:17 +0100 Subject: [rsyslog] using arbitrary facility id References: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com> You can use facilities other than local0..local7, but you cannot use a facility with a numerical value greater than 23, because the relevant standards do not permit this (see RFC5424, Table 1). It may be that rsyslog does not properly prevent this, maybe it uses modulo 24 in this case. Will check that. But it is invalid in any case (not only with rsyslog but with any syslogd). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Monday, November 30, 2009 5:20 PM > To: rsyslog-users > Subject: [rsyslog] using arbitrary facility id > > Hello All, > Is it possible to use a facility id other than local0-local7? > I was using a facility id of 50 in some of the messages , and I had > written > a selector line in my rsyslog file as well to log messages with > facility id > of 50 into a seperate file. > However, I see that sometimes the messages are being written into all > the > files(/var/log/messages, boot.log,/var/log/secure etc) instead of the > one I > specified in the rsyslog.conf. If I restart rsyslog the problem goes > away. > I am using rsyslog version 4.2.0. > > Here is the selector line in my config > > > if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and > $syslogseverity <= '6' then $log_rotation_50 > > where $log_rotation_50 is an outchannel which is configured to rotate > the > file when it reaches the size of 2MB. > Regards, > Sayan > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From iamsayan at gmail.com Mon Nov 30 17:51:40 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Mon, 30 Nov 2009 11:51:40 -0500 Subject: [rsyslog] using arbitrary facility id In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com> References: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540911300851r1f0bab68ldf7f995ed0a094d8@mail.gmail.com> Hello Rainer, Thanks... Yes I agree what I am doing is invalid. I was just trying it out, my idea was if I can use numbers greater than 23, then I can maybe use it as my own customized logging system, and I would use numbers greater than 23 for my application and use different ids (>23) for different kind of application level log messages. I would try to look into the code as well, is this something you thing may be useful in general? Regards, Sayan On Mon, Nov 30, 2009 at 11:40 AM, Rainer Gerhards wrote: > You can use facilities other than local0..local7, but you cannot use a > facility with a numerical value greater than 23, because the relevant > standards do not permit this (see RFC5424, Table 1). > > It may be that rsyslog does not properly prevent this, maybe it uses modulo > 24 in this case. Will check that. But it is invalid in any case (not only > with rsyslog but with any syslogd). > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > Sent: Monday, November 30, 2009 5:20 PM > > To: rsyslog-users > > Subject: [rsyslog] using arbitrary facility id > > > > Hello All, > > Is it possible to use a facility id other than local0-local7? > > I was using a facility id of 50 in some of the messages , and I had > > written > > a selector line in my rsyslog file as well to log messages with > > facility id > > of 50 into a seperate file. > > However, I see that sometimes the messages are being written into all > > the > > files(/var/log/messages, boot.log,/var/log/secure etc) instead of the > > one I > > specified in the rsyslog.conf. If I restart rsyslog the problem goes > > away. > > I am using rsyslog version 4.2.0. > > > > Here is the selector line in my config > > > > > > if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and > > $syslogseverity <= '6' then $log_rotation_50 > > > > where $log_rotation_50 is an outchannel which is configured to rotate > > the > > file when it reaches the size of 2MB. > > Regards, > > Sayan > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iamsayan at gmail.com Mon Nov 30 17:58:46 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Mon, 30 Nov 2009 11:58:46 -0500 Subject: [rsyslog] using arbitrary facility id In-Reply-To: <87a1ae540911300851r1f0bab68ldf7f995ed0a094d8@mail.gmail.com> References: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com> <87a1ae540911300851r1f0bab68ldf7f995ed0a094d8@mail.gmail.com> Message-ID: <87a1ae540911300858v1d4cda71k787a53158e162b7d@mail.gmail.com> BTW, just out of interest, why is this number restricted to 23 in the rfc? also each facility other than local0-7 seems to be rigidly defined. Just wanted to know why is it so ..shouldn't they have left scope for extension on this? On Mon, Nov 30, 2009 at 11:51 AM, Sayan Chowdhury wrote: > Hello Rainer, > Thanks... Yes I agree what I am doing is invalid. > I was just trying it out, my idea was if I can use numbers greater than > 23, then I can maybe use it as my own customized logging system, and I would > use numbers greater than 23 for my application and use different ids (>23) > for different kind of application level log messages. I would try to look > into the code as well, is this something you thing may be useful in general? > Regards, > Sayan > > > > On Mon, Nov 30, 2009 at 11:40 AM, Rainer Gerhards < > rgerhards at hq.adiscon.com> wrote: > >> You can use facilities other than local0..local7, but you cannot use a >> facility with a numerical value greater than 23, because the relevant >> standards do not permit this (see RFC5424, Table 1). >> >> It may be that rsyslog does not properly prevent this, maybe it uses >> modulo >> 24 in this case. Will check that. But it is invalid in any case (not only >> with rsyslog but with any syslogd). >> >> Rainer >> >> > -----Original Message----- >> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury >> > Sent: Monday, November 30, 2009 5:20 PM >> > To: rsyslog-users >> > Subject: [rsyslog] using arbitrary facility id >> > >> > Hello All, >> > Is it possible to use a facility id other than local0-local7? >> > I was using a facility id of 50 in some of the messages , and I had >> > written >> > a selector line in my rsyslog file as well to log messages with >> > facility id >> > of 50 into a seperate file. >> > However, I see that sometimes the messages are being written into all >> > the >> > files(/var/log/messages, boot.log,/var/log/secure etc) instead of the >> > one I >> > specified in the rsyslog.conf. If I restart rsyslog the problem goes >> > away. >> > I am using rsyslog version 4.2.0. >> > >> > Here is the selector line in my config >> > >> > >> > if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and >> > $syslogseverity <= '6' then $log_rotation_50 >> > >> > where $log_rotation_50 is an outchannel which is configured to rotate >> > the >> > file when it reaches the size of 2MB. >> > Regards, >> > Sayan >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > From jbondc at openmv.com Mon Nov 30 21:49:24 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Mon, 30 Nov 2009 15:49:24 -0500 Subject: [rsyslog] TLS/GSSAPI client message lost In-Reply-To: <4B13CEEB.4040504@fi.muni.cz> References: <4B13CEEB.4040504@fi.muni.cz> Message-ID: <006d01ca71fe$9a625d50$cf2717f0$@com> On Mon Nov 30 08:55 AM, Tomas Kubina wrote: > Hi, > > I want to use TLS or GSS for message delivering to central rsyslog > server. > The problem is that the first message logged after server's shutdown > is lost, but when I use plain TCP this issue doesn't happen. Is it a > feature or mistake in my config? > I think your problem will be fixed in the latest rsyslog: http://www.rsyslog.com/Article433.phtml From epiphani at gmail.com Mon Nov 2 23:51:40 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Mon, 2 Nov 2009 17:51:40 -0500 Subject: [rsyslog] Queuing bug (4.5.5) Message-ID: Greetings all, I appear to have run into an issue with 4.5.5 where queues are not being flushed in a timely manner. In this specific case, I have data from last wednesday that is being written to disk in small chunks today since last wednesday. Unfortunately I cannot be too specific in details, but here is what I can see: Using omfile+gzip, there appears to have been a decent burst in traffic sometime last wednesday. The rsyslog instance has grown to 1.8GB of memory and stayed there for a while. I have both main message and action queues defined using fixedarray, and I see no on-disk queues (appears to be entirely in memory). I've got templates writing out to filenames using an hourly timestamp (filenames like: [token]-[host]-YYYYMMDD-HH.txt.gz) In some of those files, all of them less than 5k in size, there are between 5 and 15 lines of content, all of them from last wednesday, and within a few seconds of each other. It's almost like there was a significant queue built up, the hour rolled over, and only the first block of lines were pulled from the queue. Then the hour rolled over again, and another block of lines were pulled from the queue. Then the next hour, then another 5-15 lines. Is it possible that one of the queues still has a good chunk of data built up, and is flushing it out very slowly? It hasn't been consistantly at the top of the hour either, and not every hour. But the log lines themselves are sequentially written out, and usually the lines are within a few seconds of each other. For example: syslog-myhost-20091102-18.txt.gz: 3 lines, 2 with TS Oct 21 18:46:34 and one 18:46:35 syslog-myhost-20091102-19.txt.gz: 17 lines, 3 Oct 21 18:46:35, 14 lines Oct 21 18:46:36 syslog-myhost-20091102-20.txt.gz: 12 lines, 8 Oct 21 18:46:36, 4 lines Oct 21 18:46:37 Thoughts? -Aaron From rgerhards at hq.adiscon.com Tue Nov 3 07:46:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 07:46:29 +0100 Subject: [rsyslog] Queuing bug (4.5.5) References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> mhhh... This is obviously not intended behavior. There are some rate-limiting features that can deliberately slow down a queue, but I guess you have not configured these. So there either is a bug that activates them at some point during processing (e.g. an invalid memory access could do that) or there is some other bug that causes the dequeue to happen very slowly. In any case, it is hard to guess. Given the volume of the messages, a debug log probably does not help. We could gain a little insight in at least the queue sizes that rsyslog sees via imdiag plus the (very rudamentary) v5 debug front-end (it doesn't require the engine to be v5!). I would need to spend at least a little work on the front-end to enable remote access, but that's not really a problem. One other thing is that I am still holding a v4-beta with additional fixes as I didn't want to put these in the middle of some other debugging. However, these fixes address potential memory problems, so the most appropriate course of action would be to give that version at least a try. It needs to be released in the next days in any case. I have uploaded that (pre-4.5.6) version so that you can give it a try if you like: http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz I think it would good if you could try it at least once, because I think any other troubleshooting will boil down to looking at the fixes this version contains. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Monday, November 02, 2009 11:52 PM > To: rsyslog-users > Subject: [rsyslog] Queuing bug (4.5.5) > > Greetings all, > > I appear to have run into an issue with 4.5.5 where queues are not > being flushed in a timely manner. In this specific case, I have data > from last wednesday that is being written to disk in small chunks > today since last wednesday. Unfortunately I cannot be too specific in > details, but here is what I can see: > > Using omfile+gzip, there appears to have been a decent burst in > traffic sometime last wednesday. The rsyslog instance has grown to > 1.8GB of memory and stayed there for a while. I have both main > message and action queues defined using fixedarray, and I see no > on-disk queues (appears to be entirely in memory). > > I've got templates writing out to filenames using an hourly timestamp > (filenames like: [token]-[host]-YYYYMMDD-HH.txt.gz) In some of those > files, all of them less than 5k in size, there are between 5 and 15 > lines of content, all of them from last wednesday, and within a few > seconds of each other. It's almost like there was a significant queue > built up, the hour rolled over, and only the first block of lines were > pulled from the queue. Then the hour rolled over again, and another > block of lines were pulled from the queue. Then the next hour, then > another 5-15 lines. > > Is it possible that one of the queues still has a good chunk of data > built up, and is flushing it out very slowly? It hasn't been > consistantly at the top of the hour either, and not every hour. But > the log lines themselves are sequentially written out, and usually the > lines are within a few seconds of each other. > > For example: > > syslog-myhost-20091102-18.txt.gz: 3 lines, 2 with TS Oct 21 18:46:34 > and one 18:46:35 > syslog-myhost-20091102-19.txt.gz: 17 lines, 3 Oct 21 18:46:35, 14 > lines Oct 21 18:46:36 > syslog-myhost-20091102-20.txt.gz: 12 lines, 8 Oct 21 18:46:36, 4 > lines Oct 21 18:46:37 > > Thoughts? > -Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 3 07:59:23 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 07:59:23 +0100 Subject: [rsyslog] Queue sizing References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103316@GRFEXC.intern.adiscon.com> Sorry for the late reply, but I was really busy with my article last week. Let me provide some rough numbers. For usual syslog traffic (< 100 bytes message size), 768 bytes is a good size to be expected for use by each message. If you have larger traffic, add the full average size to that picture (e.g. average size is 200 bytes, the typical size of a message object is around 968 bytes, NOT 868 - this has to do with some internal optimization). Then, simply multiply the configured size of the queue with that number. For example, you have a 2,000,000 message max configured, so 2,000,000 * 768 which gives you ~ 1.4 GB message store if the queue is full. The same applies to the action queues, here we have 200,000 entries per queue, that would sum up to ~140 MB per queue. Multiply that with the number of action queues. If, for example, you have 20 action queues, that would sum up to a total memory requirement of 20*.14+1.4 = 4.2 GB of total queue memory. Of course, this is an extreme number, and typically no system will have all action queues totally filled up. For omfile, with background writing (always enabled in ZIP mode), we use double-buffering with the configured buffers sizes (256KB being the default). So add 0.5 MB per open dynafile in this case. Multiply that by the maximum number of dynafiles you permit to be kept open. Add these numbers for each dynafile action. That gives you a (very theoretical) upper bound of memory use for the output system. Limiting the memory use is so simple that it probably is easy to overlook: just reduce the maximum numbers - that's why they exist ;) Depending on the input source, rsyslog then tries to slow down too-fast senders if it can't keep up. This works pretty well with TCP and not at all with UDP. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Wednesday, October 28, 2009 4:14 PM > To: rsyslog-users > Subject: [rsyslog] Queue sizing > > Greetings, > > I'm trying to get a good handle on queue sizing and memory usage. In > a higher-traffic environment, I have a dedicated rsyslog machine with > a large amount of ram. I want to try and find the right balance > between large queues for burst traffic, but I also want to make sure > that the queue sizes are small enough that we don't run out of memory > and crash. I'm using 4.5.5 currently. > > I'm currently using disk-backed memory queues, configured in > fixedarray, like so: > > $MainMsgQueueSize 2000000 > $MainMsgQueueMaxFileSize 50g > $MainMsgQueueSaveOnShutdown on > $MainMsgQueueHighWaterMark 15000000 > $MainMsgQueueLowWaterMark 10000000 > $MainMsgQueueType FixedArray > $MainMsgQueueWorkerThreads 2 > > $ActionQueueSize 200000 > $ActionQueueMaxFileSize 50g > $ActionQueueSaveOnShutdown on > $ActionQueueHighWaterMark 1500000 > $ActionQueueLowWaterMark 1000000 > $ActionQueueType FixedArray > $ActionQueueWorkerThreads 1 > > > I usually have upwards of a dozen action queue definitions, and my > omfile outputs are based on hostname of the incoming connection. Is > there some way I can calculate maximum memory use by the queuing > system, or force the system to a set memory maximum where I -know- > that it will simply start pushing back on the incoming connection > instead of adding more data to the system? > > Thoughts? > -Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 3 09:59:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 09:59:44 +0100 Subject: [rsyslog] queue configuration References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103318@GRFEXC.intern.adiscon.com> David and all, I have created a new output plugin, omruleset, which permits to nest rulesets. What it does is copy a message over from one ruleset to another and make it process in parallel. Some more details in the doc: http://www.rsyslog.com/doc-omruleset.html It permits to do what you asked for below. It also permits to do many more things and create very advanced configurations. But one must be VERY careful to receive the intended results. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 23, 2009 10:23 PM > To: rsyslog-users > Subject: [rsyslog] queue configuration > > I know that you can create an action queue for a specific output. > > is there any way to create an action queue for multiple outputs? > > for example, in my configuration I have > > :fromhost, !isequal, "127.0.0.1" > /var/log/messages;TraditionalFormat > :fromhost, isequal, "127.0.0.1" > @192.168.1.1;TraditionalForwardFormat > *.* @192.168.1.115 > *.* @192.168.1.241 > *.* @192.168.1.242 > *.* @192.168.1.6 > *.* @192.168.1.7 > *.* @192.168.1.122 > :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 > & @192.168.1.1;fixsnareForwardFormat2 > & ~ > :syslogtag, startswith, "MSWinEventLog#011" > /var/log/messages;fixsnareFormat > & @192.168.1.1;fixsnareForwardFormat > & ~ > *.* /var/log/messages;TraditionalFormat > *.* @192.168.1.1 > > it would be nice to move the outbound relays off to a different queue, > and > to put the list of rules that have order dependancies (because they > output > in different formats to fix up formatting problem) to a seperate queue > to > spread the work across multiple processes and not slow down the basic > writing to file. > > but as far as I can tell I would have to create a queue for each > individual item, and I can't create a queue for the part of the config > where I need to discard. > > am I missing something (including a better way to do everything ;-) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Nov 3 11:58:02 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 3 Nov 2009 11:58:02 +0100 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200910201639.43143.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> Message-ID: <200911031158.02666.marc.schiffbauer@mightycare.de> Hello Rainer, is there some news about this issue? TIA -Marc Am Dienstag, 20. Oktober 2009 18:38:16 schrieb Rainer Gerhards: > thanks! > > Interesting, I see that only one of the messages that should be in the .qi > are actually present. I wonder if this is related to the fact that ompgsql > emits an error message itself while being down (the others don't do this > AFIK). Will try to dig down to this (but I have to do so as time permits). > > Rainer > From mikel at irontec.com Tue Nov 3 12:20:18 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 03 Nov 2009 12:20:18 +0100 Subject: [rsyslog] working directoy dude Message-ID: <4AF011F2.7040501@irontec.com> Hi! I have a simple question: When you configure reliable TCP mesage forwarding, the "$WorkDirectory /rsyslog/work" have to be created? So, mkdir -p /rsyslog/work ? Thanks From rgerhards at hq.adiscon.com Tue Nov 3 12:28:19 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 12:28:19 +0100 Subject: [rsyslog] working directoy dude References: <4AF011F2.7040501@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710331C@GRFEXC.intern.adiscon.com> yes :) > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, November 03, 2009 12:20 PM > To: rsyslog-users > Subject: [rsyslog] working directoy dude > > Hi! > > I have a simple question: > > When you configure reliable TCP mesage forwarding, the "$WorkDirectory > /rsyslog/work" have to be created? > > So, mkdir -p /rsyslog/work ? > > Thanks > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mikel at irontec.com Tue Nov 3 12:36:02 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 03 Nov 2009 12:36:02 +0100 Subject: [rsyslog] working directoy dude In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710331C@GRFEXC.intern.adiscon.com> References: <4AF011F2.7040501@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA710331C@GRFEXC.intern.adiscon.com> Message-ID: <4AF015A2.6090303@irontec.com> Thanks Rainer!! Rainer Gerhards wrote: > yes :) > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Tuesday, November 03, 2009 12:20 PM >> To: rsyslog-users >> Subject: [rsyslog] working directoy dude >> >> Hi! >> >> I have a simple question: >> >> When you configure reliable TCP mesage forwarding, the "$WorkDirectory >> /rsyslog/work" have to be created? >> >> So, mkdir -p /rsyslog/work ? >> >> Thanks >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From epiphani at gmail.com Tue Nov 3 15:59:57 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 3 Nov 2009 09:59:57 -0500 Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> Message-ID: This is still taking place this hour - though I obviously can't restart onto a newer version without losing this case. Is there anything I can do in the current configuration to try and debug this situation? (We're up to 18:46:51 now!) -Aaron On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards wrote: > mhhh... This is obviously not intended behavior. There are some rate-limiting > features that can deliberately slow down a queue, but I guess you have not > configured these. So there either is a bug that activates them at some point > during processing (e.g. an invalid memory access could do that) or there is > some other bug that causes the dequeue to happen very slowly. In any case, it > is hard to guess. > > Given the volume of the messages, a debug log probably does not help. We > could gain a little insight in at least the queue sizes that rsyslog sees via > imdiag plus the (very rudamentary) v5 debug front-end (it doesn't require the > engine to be v5!). I would need to spend at least a little work on the > front-end to enable remote access, but that's not really a problem. > > One other thing is that I am still holding a v4-beta with additional fixes as > I didn't want to put these in the middle of some other debugging. However, > these fixes address potential memory problems, so the most appropriate course > of action would be to give that version at least a try. It needs to be > released in the next days in any case. > > I have uploaded that (pre-4.5.6) version so that you can give it a try if you > like: > > http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz > > I think it would good if you could try it at least once, because I think any > other troubleshooting will boil down to looking at the fixes this version > contains. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> Sent: Monday, November 02, 2009 11:52 PM >> To: rsyslog-users >> Subject: [rsyslog] Queuing bug (4.5.5) >> >> Greetings all, >> >> I appear to have run into an issue with 4.5.5 where queues are not >> being flushed in a timely manner. ?In this specific case, I have data >> from last wednesday that is being written to disk in small chunks >> today since last wednesday. ?Unfortunately I cannot be too specific in >> details, but here is what I can see: >> >> Using omfile+gzip, there appears to have been a decent burst in >> traffic sometime last wednesday. ?The rsyslog instance has grown to >> 1.8GB of memory and stayed there for a while. ?I have both main >> message and action queues defined using fixedarray, and I see no >> on-disk queues (appears to be entirely in memory). >> >> I've got templates writing out to filenames using an hourly timestamp >> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of those >> files, all of them less than 5k in size, there are between 5 and 15 >> lines of content, all of them from last wednesday, and within a few >> seconds of each other. ?It's almost like there was a significant queue >> built up, the hour rolled over, and only the first block of lines were >> pulled from the queue. ?Then the hour rolled over again, and another >> block of lines were pulled from the queue. ?Then the next hour, then >> another 5-15 lines. >> >> Is it possible that one of the queues still has a good chunk of data >> built up, and is flushing it out very slowly? ?It hasn't been >> consistantly at the top of the hour either, and not every hour. ?But >> the log lines themselves are sequentially written out, and usually the >> lines are within a few seconds of each other. >> >> For example: >> >> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 18:46:34 >> and one 18:46:35 >> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, 14 >> lines Oct 21 18:46:36 >> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, 4 >> lines Oct 21 18:46:37 >> >> Thoughts? >> -Aaron >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Tue Nov 3 16:06:15 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 3 Nov 2009 07:06:15 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 3 Nov 2009, Aaron Wiebe wrote: > This is still taking place this hour - though I obviously can't > restart onto a newer version without losing this case. Is there > anything I can do in the current configuration to try and debug this > situation? if you can strace each thread for a few seconds it may give you an idea what's happening. in the meantime you need to stop the system from getting further behind, can you redirect or reconfigure the senders so that they are not sending new logs to this box so that it can dig itself out (stopping the input may be enough by itself, rsyslog has historicly done a LOT of locking around the main queue, and if you have a full queue with more data trying to be delivered it can really slow things down. I wouldn't expect it to be this much, but it could be part of it) David Lang > (We're up to 18:46:51 now!) > > -Aaron > > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards > wrote: >> mhhh... This is obviously not intended behavior. There are some rate-limiting >> features that can deliberately slow down a queue, but I guess you have not >> configured these. So there either is a bug that activates them at some point >> during processing (e.g. an invalid memory access could do that) or there is >> some other bug that causes the dequeue to happen very slowly. In any case, it >> is hard to guess. >> >> Given the volume of the messages, a debug log probably does not help. We >> could gain a little insight in at least the queue sizes that rsyslog sees via >> imdiag plus the (very rudamentary) v5 debug front-end (it doesn't require the >> engine to be v5!). I would need to spend at least a little work on the >> front-end to enable remote access, but that's not really a problem. >> >> One other thing is that I am still holding a v4-beta with additional fixes as >> I didn't want to put these in the middle of some other debugging. However, >> these fixes address potential memory problems, so the most appropriate course >> of action would be to give that version at least a try. It needs to be >> released in the next days in any case. >> >> I have uploaded that (pre-4.5.6) version so that you can give it a try if you >> like: >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz >> >> I think it would good if you could try it at least once, because I think any >> other troubleshooting will boil down to looking at the fixes this version >> contains. >> >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>> Sent: Monday, November 02, 2009 11:52 PM >>> To: rsyslog-users >>> Subject: [rsyslog] Queuing bug (4.5.5) >>> >>> Greetings all, >>> >>> I appear to have run into an issue with 4.5.5 where queues are not >>> being flushed in a timely manner. ?In this specific case, I have data >>> from last wednesday that is being written to disk in small chunks >>> today since last wednesday. ?Unfortunately I cannot be too specific in >>> details, but here is what I can see: >>> >>> Using omfile+gzip, there appears to have been a decent burst in >>> traffic sometime last wednesday. ?The rsyslog instance has grown to >>> 1.8GB of memory and stayed there for a while. ?I have both main >>> message and action queues defined using fixedarray, and I see no >>> on-disk queues (appears to be entirely in memory). >>> >>> I've got templates writing out to filenames using an hourly timestamp >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of those >>> files, all of them less than 5k in size, there are between 5 and 15 >>> lines of content, all of them from last wednesday, and within a few >>> seconds of each other. ?It's almost like there was a significant queue >>> built up, the hour rolled over, and only the first block of lines were >>> pulled from the queue. ?Then the hour rolled over again, and another >>> block of lines were pulled from the queue. ?Then the next hour, then >>> another 5-15 lines. >>> >>> Is it possible that one of the queues still has a good chunk of data >>> built up, and is flushing it out very slowly? ?It hasn't been >>> consistantly at the top of the hour either, and not every hour. ?But >>> the log lines themselves are sequentially written out, and usually the >>> lines are within a few seconds of each other. >>> >>> For example: >>> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 18:46:34 >>> and one 18:46:35 >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, 14 >>> lines Oct 21 18:46:36 >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, 4 >>> lines Oct 21 18:46:37 >>> >>> Thoughts? >>> -Aaron >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 3 16:25:30 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 16:25:30 +0100 Subject: [rsyslog] Queuing bug (4.5.5) References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, November 03, 2009 4:06 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) > > On Tue, 3 Nov 2009, Aaron Wiebe wrote: > > > This is still taking place this hour - though I obviously can't > > restart onto a newer version without losing this case. Is there > > anything I can do in the current configuration to try and debug this > > situation? > > if you can strace each thread for a few seconds it may give you an idea > what's happening. This is a good suggestion. It would also potentially be enlightening to attach gdb to the process at various points in time and get a backtrace from all running threads. Other than that, there is little you can do on a running version. If it were compiled for debug, and the environment setup so that we could capture stdout/stderr, sending SIGUSR2 would yield to much the same information as the gdb backtrace BUT when runtime instrumentation is enabled, the build is operating at a third to a quarter of its normal speed. > in the meantime you need to stop the system from getting further > behind, > can you redirect or reconfigure the senders so that they are not > sending > new logs to this box so that it can dig itself out (stopping the input > may > be enough by itself, rsyslog has historicly done a LOT of locking > around > the main queue, and if you have a full queue with more data trying to > be > delivered it can really slow things down. I wouldn't expect it to be > this > much, but it could be part of it) > This may clean up things, but I really doubt it, given the magnitude of delays. Also, Aaron runs 4.4.5, which already has lots of the locking removed/restructure (not to compare with the recent v5-devel, but much more effcient than early v4 and v3). Rainer > David Lang > > > (We're up to 18:46:51 now!) > > > > -Aaron > > > > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards > > wrote: > >> mhhh... This is obviously not intended behavior. There are some > rate-limiting > >> features that can deliberately slow down a queue, but I guess you > have not > >> configured these. So there either is a bug that activates them at > some point > >> during processing (e.g. an invalid memory access could do that) or > there is > >> some other bug that causes the dequeue to happen very slowly. In any > case, it > >> is hard to guess. > >> > >> Given the volume of the messages, a debug log probably does not > help. We > >> could gain a little insight in at least the queue sizes that rsyslog > sees via > >> imdiag plus the (very rudamentary) v5 debug front-end (it doesn't > require the > >> engine to be v5!). I would need to spend at least a little work on > the > >> front-end to enable remote access, but that's not really a problem. > >> > >> One other thing is that I am still holding a v4-beta with additional > fixes as > >> I didn't want to put these in the middle of some other debugging. > However, > >> these fixes address potential memory problems, so the most > appropriate course > >> of action would be to give that version at least a try. It needs to > be > >> released in the next days in any case. > >> > >> I have uploaded that (pre-4.5.6) version so that you can give it a > try if you > >> like: > >> > >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz > >> > >> I think it would good if you could try it at least once, because I > think any > >> other troubleshooting will boil down to looking at the fixes this > version > >> contains. > >> > >> Rainer > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > >>> Sent: Monday, November 02, 2009 11:52 PM > >>> To: rsyslog-users > >>> Subject: [rsyslog] Queuing bug (4.5.5) > >>> > >>> Greetings all, > >>> > >>> I appear to have run into an issue with 4.5.5 where queues are not > >>> being flushed in a timely manner. ?In this specific case, I have > data > >>> from last wednesday that is being written to disk in small chunks > >>> today since last wednesday. ?Unfortunately I cannot be too specific > in > >>> details, but here is what I can see: > >>> > >>> Using omfile+gzip, there appears to have been a decent burst in > >>> traffic sometime last wednesday. ?The rsyslog instance has grown to > >>> 1.8GB of memory and stayed there for a while. ?I have both main > >>> message and action queues defined using fixedarray, and I see no > >>> on-disk queues (appears to be entirely in memory). > >>> > >>> I've got templates writing out to filenames using an hourly > timestamp > >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of > those > >>> files, all of them less than 5k in size, there are between 5 and 15 > >>> lines of content, all of them from last wednesday, and within a few > >>> seconds of each other. ?It's almost like there was a significant > queue > >>> built up, the hour rolled over, and only the first block of lines > were > >>> pulled from the queue. ?Then the hour rolled over again, and > another > >>> block of lines were pulled from the queue. ?Then the next hour, > then > >>> another 5-15 lines. > >>> > >>> Is it possible that one of the queues still has a good chunk of > data > >>> built up, and is flushing it out very slowly? ?It hasn't been > >>> consistantly at the top of the hour either, and not every hour. > ?But > >>> the log lines themselves are sequentially written out, and usually > the > >>> lines are within a few seconds of each other. > >>> > >>> For example: > >>> > >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 > 18:46:34 > >>> and one 18:46:35 > >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, 14 > >>> lines Oct 21 18:46:36 > >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, 4 > >>> lines Oct 21 18:46:37 > >>> > >>> Thoughts? > >>> -Aaron > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > From epiphani at gmail.com Tue Nov 3 18:27:07 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 3 Nov 2009 12:27:07 -0500 Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> Message-ID: Ok - I captured an strace. This appears to be the action thread. This specific set of calls took minutes. I'll re-run this with -t and/or -T. Note that this syslog instance is not actually receiving any data right now. -Aaron 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL 26365 <... futex resumed> ) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL 26365 <... futex resumed> ) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL 26365 <... futex resumed> ) = 0 26365 clock_gettime(CLOCK_REALTIME, 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 26365 <... futex resumed> ) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL 26365 <... futex resumed> ) = 0 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL 26365 <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable) 26365 clock_gettime(CLOCK_REALTIME, 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 26365 <... futex resumed> ) = 0 26365 clock_gettime(CLOCK_REALTIME, 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Tuesday, November 03, 2009 4:06 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >> >> > This is still taking place this hour - though I obviously can't >> > restart onto a newer version without losing this case. ?Is there >> > anything I can do in the current configuration to try and debug this >> > situation? >> >> if you can strace each thread for a few seconds it may give you an idea >> what's happening. > > This is a good suggestion. > > It would also potentially be enlightening to attach gdb to the process at > various points in time and get a backtrace from all running threads. > > Other than that, there is little you can do on a running version. If it were > compiled for debug, and the environment setup so that we could capture > stdout/stderr, sending SIGUSR2 would yield to much the same information as > the gdb backtrace BUT when runtime instrumentation is enabled, the build is > operating at a third to a quarter of its normal speed. > >> in the meantime you need to stop the system from getting further >> behind, >> can you redirect or reconfigure the senders so that they are not >> sending >> new logs to this box so that it can dig itself out (stopping the input >> may >> be enough by itself, rsyslog has historicly done a LOT of locking >> around >> the main queue, and if you have a full queue with more data trying to >> be >> delivered it can really slow things down. I wouldn't expect it to be >> this >> much, but it could be part of it) >> > > This may clean up things, but I really doubt it, given the magnitude of > delays. Also, Aaron runs 4.4.5, which already has lots of the locking > removed/restructure (not to compare with the recent v5-devel, but much more > effcient than early v4 and v3). > > Rainer >> David Lang >> >> > (We're up to 18:46:51 now!) >> > >> > -Aaron >> > >> > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >> > wrote: >> >> mhhh... This is obviously not intended behavior. There are some >> rate-limiting >> >> features that can deliberately slow down a queue, but I guess you >> have not >> >> configured these. So there either is a bug that activates them at >> some point >> >> during processing (e.g. an invalid memory access could do that) or >> there is >> >> some other bug that causes the dequeue to happen very slowly. In any >> case, it >> >> is hard to guess. >> >> >> >> Given the volume of the messages, a debug log probably does not >> help. We >> >> could gain a little insight in at least the queue sizes that rsyslog >> sees via >> >> imdiag plus the (very rudamentary) v5 debug front-end (it doesn't >> require the >> >> engine to be v5!). I would need to spend at least a little work on >> the >> >> front-end to enable remote access, but that's not really a problem. >> >> >> >> One other thing is that I am still holding a v4-beta with additional >> fixes as >> >> I didn't want to put these in the middle of some other debugging. >> However, >> >> these fixes address potential memory problems, so the most >> appropriate course >> >> of action would be to give that version at least a try. It needs to >> be >> >> released in the next days in any case. >> >> >> >> I have uploaded that (pre-4.5.6) version so that you can give it a >> try if you >> >> like: >> >> >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz >> >> >> >> I think it would good if you could try it at least once, because I >> think any >> >> other troubleshooting will boil down to looking at the fixes this >> version >> >> contains. >> >> >> >> Rainer >> >> >> >>> -----Original Message----- >> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> >>> Sent: Monday, November 02, 2009 11:52 PM >> >>> To: rsyslog-users >> >>> Subject: [rsyslog] Queuing bug (4.5.5) >> >>> >> >>> Greetings all, >> >>> >> >>> I appear to have run into an issue with 4.5.5 where queues are not >> >>> being flushed in a timely manner. ?In this specific case, I have >> data >> >>> from last wednesday that is being written to disk in small chunks >> >>> today since last wednesday. ?Unfortunately I cannot be too specific >> in >> >>> details, but here is what I can see: >> >>> >> >>> Using omfile+gzip, there appears to have been a decent burst in >> >>> traffic sometime last wednesday. ?The rsyslog instance has grown to >> >>> 1.8GB of memory and stayed there for a while. ?I have both main >> >>> message and action queues defined using fixedarray, and I see no >> >>> on-disk queues (appears to be entirely in memory). >> >>> >> >>> I've got templates writing out to filenames using an hourly >> timestamp >> >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of >> those >> >>> files, all of them less than 5k in size, there are between 5 and 15 >> >>> lines of content, all of them from last wednesday, and within a few >> >>> seconds of each other. ?It's almost like there was a significant >> queue >> >>> built up, the hour rolled over, and only the first block of lines >> were >> >>> pulled from the queue. ?Then the hour rolled over again, and >> another >> >>> block of lines were pulled from the queue. ?Then the next hour, >> then >> >>> another 5-15 lines. >> >>> >> >>> Is it possible that one of the queues still has a good chunk of >> data >> >>> built up, and is flushing it out very slowly? ?It hasn't been >> >>> consistantly at the top of the hour either, and not every hour. >> ?But >> >>> the log lines themselves are sequentially written out, and usually >> the >> >>> lines are within a few seconds of each other. >> >>> >> >>> For example: >> >>> >> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >> 18:46:34 >> >>> and one 18:46:35 >> >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, 14 >> >>> lines Oct 21 18:46:36 >> >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, 4 >> >>> lines Oct 21 18:46:37 >> >>> >> >>> Thoughts? >> >>> -Aaron >> >>> _______________________________________________ >> >>> rsyslog mailing list >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>> http://www.rsyslog.com >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com >> >> >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 3 18:30:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 18:30:17 +0100 Subject: [rsyslog] Queuing bug (4.5.5) References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> first shot at the data: this looks like a full queue where the enqueue operation is waiting on a drain that does not happen (fast enough). Of course, that doesn't explain why this happens... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Tuesday, November 03, 2009 6:27 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) > > Ok - I captured an strace. This appears to be the action thread. > This specific set of calls took minutes. I'll re-run this with -t > and/or -T. > > Note that this syslog instance is not actually receiving any data right > now. > > -Aaron > > 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- > 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL > 26365 <... futex resumed> ) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} > 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) > 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL > 26365 <... futex resumed> ) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} > 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) > 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL > 26365 <... futex resumed> ) = 0 > 26365 clock_gettime(CLOCK_REALTIME, > 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 > 26365 <... futex resumed> ) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} > 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) > 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL > 26365 <... futex resumed> ) = 0 > 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL > 26365 <... futex resumed> ) = -1 EAGAIN (Resource > temporarily unavailable) > 26365 clock_gettime(CLOCK_REALTIME, > 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 > 26365 <... futex resumed> ) = 0 > 26365 clock_gettime(CLOCK_REALTIME, > 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} > 26365 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) > 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 > 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 > 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 > 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL > > > On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards > wrote: > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >> Sent: Tuesday, November 03, 2009 4:06 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Queuing bug (4.5.5) > >> > >> On Tue, 3 Nov 2009, Aaron Wiebe wrote: > >> > >> > This is still taking place this hour - though I obviously can't > >> > restart onto a newer version without losing this case. ?Is there > >> > anything I can do in the current configuration to try and debug > this > >> > situation? > >> > >> if you can strace each thread for a few seconds it may give you an > idea > >> what's happening. > > > > This is a good suggestion. > > > > It would also potentially be enlightening to attach gdb to the > process at > > various points in time and get a backtrace from all running threads. > > > > Other than that, there is little you can do on a running version. If > it were > > compiled for debug, and the environment setup so that we could > capture > > stdout/stderr, sending SIGUSR2 would yield to much the same > information as > > the gdb backtrace BUT when runtime instrumentation is enabled, the > build is > > operating at a third to a quarter of its normal speed. > > > >> in the meantime you need to stop the system from getting further > >> behind, > >> can you redirect or reconfigure the senders so that they are not > >> sending > >> new logs to this box so that it can dig itself out (stopping the > input > >> may > >> be enough by itself, rsyslog has historicly done a LOT of locking > >> around > >> the main queue, and if you have a full queue with more data trying > to > >> be > >> delivered it can really slow things down. I wouldn't expect it to be > >> this > >> much, but it could be part of it) > >> > > > > This may clean up things, but I really doubt it, given the magnitude > of > > delays. Also, Aaron runs 4.4.5, which already has lots of the locking > > removed/restructure (not to compare with the recent v5-devel, but > much more > > effcient than early v4 and v3). > > > > Rainer > >> David Lang > >> > >> > (We're up to 18:46:51 now!) > >> > > >> > -Aaron > >> > > >> > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards > >> > wrote: > >> >> mhhh... This is obviously not intended behavior. There are some > >> rate-limiting > >> >> features that can deliberately slow down a queue, but I guess you > >> have not > >> >> configured these. So there either is a bug that activates them at > >> some point > >> >> during processing (e.g. an invalid memory access could do that) > or > >> there is > >> >> some other bug that causes the dequeue to happen very slowly. In > any > >> case, it > >> >> is hard to guess. > >> >> > >> >> Given the volume of the messages, a debug log probably does not > >> help. We > >> >> could gain a little insight in at least the queue sizes that > rsyslog > >> sees via > >> >> imdiag plus the (very rudamentary) v5 debug front-end (it doesn't > >> require the > >> >> engine to be v5!). I would need to spend at least a little work > on > >> the > >> >> front-end to enable remote access, but that's not really a > problem. > >> >> > >> >> One other thing is that I am still holding a v4-beta with > additional > >> fixes as > >> >> I didn't want to put these in the middle of some other debugging. > >> However, > >> >> these fixes address potential memory problems, so the most > >> appropriate course > >> >> of action would be to give that version at least a try. It needs > to > >> be > >> >> released in the next days in any case. > >> >> > >> >> I have uploaded that (pre-4.5.6) version so that you can give it > a > >> try if you > >> >> like: > >> >> > >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz > >> >> > >> >> I think it would good if you could try it at least once, because > I > >> think any > >> >> other troubleshooting will boil down to looking at the fixes this > >> version > >> >> contains. > >> >> > >> >> Rainer > >> >> > >> >>> -----Original Message----- > >> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > >> >>> Sent: Monday, November 02, 2009 11:52 PM > >> >>> To: rsyslog-users > >> >>> Subject: [rsyslog] Queuing bug (4.5.5) > >> >>> > >> >>> Greetings all, > >> >>> > >> >>> I appear to have run into an issue with 4.5.5 where queues are > not > >> >>> being flushed in a timely manner. ?In this specific case, I have > >> data > >> >>> from last wednesday that is being written to disk in small > chunks > >> >>> today since last wednesday. ?Unfortunately I cannot be too > specific > >> in > >> >>> details, but here is what I can see: > >> >>> > >> >>> Using omfile+gzip, there appears to have been a decent burst in > >> >>> traffic sometime last wednesday. ?The rsyslog instance has grown > to > >> >>> 1.8GB of memory and stayed there for a while. ?I have both main > >> >>> message and action queues defined using fixedarray, and I see no > >> >>> on-disk queues (appears to be entirely in memory). > >> >>> > >> >>> I've got templates writing out to filenames using an hourly > >> timestamp > >> >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of > >> those > >> >>> files, all of them less than 5k in size, there are between 5 and > 15 > >> >>> lines of content, all of them from last wednesday, and within a > few > >> >>> seconds of each other. ?It's almost like there was a significant > >> queue > >> >>> built up, the hour rolled over, and only the first block of > lines > >> were > >> >>> pulled from the queue. ?Then the hour rolled over again, and > >> another > >> >>> block of lines were pulled from the queue. ?Then the next hour, > >> then > >> >>> another 5-15 lines. > >> >>> > >> >>> Is it possible that one of the queues still has a good chunk of > >> data > >> >>> built up, and is flushing it out very slowly? ?It hasn't been > >> >>> consistantly at the top of the hour either, and not every hour. > >> ?But > >> >>> the log lines themselves are sequentially written out, and > usually > >> the > >> >>> lines are within a few seconds of each other. > >> >>> > >> >>> For example: > >> >>> > >> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 > >> 18:46:34 > >> >>> and one 18:46:35 > >> >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, > 14 > >> >>> lines Oct 21 18:46:36 > >> >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, > 4 > >> >>> lines Oct 21 18:46:37 > >> >>> > >> >>> Thoughts? > >> >>> -Aaron > >> >>> _______________________________________________ > >> >>> rsyslog mailing list > >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >>> http://www.rsyslog.com > >> >> _______________________________________________ > >> >> rsyslog mailing list > >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> http://www.rsyslog.com > >> >> > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com > >> > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From epiphani at gmail.com Tue Nov 3 18:33:40 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 3 Nov 2009 12:33:40 -0500 Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> Message-ID: Here's one with times: 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, 3889000}) = 0 <0.000047> 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, 4147000}) = 0 <0.000013> 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) <2.002462> 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 250) = 250 <0.000305> 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, 7292000}) = 0 <0.000027> 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, 414932000}) = 0 <0.000048> 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, 415191000}) = 0 <0.000013> 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) <2.002220> 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 197) = 197 <0.000279> 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, 417992000}) = 0 <0.000027> 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, 608226000}) = 0 <0.000047> 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, 608486000}) = 0 <0.000014> 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} It looks like the locks are waiting a -very- long time. -Aaron On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards wrote: > first shot at the data: this looks like a full queue where the enqueue > operation is waiting on a drain that does not happen (fast enough). Of > course, that doesn't explain why this happens... > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> Sent: Tuesday, November 03, 2009 6:27 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> Ok - I captured an strace. ?This appears to be the action thread. >> This specific set of calls took minutes. ?I'll re-run this with -t >> and/or -T. >> >> Note that this syslog instance is not actually receiving any data right >> now. >> >> -Aaron >> >> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> timed out) >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> timed out) >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, ? >> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> timed out) >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource >> temporarily unavailable) >> 26365 clock_gettime(CLOCK_REALTIME, ? >> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, ? >> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> timed out) >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 >> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL >> >> >> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards >> wrote: >> >> -----Original Message----- >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >> Sent: Tuesday, November 03, 2009 4:06 PM >> >> To: rsyslog-users >> >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> >> >> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >> >> >> >> > This is still taking place this hour - though I obviously can't >> >> > restart onto a newer version without losing this case. ?Is there >> >> > anything I can do in the current configuration to try and debug >> this >> >> > situation? >> >> >> >> if you can strace each thread for a few seconds it may give you an >> idea >> >> what's happening. >> > >> > This is a good suggestion. >> > >> > It would also potentially be enlightening to attach gdb to the >> process at >> > various points in time and get a backtrace from all running threads. >> > >> > Other than that, there is little you can do on a running version. If >> it were >> > compiled for debug, and the environment setup so that we could >> capture >> > stdout/stderr, sending SIGUSR2 would yield to much the same >> information as >> > the gdb backtrace BUT when runtime instrumentation is enabled, the >> build is >> > operating at a third to a quarter of its normal speed. >> > >> >> in the meantime you need to stop the system from getting further >> >> behind, >> >> can you redirect or reconfigure the senders so that they are not >> >> sending >> >> new logs to this box so that it can dig itself out (stopping the >> input >> >> may >> >> be enough by itself, rsyslog has historicly done a LOT of locking >> >> around >> >> the main queue, and if you have a full queue with more data trying >> to >> >> be >> >> delivered it can really slow things down. I wouldn't expect it to be >> >> this >> >> much, but it could be part of it) >> >> >> > >> > This may clean up things, but I really doubt it, given the magnitude >> of >> > delays. Also, Aaron runs 4.4.5, which already has lots of the locking >> > removed/restructure (not to compare with the recent v5-devel, but >> much more >> > effcient than early v4 and v3). >> > >> > Rainer >> >> David Lang >> >> >> >> > (We're up to 18:46:51 now!) >> >> > >> >> > -Aaron >> >> > >> >> > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >> >> > wrote: >> >> >> mhhh... This is obviously not intended behavior. There are some >> >> rate-limiting >> >> >> features that can deliberately slow down a queue, but I guess you >> >> have not >> >> >> configured these. So there either is a bug that activates them at >> >> some point >> >> >> during processing (e.g. an invalid memory access could do that) >> or >> >> there is >> >> >> some other bug that causes the dequeue to happen very slowly. In >> any >> >> case, it >> >> >> is hard to guess. >> >> >> >> >> >> Given the volume of the messages, a debug log probably does not >> >> help. We >> >> >> could gain a little insight in at least the queue sizes that >> rsyslog >> >> sees via >> >> >> imdiag plus the (very rudamentary) v5 debug front-end (it doesn't >> >> require the >> >> >> engine to be v5!). I would need to spend at least a little work >> on >> >> the >> >> >> front-end to enable remote access, but that's not really a >> problem. >> >> >> >> >> >> One other thing is that I am still holding a v4-beta with >> additional >> >> fixes as >> >> >> I didn't want to put these in the middle of some other debugging. >> >> However, >> >> >> these fixes address potential memory problems, so the most >> >> appropriate course >> >> >> of action would be to give that version at least a try. It needs >> to >> >> be >> >> >> released in the next days in any case. >> >> >> >> >> >> I have uploaded that (pre-4.5.6) version so that you can give it >> a >> >> try if you >> >> >> like: >> >> >> >> >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz >> >> >> >> >> >> I think it would good if you could try it at least once, because >> I >> >> think any >> >> >> other troubleshooting will boil down to looking at the fixes this >> >> version >> >> >> contains. >> >> >> >> >> >> Rainer >> >> >> >> >> >>> -----Original Message----- >> >> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> >> >>> Sent: Monday, November 02, 2009 11:52 PM >> >> >>> To: rsyslog-users >> >> >>> Subject: [rsyslog] Queuing bug (4.5.5) >> >> >>> >> >> >>> Greetings all, >> >> >>> >> >> >>> I appear to have run into an issue with 4.5.5 where queues are >> not >> >> >>> being flushed in a timely manner. ?In this specific case, I have >> >> data >> >> >>> from last wednesday that is being written to disk in small >> chunks >> >> >>> today since last wednesday. ?Unfortunately I cannot be too >> specific >> >> in >> >> >>> details, but here is what I can see: >> >> >>> >> >> >>> Using omfile+gzip, there appears to have been a decent burst in >> >> >>> traffic sometime last wednesday. ?The rsyslog instance has grown >> to >> >> >>> 1.8GB of memory and stayed there for a while. ?I have both main >> >> >>> message and action queues defined using fixedarray, and I see no >> >> >>> on-disk queues (appears to be entirely in memory). >> >> >>> >> >> >>> I've got templates writing out to filenames using an hourly >> >> timestamp >> >> >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some of >> >> those >> >> >>> files, all of them less than 5k in size, there are between 5 and >> 15 >> >> >>> lines of content, all of them from last wednesday, and within a >> few >> >> >>> seconds of each other. ?It's almost like there was a significant >> >> queue >> >> >>> built up, the hour rolled over, and only the first block of >> lines >> >> were >> >> >>> pulled from the queue. ?Then the hour rolled over again, and >> >> another >> >> >>> block of lines were pulled from the queue. ?Then the next hour, >> >> then >> >> >>> another 5-15 lines. >> >> >>> >> >> >>> Is it possible that one of the queues still has a good chunk of >> >> data >> >> >>> built up, and is flushing it out very slowly? ?It hasn't been >> >> >>> consistantly at the top of the hour either, and not every hour. >> >> ?But >> >> >>> the log lines themselves are sequentially written out, and >> usually >> >> the >> >> >>> lines are within a few seconds of each other. >> >> >>> >> >> >>> For example: >> >> >>> >> >> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >> >> 18:46:34 >> >> >>> and one 18:46:35 >> >> >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 18:46:35, >> 14 >> >> >>> lines Oct 21 18:46:36 >> >> >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 18:46:36, >> 4 >> >> >>> lines Oct 21 18:46:37 >> >> >>> >> >> >>> Thoughts? >> >> >>> -Aaron >> >> >>> _______________________________________________ >> >> >>> rsyslog mailing list >> >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >>> http://www.rsyslog.com >> >> >> _______________________________________________ >> >> >> rsyslog mailing list >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> http://www.rsyslog.com >> >> >> >> >> > _______________________________________________ >> >> > rsyslog mailing list >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> > http://www.rsyslog.com >> >> > >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 3 18:38:40 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 18:38:40 +0100 Subject: [rsyslog] Queuing bug (4.5.5) References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> It really looks like a full queue. The 2-second wait is the default wait interval before discarding a message when a queue is full. So for some reason the action queue seems to think it is full, so that the main queue worker can no longer add any data to it. Out of the strace, I unfortunately do not see why this happens. If possible, it would be good to try the 4.5.6 release, as this may actually be caused by the memory bug that is solved in that release. If it doesn't help, we would need to think about a way to get more in-depth info when the problem happens. I have an idea, but need to check if it can be done. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Tuesday, November 03, 2009 6:34 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) > > Here's one with times: > > 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- > 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL > > 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> > 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, > 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, > 3889000}) = 0 <0.000047> > 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 > 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> > 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, > 4147000}) = 0 <0.000013> > 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} > > 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) <2.002462> > 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 > WT-GARULF6"..., 250) = 250 <0.000305> > 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> > 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, > 7292000}) = 0 <0.000027> > 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL > > 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> > 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, > 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, > 414932000}) = 0 <0.000048> > 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 > 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> > 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, > 415191000}) = 0 <0.000013> > 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} > > 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection > timed out) <2.002220> > 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 > WT-GARULF6"..., 197) = 197 <0.000279> > 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> > 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, > 417992000}) = 0 <0.000027> > 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL > > 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> > 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, > 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, > 608226000}) = 0 <0.000047> > 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 > 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> > 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, > 608486000}) = 0 <0.000014> > 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} > > > It looks like the locks are waiting a -very- long time. > > -Aaron > > On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards > wrote: > > first shot at the data: this looks like a full queue where the > enqueue > > operation is waiting on a drain that does not happen (fast enough). > Of > > course, that doesn't explain why this happens... > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > >> Sent: Tuesday, November 03, 2009 6:27 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Queuing bug (4.5.5) > >> > >> Ok - I captured an strace. ?This appears to be the action thread. > >> This specific set of calls took minutes. ?I'll re-run this with -t > >> and/or -T. > >> > >> Note that this syslog instance is not actually receiving any data > right > >> now. > >> > >> -Aaron > >> > >> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- > >> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} ...> > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection > >> timed out) > >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} ...> > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection > >> timed out) > >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, ? > >> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} ...> > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection > >> timed out) > >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource > >> temporarily unavailable) > >> 26365 clock_gettime(CLOCK_REALTIME, ? > >> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, ? > >> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} ...> > >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection > >> timed out) > >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 > >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 > >> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 > >> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL > >> > >> > >> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards > >> wrote: > >> >> -----Original Message----- > >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >> >> Sent: Tuesday, November 03, 2009 4:06 PM > >> >> To: rsyslog-users > >> >> Subject: Re: [rsyslog] Queuing bug (4.5.5) > >> >> > >> >> On Tue, 3 Nov 2009, Aaron Wiebe wrote: > >> >> > >> >> > This is still taking place this hour - though I obviously can't > >> >> > restart onto a newer version without losing this case. ?Is > there > >> >> > anything I can do in the current configuration to try and debug > >> this > >> >> > situation? > >> >> > >> >> if you can strace each thread for a few seconds it may give you > an > >> idea > >> >> what's happening. > >> > > >> > This is a good suggestion. > >> > > >> > It would also potentially be enlightening to attach gdb to the > >> process at > >> > various points in time and get a backtrace from all running > threads. > >> > > >> > Other than that, there is little you can do on a running version. > If > >> it were > >> > compiled for debug, and the environment setup so that we could > >> capture > >> > stdout/stderr, sending SIGUSR2 would yield to much the same > >> information as > >> > the gdb backtrace BUT when runtime instrumentation is enabled, the > >> build is > >> > operating at a third to a quarter of its normal speed. > >> > > >> >> in the meantime you need to stop the system from getting further > >> >> behind, > >> >> can you redirect or reconfigure the senders so that they are not > >> >> sending > >> >> new logs to this box so that it can dig itself out (stopping the > >> input > >> >> may > >> >> be enough by itself, rsyslog has historicly done a LOT of locking > >> >> around > >> >> the main queue, and if you have a full queue with more data > trying > >> to > >> >> be > >> >> delivered it can really slow things down. I wouldn't expect it to > be > >> >> this > >> >> much, but it could be part of it) > >> >> > >> > > >> > This may clean up things, but I really doubt it, given the > magnitude > >> of > >> > delays. Also, Aaron runs 4.4.5, which already has lots of the > locking > >> > removed/restructure (not to compare with the recent v5-devel, but > >> much more > >> > effcient than early v4 and v3). > >> > > >> > Rainer > >> >> David Lang > >> >> > >> >> > (We're up to 18:46:51 now!) > >> >> > > >> >> > -Aaron > >> >> > > >> >> > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards > >> >> > wrote: > >> >> >> mhhh... This is obviously not intended behavior. There are > some > >> >> rate-limiting > >> >> >> features that can deliberately slow down a queue, but I guess > you > >> >> have not > >> >> >> configured these. So there either is a bug that activates them > at > >> >> some point > >> >> >> during processing (e.g. an invalid memory access could do > that) > >> or > >> >> there is > >> >> >> some other bug that causes the dequeue to happen very slowly. > In > >> any > >> >> case, it > >> >> >> is hard to guess. > >> >> >> > >> >> >> Given the volume of the messages, a debug log probably does > not > >> >> help. We > >> >> >> could gain a little insight in at least the queue sizes that > >> rsyslog > >> >> sees via > >> >> >> imdiag plus the (very rudamentary) v5 debug front-end (it > doesn't > >> >> require the > >> >> >> engine to be v5!). I would need to spend at least a little > work > >> on > >> >> the > >> >> >> front-end to enable remote access, but that's not really a > >> problem. > >> >> >> > >> >> >> One other thing is that I am still holding a v4-beta with > >> additional > >> >> fixes as > >> >> >> I didn't want to put these in the middle of some other > debugging. > >> >> However, > >> >> >> these fixes address potential memory problems, so the most > >> >> appropriate course > >> >> >> of action would be to give that version at least a try. It > needs > >> to > >> >> be > >> >> >> released in the next days in any case. > >> >> >> > >> >> >> I have uploaded that (pre-4.5.6) version so that you can give > it > >> a > >> >> try if you > >> >> >> like: > >> >> >> > >> >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog- > 4.5.6.tar.gz > >> >> >> > >> >> >> I think it would good if you could try it at least once, > because > >> I > >> >> think any > >> >> >> other troubleshooting will boil down to looking at the fixes > this > >> >> version > >> >> >> contains. > >> >> >> > >> >> >> Rainer > >> >> >> > >> >> >>> -----Original Message----- > >> >> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> >> >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > >> >> >>> Sent: Monday, November 02, 2009 11:52 PM > >> >> >>> To: rsyslog-users > >> >> >>> Subject: [rsyslog] Queuing bug (4.5.5) > >> >> >>> > >> >> >>> Greetings all, > >> >> >>> > >> >> >>> I appear to have run into an issue with 4.5.5 where queues > are > >> not > >> >> >>> being flushed in a timely manner. ?In this specific case, I > have > >> >> data > >> >> >>> from last wednesday that is being written to disk in small > >> chunks > >> >> >>> today since last wednesday. ?Unfortunately I cannot be too > >> specific > >> >> in > >> >> >>> details, but here is what I can see: > >> >> >>> > >> >> >>> Using omfile+gzip, there appears to have been a decent burst > in > >> >> >>> traffic sometime last wednesday. ?The rsyslog instance has > grown > >> to > >> >> >>> 1.8GB of memory and stayed there for a while. ?I have both > main > >> >> >>> message and action queues defined using fixedarray, and I see > no > >> >> >>> on-disk queues (appears to be entirely in memory). > >> >> >>> > >> >> >>> I've got templates writing out to filenames using an hourly > >> >> timestamp > >> >> >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some > of > >> >> those > >> >> >>> files, all of them less than 5k in size, there are between 5 > and > >> 15 > >> >> >>> lines of content, all of them from last wednesday, and within > a > >> few > >> >> >>> seconds of each other. ?It's almost like there was a > significant > >> >> queue > >> >> >>> built up, the hour rolled over, and only the first block of > >> lines > >> >> were > >> >> >>> pulled from the queue. ?Then the hour rolled over again, and > >> >> another > >> >> >>> block of lines were pulled from the queue. ?Then the next > hour, > >> >> then > >> >> >>> another 5-15 lines. > >> >> >>> > >> >> >>> Is it possible that one of the queues still has a good chunk > of > >> >> data > >> >> >>> built up, and is flushing it out very slowly? ?It hasn't been > >> >> >>> consistantly at the top of the hour either, and not every > hour. > >> >> ?But > >> >> >>> the log lines themselves are sequentially written out, and > >> usually > >> >> the > >> >> >>> lines are within a few seconds of each other. > >> >> >>> > >> >> >>> For example: > >> >> >>> > >> >> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 > >> >> 18:46:34 > >> >> >>> and one 18:46:35 > >> >> >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 > 18:46:35, > >> 14 > >> >> >>> lines Oct 21 18:46:36 > >> >> >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 > 18:46:36, > >> 4 > >> >> >>> lines Oct 21 18:46:37 > >> >> >>> > >> >> >>> Thoughts? > >> >> >>> -Aaron > >> >> >>> _______________________________________________ > >> >> >>> rsyslog mailing list > >> >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> >>> http://www.rsyslog.com > >> >> >> _______________________________________________ > >> >> >> rsyslog mailing list > >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> >> http://www.rsyslog.com > >> >> >> > >> >> > _______________________________________________ > >> >> > rsyslog mailing list > >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> > http://www.rsyslog.com > >> >> > > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com > >> > > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From epiphani at gmail.com Tue Nov 3 18:41:46 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 3 Nov 2009 12:41:46 -0500 Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> Message-ID: I'll see what I can do. I'm actually somewhat tied up with other stuff this week, but I may be able to find time to update. -Aaron On Tue, Nov 3, 2009 at 12:38 PM, Rainer Gerhards wrote: > It really looks like a full queue. The 2-second wait is the default wait > interval before discarding a message when a queue is full. So for some reason > the action queue seems to think it is full, so that the main queue worker can > no longer add any data to it. Out of the strace, I unfortunately do not see > why this happens. > > If possible, it would be good to try the 4.5.6 release, as this may actually > be caused by the memory bug that is solved in that release. If it doesn't > help, we would need to think about a way to get more in-depth info when the > problem happens. I have an idea, but need to check if it can be done. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> Sent: Tuesday, November 03, 2009 6:34 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> Here's one with times: >> >> 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >> 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL >> >> 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> >> 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, ? >> 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, >> 3889000}) = 0 <0.000047> >> 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> >> 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, >> 4147000}) = 0 <0.000013> >> 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} >> >> 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection >> timed out) <2.002462> >> 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 >> WT-GARULF6"..., 250) = 250 <0.000305> >> 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> >> 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, >> 7292000}) = 0 <0.000027> >> 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL >> >> 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> >> 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, ? >> 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, >> 414932000}) = 0 <0.000048> >> 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> >> 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, >> 415191000}) = 0 <0.000013> >> 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} >> >> 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection >> timed out) <2.002220> >> 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 >> WT-GARULF6"..., 197) = 197 <0.000279> >> 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> >> 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, >> 417992000}) = 0 <0.000027> >> 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL >> >> 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> >> 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, ? >> 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, >> 608226000}) = 0 <0.000047> >> 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> >> 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, >> 608486000}) = 0 <0.000014> >> 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} >> >> >> It looks like the locks are waiting a -very- long time. >> >> -Aaron >> >> On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards >> wrote: >> > first shot at the data: this looks like a full queue where the >> enqueue >> > operation is waiting on a drain that does not happen (fast enough). >> Of >> > course, that doesn't explain why this happens... >> > >> > Rainer >> > >> >> -----Original Message----- >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> >> Sent: Tuesday, November 03, 2009 6:27 PM >> >> To: rsyslog-users >> >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> >> >> Ok - I captured an strace. ?This appears to be the action thread. >> >> This specific set of calls took minutes. ?I'll re-run this with -t >> >> and/or -T. >> >> >> >> Note that this syslog instance is not actually receiving any data >> right >> >> now. >> >> >> >> -Aaron >> >> >> >> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} > ...> >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> >> timed out) >> >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} > ...> >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> >> timed out) >> >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, ? >> >> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} > ...> >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> >> timed out) >> >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource >> >> temporarily unavailable) >> >> 26365 clock_gettime(CLOCK_REALTIME, ? >> >> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, ? >> >> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} > ...> >> >> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >> >> timed out) >> >> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 >> >> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >> >> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 >> >> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL >> >> >> >> >> >> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards >> >> wrote: >> >> >> -----Original Message----- >> >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >> >> Sent: Tuesday, November 03, 2009 4:06 PM >> >> >> To: rsyslog-users >> >> >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> >> >> >> >> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >> >> >> >> >> >> > This is still taking place this hour - though I obviously can't >> >> >> > restart onto a newer version without losing this case. ?Is >> there >> >> >> > anything I can do in the current configuration to try and debug >> >> this >> >> >> > situation? >> >> >> >> >> >> if you can strace each thread for a few seconds it may give you >> an >> >> idea >> >> >> what's happening. >> >> > >> >> > This is a good suggestion. >> >> > >> >> > It would also potentially be enlightening to attach gdb to the >> >> process at >> >> > various points in time and get a backtrace from all running >> threads. >> >> > >> >> > Other than that, there is little you can do on a running version. >> If >> >> it were >> >> > compiled for debug, and the environment setup so that we could >> >> capture >> >> > stdout/stderr, sending SIGUSR2 would yield to much the same >> >> information as >> >> > the gdb backtrace BUT when runtime instrumentation is enabled, the >> >> build is >> >> > operating at a third to a quarter of its normal speed. >> >> > >> >> >> in the meantime you need to stop the system from getting further >> >> >> behind, >> >> >> can you redirect or reconfigure the senders so that they are not >> >> >> sending >> >> >> new logs to this box so that it can dig itself out (stopping the >> >> input >> >> >> may >> >> >> be enough by itself, rsyslog has historicly done a LOT of locking >> >> >> around >> >> >> the main queue, and if you have a full queue with more data >> trying >> >> to >> >> >> be >> >> >> delivered it can really slow things down. I wouldn't expect it to >> be >> >> >> this >> >> >> much, but it could be part of it) >> >> >> >> >> > >> >> > This may clean up things, but I really doubt it, given the >> magnitude >> >> of >> >> > delays. Also, Aaron runs 4.4.5, which already has lots of the >> locking >> >> > removed/restructure (not to compare with the recent v5-devel, but >> >> much more >> >> > effcient than early v4 and v3). >> >> > >> >> > Rainer >> >> >> David Lang >> >> >> >> >> >> > (We're up to 18:46:51 now!) >> >> >> > >> >> >> > -Aaron >> >> >> > >> >> >> > On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >> >> >> > wrote: >> >> >> >> mhhh... This is obviously not intended behavior. There are >> some >> >> >> rate-limiting >> >> >> >> features that can deliberately slow down a queue, but I guess >> you >> >> >> have not >> >> >> >> configured these. So there either is a bug that activates them >> at >> >> >> some point >> >> >> >> during processing (e.g. an invalid memory access could do >> that) >> >> or >> >> >> there is >> >> >> >> some other bug that causes the dequeue to happen very slowly. >> In >> >> any >> >> >> case, it >> >> >> >> is hard to guess. >> >> >> >> >> >> >> >> Given the volume of the messages, a debug log probably does >> not >> >> >> help. We >> >> >> >> could gain a little insight in at least the queue sizes that >> >> rsyslog >> >> >> sees via >> >> >> >> imdiag plus the (very rudamentary) v5 debug front-end (it >> doesn't >> >> >> require the >> >> >> >> engine to be v5!). I would need to spend at least a little >> work >> >> on >> >> >> the >> >> >> >> front-end to enable remote access, but that's not really a >> >> problem. >> >> >> >> >> >> >> >> One other thing is that I am still holding a v4-beta with >> >> additional >> >> >> fixes as >> >> >> >> I didn't want to put these in the middle of some other >> debugging. >> >> >> However, >> >> >> >> these fixes address potential memory problems, so the most >> >> >> appropriate course >> >> >> >> of action would be to give that version at least a try. It >> needs >> >> to >> >> >> be >> >> >> >> released in the next days in any case. >> >> >> >> >> >> >> >> I have uploaded that (pre-4.5.6) version so that you can give >> it >> >> a >> >> >> try if you >> >> >> >> like: >> >> >> >> >> >> >> >> http://www.rsyslog.com/download/rsyslog/pre/rsyslog- >> 4.5.6.tar.gz >> >> >> >> >> >> >> >> I think it would good if you could try it at least once, >> because >> >> I >> >> >> think any >> >> >> >> other troubleshooting will boil down to looking at the fixes >> this >> >> >> version >> >> >> >> contains. >> >> >> >> >> >> >> >> Rainer >> >> >> >> >> >> >> >>> -----Original Message----- >> >> >> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> >> >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> >> >> >>> Sent: Monday, November 02, 2009 11:52 PM >> >> >> >>> To: rsyslog-users >> >> >> >>> Subject: [rsyslog] Queuing bug (4.5.5) >> >> >> >>> >> >> >> >>> Greetings all, >> >> >> >>> >> >> >> >>> I appear to have run into an issue with 4.5.5 where queues >> are >> >> not >> >> >> >>> being flushed in a timely manner. ?In this specific case, I >> have >> >> >> data >> >> >> >>> from last wednesday that is being written to disk in small >> >> chunks >> >> >> >>> today since last wednesday. ?Unfortunately I cannot be too >> >> specific >> >> >> in >> >> >> >>> details, but here is what I can see: >> >> >> >>> >> >> >> >>> Using omfile+gzip, there appears to have been a decent burst >> in >> >> >> >>> traffic sometime last wednesday. ?The rsyslog instance has >> grown >> >> to >> >> >> >>> 1.8GB of memory and stayed there for a while. ?I have both >> main >> >> >> >>> message and action queues defined using fixedarray, and I see >> no >> >> >> >>> on-disk queues (appears to be entirely in memory). >> >> >> >>> >> >> >> >>> I've got templates writing out to filenames using an hourly >> >> >> timestamp >> >> >> >>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some >> of >> >> >> those >> >> >> >>> files, all of them less than 5k in size, there are between 5 >> and >> >> 15 >> >> >> >>> lines of content, all of them from last wednesday, and within >> a >> >> few >> >> >> >>> seconds of each other. ?It's almost like there was a >> significant >> >> >> queue >> >> >> >>> built up, the hour rolled over, and only the first block of >> >> lines >> >> >> were >> >> >> >>> pulled from the queue. ?Then the hour rolled over again, and >> >> >> another >> >> >> >>> block of lines were pulled from the queue. ?Then the next >> hour, >> >> >> then >> >> >> >>> another 5-15 lines. >> >> >> >>> >> >> >> >>> Is it possible that one of the queues still has a good chunk >> of >> >> >> data >> >> >> >>> built up, and is flushing it out very slowly? ?It hasn't been >> >> >> >>> consistantly at the top of the hour either, and not every >> hour. >> >> >> ?But >> >> >> >>> the log lines themselves are sequentially written out, and >> >> usually >> >> >> the >> >> >> >>> lines are within a few seconds of each other. >> >> >> >>> >> >> >> >>> For example: >> >> >> >>> >> >> >> >>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >> >> >> 18:46:34 >> >> >> >>> and one 18:46:35 >> >> >> >>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 >> 18:46:35, >> >> 14 >> >> >> >>> lines Oct 21 18:46:36 >> >> >> >>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 >> 18:46:36, >> >> 4 >> >> >> >>> lines Oct 21 18:46:37 >> >> >> >>> >> >> >> >>> Thoughts? >> >> >> >>> -Aaron >> >> >> >>> _______________________________________________ >> >> >> >>> rsyslog mailing list >> >> >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> >>> http://www.rsyslog.com >> >> >> >> _______________________________________________ >> >> >> >> rsyslog mailing list >> >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> >> http://www.rsyslog.com >> >> >> >> >> >> >> > _______________________________________________ >> >> >> > rsyslog mailing list >> >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> > http://www.rsyslog.com >> >> >> > >> >> > _______________________________________________ >> >> > rsyslog mailing list >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> > http://www.rsyslog.com >> >> > >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From Luis.Fernando.Munoz.Mejias at cern.ch Tue Nov 3 19:04:36 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?utf-8?q?Mu=C3=B1oz_Mej=C3=ADas?=) Date: Tue, 3 Nov 2009 19:04:36 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5? Message-ID: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> Hello, world I must be doing something really dumb. I need to send some output to a named pipe that I have already created, but rsyslog is refusing to open it. My configuration states: *.* |/var/log/external/all/unknown;RSYSLOG_FileFormat And, of course, the named pipe exists: # ls -Z /var/log/external/all/unknown prw-r----- root logreaders user_u:object_r:var_log_t /var/log/external/all/unknown But when I start the syslog daemon it complains with this message: rsyslogd-2039: Could no open output file '|/var/log/external/all/unknown' [try http://www.rsyslog.com/e/2039 ] It may be a SELinux problem, but the logs show nothing about any denied requests. And the context for the named pipe is the exact same I have for the log files, which lie in the same directory as well. Am I the only one experiencing such problems? Thanks. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From david at lang.hm Tue Nov 3 19:20:57 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 3 Nov 2009 10:20:57 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> Message-ID: what is the configuration? does it have seperate action queues defined? if you have some way to stop the input, it would let the main queue drop down from being full (and eliminate the input modules from needing to lock it to try and deliver messages) I did see some times during my testing of 4.x stuff where the throughput would drop drasticly when the queue was full and there ws more incoming traffic hammering on it. it would drop from 500K logs/min to ~3k logs/min with a trivial config, slowing down the input would let it get out of this mode and speed up again after a little while. so much other stuff was happening at the time that I don't think that I reported it. David Lang On Tue, 3 Nov 2009, Rainer Gerhards wrote: > Date: Tue, 3 Nov 2009 18:38:40 +0100 > From: Rainer Gerhards > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) > > It really looks like a full queue. The 2-second wait is the default wait > interval before discarding a message when a queue is full. So for some reason > the action queue seems to think it is full, so that the main queue worker can > no longer add any data to it. Out of the strace, I unfortunately do not see > why this happens. > > If possible, it would be good to try the 4.5.6 release, as this may actually > be caused by the memory bug that is solved in that release. If it doesn't > help, we would need to think about a way to get more in-depth info when the > problem happens. I have an idea, but need to check if it can be done. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> Sent: Tuesday, November 03, 2009 6:34 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> Here's one with times: >> >> 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >> 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL >> >> 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> >> 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, >> 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, >> 3889000}) = 0 <0.000047> >> 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> >> 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, >> 4147000}) = 0 <0.000013> >> 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} >> >> 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection >> timed out) <2.002462> >> 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 >> WT-GARULF6"..., 250) = 250 <0.000305> >> 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> >> 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, >> 7292000}) = 0 <0.000027> >> 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL >> >> 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> >> 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, >> 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, >> 414932000}) = 0 <0.000048> >> 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> >> 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, >> 415191000}) = 0 <0.000013> >> 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} >> >> 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection >> timed out) <2.002220> >> 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 >> WT-GARULF6"..., 197) = 197 <0.000279> >> 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> >> 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, >> 417992000}) = 0 <0.000027> >> 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL >> >> 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> >> 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, >> 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, >> 608226000}) = 0 <0.000047> >> 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 >> 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> >> 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, >> 608486000}) = 0 <0.000014> >> 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} >> >> >> It looks like the locks are waiting a -very- long time. >> >> -Aaron >> >> On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards >> wrote: >>> first shot at the data: this looks like a full queue where the >> enqueue >>> operation is waiting on a drain that does not happen (fast enough). >> Of >>> course, that doesn't explain why this happens... >>> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>> Sent: Tuesday, November 03, 2009 6:27 PM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>> >>>> Ok - I captured an strace. ?This appears to be the action thread. >>>> This specific set of calls took minutes. ?I'll re-run this with -t >>>> and/or -T. >>>> >>>> Note that this syslog instance is not actually receiving any data >> right >>>> now. >>>> >>>> -Aaron >>>> >>>> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} > ...> >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>> timed out) >>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} > ...> >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>> timed out) >>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} > ...> >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>> timed out) >>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource >>>> temporarily unavailable) >>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} > ...> >>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>> timed out) >>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 >>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 >>>> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL >>>> >>>> >>>> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards >>>> wrote: >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>> Sent: Tuesday, November 03, 2009 4:06 PM >>>>>> To: rsyslog-users >>>>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>>>> >>>>>> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >>>>>> >>>>>>> This is still taking place this hour - though I obviously can't >>>>>>> restart onto a newer version without losing this case. ?Is >> there >>>>>>> anything I can do in the current configuration to try and debug >>>> this >>>>>>> situation? >>>>>> >>>>>> if you can strace each thread for a few seconds it may give you >> an >>>> idea >>>>>> what's happening. >>>>> >>>>> This is a good suggestion. >>>>> >>>>> It would also potentially be enlightening to attach gdb to the >>>> process at >>>>> various points in time and get a backtrace from all running >> threads. >>>>> >>>>> Other than that, there is little you can do on a running version. >> If >>>> it were >>>>> compiled for debug, and the environment setup so that we could >>>> capture >>>>> stdout/stderr, sending SIGUSR2 would yield to much the same >>>> information as >>>>> the gdb backtrace BUT when runtime instrumentation is enabled, the >>>> build is >>>>> operating at a third to a quarter of its normal speed. >>>>> >>>>>> in the meantime you need to stop the system from getting further >>>>>> behind, >>>>>> can you redirect or reconfigure the senders so that they are not >>>>>> sending >>>>>> new logs to this box so that it can dig itself out (stopping the >>>> input >>>>>> may >>>>>> be enough by itself, rsyslog has historicly done a LOT of locking >>>>>> around >>>>>> the main queue, and if you have a full queue with more data >> trying >>>> to >>>>>> be >>>>>> delivered it can really slow things down. I wouldn't expect it to >> be >>>>>> this >>>>>> much, but it could be part of it) >>>>>> >>>>> >>>>> This may clean up things, but I really doubt it, given the >> magnitude >>>> of >>>>> delays. Also, Aaron runs 4.4.5, which already has lots of the >> locking >>>>> removed/restructure (not to compare with the recent v5-devel, but >>>> much more >>>>> effcient than early v4 and v3). >>>>> >>>>> Rainer >>>>>> David Lang >>>>>> >>>>>>> (We're up to 18:46:51 now!) >>>>>>> >>>>>>> -Aaron >>>>>>> >>>>>>> On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >>>>>>> wrote: >>>>>>>> mhhh... This is obviously not intended behavior. There are >> some >>>>>> rate-limiting >>>>>>>> features that can deliberately slow down a queue, but I guess >> you >>>>>> have not >>>>>>>> configured these. So there either is a bug that activates them >> at >>>>>> some point >>>>>>>> during processing (e.g. an invalid memory access could do >> that) >>>> or >>>>>> there is >>>>>>>> some other bug that causes the dequeue to happen very slowly. >> In >>>> any >>>>>> case, it >>>>>>>> is hard to guess. >>>>>>>> >>>>>>>> Given the volume of the messages, a debug log probably does >> not >>>>>> help. We >>>>>>>> could gain a little insight in at least the queue sizes that >>>> rsyslog >>>>>> sees via >>>>>>>> imdiag plus the (very rudamentary) v5 debug front-end (it >> doesn't >>>>>> require the >>>>>>>> engine to be v5!). I would need to spend at least a little >> work >>>> on >>>>>> the >>>>>>>> front-end to enable remote access, but that's not really a >>>> problem. >>>>>>>> >>>>>>>> One other thing is that I am still holding a v4-beta with >>>> additional >>>>>> fixes as >>>>>>>> I didn't want to put these in the middle of some other >> debugging. >>>>>> However, >>>>>>>> these fixes address potential memory problems, so the most >>>>>> appropriate course >>>>>>>> of action would be to give that version at least a try. It >> needs >>>> to >>>>>> be >>>>>>>> released in the next days in any case. >>>>>>>> >>>>>>>> I have uploaded that (pre-4.5.6) version so that you can give >> it >>>> a >>>>>> try if you >>>>>>>> like: >>>>>>>> >>>>>>>> http://www.rsyslog.com/download/rsyslog/pre/rsyslog- >> 4.5.6.tar.gz >>>>>>>> >>>>>>>> I think it would good if you could try it at least once, >> because >>>> I >>>>>> think any >>>>>>>> other troubleshooting will boil down to looking at the fixes >> this >>>>>> version >>>>>>>> contains. >>>>>>>> >>>>>>>> Rainer >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>>>>>>> Sent: Monday, November 02, 2009 11:52 PM >>>>>>>>> To: rsyslog-users >>>>>>>>> Subject: [rsyslog] Queuing bug (4.5.5) >>>>>>>>> >>>>>>>>> Greetings all, >>>>>>>>> >>>>>>>>> I appear to have run into an issue with 4.5.5 where queues >> are >>>> not >>>>>>>>> being flushed in a timely manner. ?In this specific case, I >> have >>>>>> data >>>>>>>>> from last wednesday that is being written to disk in small >>>> chunks >>>>>>>>> today since last wednesday. ?Unfortunately I cannot be too >>>> specific >>>>>> in >>>>>>>>> details, but here is what I can see: >>>>>>>>> >>>>>>>>> Using omfile+gzip, there appears to have been a decent burst >> in >>>>>>>>> traffic sometime last wednesday. ?The rsyslog instance has >> grown >>>> to >>>>>>>>> 1.8GB of memory and stayed there for a while. ?I have both >> main >>>>>>>>> message and action queues defined using fixedarray, and I see >> no >>>>>>>>> on-disk queues (appears to be entirely in memory). >>>>>>>>> >>>>>>>>> I've got templates writing out to filenames using an hourly >>>>>> timestamp >>>>>>>>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some >> of >>>>>> those >>>>>>>>> files, all of them less than 5k in size, there are between 5 >> and >>>> 15 >>>>>>>>> lines of content, all of them from last wednesday, and within >> a >>>> few >>>>>>>>> seconds of each other. ?It's almost like there was a >> significant >>>>>> queue >>>>>>>>> built up, the hour rolled over, and only the first block of >>>> lines >>>>>> were >>>>>>>>> pulled from the queue. ?Then the hour rolled over again, and >>>>>> another >>>>>>>>> block of lines were pulled from the queue. ?Then the next >> hour, >>>>>> then >>>>>>>>> another 5-15 lines. >>>>>>>>> >>>>>>>>> Is it possible that one of the queues still has a good chunk >> of >>>>>> data >>>>>>>>> built up, and is flushing it out very slowly? ?It hasn't been >>>>>>>>> consistantly at the top of the hour either, and not every >> hour. >>>>>> ?But >>>>>>>>> the log lines themselves are sequentially written out, and >>>> usually >>>>>> the >>>>>>>>> lines are within a few seconds of each other. >>>>>>>>> >>>>>>>>> For example: >>>>>>>>> >>>>>>>>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >>>>>> 18:46:34 >>>>>>>>> and one 18:46:35 >>>>>>>>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 >> 18:46:35, >>>> 14 >>>>>>>>> lines Oct 21 18:46:36 >>>>>>>>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 >> 18:46:36, >>>> 4 >>>>>>>>> lines Oct 21 18:46:37 >>>>>>>>> >>>>>>>>> Thoughts? >>>>>>>>> -Aaron >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Tue Nov 3 19:52:12 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 3 Nov 2009 10:52:12 -0800 (PST) Subject: [rsyslog] queue configuration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103318@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103318@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 3 Nov 2009, Rainer Gerhards wrote: > David and all, > > I have created a new output plugin, omruleset, which permits to nest > rulesets. What it does is copy a message over from one ruleset to another and > make it process in parallel. Some more details in the doc: > > http://www.rsyslog.com/doc-omruleset.html this looks very interesting, however there is one thing mentioned here that concerns me. it sounds like you are saying that if you have multiple rules that write to the same file that you may get the content of various lines mixed togeather. am I understanding this correctly? if so, is there any way to work around this? David Lang > It permits to do what you asked for below. It also permits to do many more > things and create very advanced configurations. But one must be VERY careful > to receive the intended results. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Friday, October 23, 2009 10:23 PM >> To: rsyslog-users >> Subject: [rsyslog] queue configuration >> >> I know that you can create an action queue for a specific output. >> >> is there any way to create an action queue for multiple outputs? >> >> for example, in my configuration I have >> >> :fromhost, !isequal, "127.0.0.1" >> /var/log/messages;TraditionalFormat >> :fromhost, isequal, "127.0.0.1" >> @192.168.1.1;TraditionalForwardFormat >> *.* @192.168.1.115 >> *.* @192.168.1.241 >> *.* @192.168.1.242 >> *.* @192.168.1.6 >> *.* @192.168.1.7 >> *.* @192.168.1.122 >> :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 >> & @192.168.1.1;fixsnareForwardFormat2 >> & ~ >> :syslogtag, startswith, "MSWinEventLog#011" >> /var/log/messages;fixsnareFormat >> & @192.168.1.1;fixsnareForwardFormat >> & ~ >> *.* /var/log/messages;TraditionalFormat >> *.* @192.168.1.1 >> >> it would be nice to move the outbound relays off to a different queue, >> and >> to put the list of rules that have order dependancies (because they >> output >> in different formats to fix up formatting problem) to a seperate queue >> to >> spread the work across multiple processes and not slow down the basic >> writing to file. >> >> but as far as I can tell I would have to create a queue for each >> individual item, and I can't create a queue for the part of the config >> where I need to discard. >> >> am I missing something (including a better way to do everything ;-) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 3 20:20:38 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 20:20:38 +0100 Subject: [rsyslog] queue configuration Message-ID: <000201ca5cba$d08238ff$100013ac@intern.adiscon.com> Yes, thats right if you use the new buffered mode. The cure is either not to do that or use the new omruleset. ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 03.11.09 19:52 Betreff: Re: [rsyslog] queue configuration On Tue, 3 Nov 2009, Rainer Gerhards wrote: > David and all, > > I have created a new output plugin, omruleset, which permits to nest > rulesets. What it does is copy a message over from one ruleset to another and > make it process in parallel. Some more details in the doc: > > http://www.rsyslog.com/doc-omruleset.html this looks very interesting, however there is one thing mentioned here that concerns me. it sounds like you are saying that if you have multiple rules that write to the same file that you may get the content of various lines mixed togeather. am I understanding this correctly? if so, is there any way to work around this? David Lang > It permits to do what you asked for below. It also permits to do many more > things and create very advanced configurations. But one must be VERY careful > to receive the intended results. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Friday, October 23, 2009 10:23 PM >> To: rsyslog-users >> Subject: [rsyslog] queue configuration >> >> I know that you can create an action queue for a specific output. >> >> is there any way to create an action queue for multiple outputs? >> >> for example, in my configuration I have >> >> :fromhost, !isequal, "127.0.0.1" >> /var/log/messages;TraditionalFormat >> :fromhost, isequal, "127.0.0.1" >> @192.168.1.1;TraditionalForwardFormat >> *.* @192.168.1.115 >> *.* @192.168.1.241 >> *.* @192.168.1.242 >> *.* @192.168.1.6 >> *.* @192.168.1.7 >> *.* @192.168.1.122 >> :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 >> & @192.168.1.1;fixsnareForwardFormat2 >> & ~ >> :syslogtag, startswith, "MSWinEventLog#011" >> /var/log/messages;fixsnareFormat >> & @192.168.1.1;fixsnareForwardFormat >> & ~ >> *.* /var/log/messages;TraditionalFormat >> *.* @192.168.1.1 >> >> it would be nice to move the outbound relays off to a different queue, >> and >> to put the list of rules that have order dependancies (because they >> output >> in different formats to fix up formatting problem) to a seperate queue >> to >> spread the work across multiple processes and not slow down the basic >> writing to file. >> >> but as far as I can tell I would have to create a queue for each >> individual item, and I can't create a queue for the part of the config >> where I need to discard. >> >> am I missing something (including a better way to do everything ;-) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Tue Nov 3 20:29:30 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 3 Nov 2009 11:29:30 -0800 (PST) Subject: [rsyslog] queue configuration In-Reply-To: <000201ca5cba$d08238ff$100013ac@intern.adiscon.com> References: <000201ca5cba$d08238ff$100013ac@intern.adiscon.com> Message-ID: On Tue, 3 Nov 2009, Rainer Gerhards wrote: > Yes, thats right if you use the new buffered mode. The cure is either not to do that or use the new omruleset. so the new buffered mode even without seperate action queues or omruleset can have this problem? David Lang > ----- Urspr?ngliche Nachricht ----- > Von: "david at lang.hm" > An: "rsyslog-users" > Gesendet: 03.11.09 19:52 > Betreff: Re: [rsyslog] queue configuration > > On Tue, 3 Nov 2009, Rainer Gerhards wrote: > >> David and all, >> >> I have created a new output plugin, omruleset, which permits to nest >> rulesets. What it does is copy a message over from one ruleset to another and >> make it process in parallel. Some more details in the doc: >> >> http://www.rsyslog.com/doc-omruleset.html > > this looks very interesting, however there is one thing mentioned here > that concerns me. > > it sounds like you are saying that if you have multiple rules that write > to the same file that you may get the content of various lines mixed > togeather. > > am I understanding this correctly? > > if so, is there any way to work around this? > > David Lang > >> It permits to do what you asked for below. It also permits to do many more >> things and create very advanced configurations. But one must be VERY careful >> to receive the intended results. >> >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Friday, October 23, 2009 10:23 PM >>> To: rsyslog-users >>> Subject: [rsyslog] queue configuration >>> >>> I know that you can create an action queue for a specific output. >>> >>> is there any way to create an action queue for multiple outputs? >>> >>> for example, in my configuration I have >>> >>> :fromhost, !isequal, "127.0.0.1" >>> /var/log/messages;TraditionalFormat >>> :fromhost, isequal, "127.0.0.1" >>> @192.168.1.1;TraditionalForwardFormat >>> *.* @192.168.1.115 >>> *.* @192.168.1.241 >>> *.* @192.168.1.242 >>> *.* @192.168.1.6 >>> *.* @192.168.1.7 >>> *.* @192.168.1.122 >>> :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 >>> & @192.168.1.1;fixsnareForwardFormat2 >>> & ~ >>> :syslogtag, startswith, "MSWinEventLog#011" >>> /var/log/messages;fixsnareFormat >>> & @192.168.1.1;fixsnareForwardFormat >>> & ~ >>> *.* /var/log/messages;TraditionalFormat >>> *.* @192.168.1.1 >>> >>> it would be nice to move the outbound relays off to a different queue, >>> and >>> to put the list of rules that have order dependancies (because they >>> output >>> in different formats to fix up formatting problem) to a seperate queue >>> to >>> spread the work across multiple processes and not slow down the basic >>> writing to file. >>> >>> but as far as I can tell I would have to create a queue for each >>> individual item, and I can't create a queue for the part of the config >>> where I need to discard. >>> >>> am I missing something (including a better way to do everything ;-) >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 3 21:25:04 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 3 Nov 2009 21:25:04 +0100 Subject: [rsyslog] queue configuration Message-ID: <000301ca5cc3$d0e60e48$100013ac@intern.adiscon.com> Think about this: actions are independent of each other. If different actions write to the same file, they do that independently. If these actions now buffer before they write, they do that independently. Consequently, the writes get mixed up. Without buffering, each os write is done seperately, so the OS keeps each of the records appended to the file. ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 03.11.09 20:30 Betreff: Re: [rsyslog] queue configuration On Tue, 3 Nov 2009, Rainer Gerhards wrote: > Yes, thats right if you use the new buffered mode. The cure is either not to do that or use the new omruleset. so the new buffered mode even without seperate action queues or omruleset can have this problem? David Lang > ----- Urspr?ngliche Nachricht ----- > Von: "david at lang.hm" > An: "rsyslog-users" > Gesendet: 03.11.09 19:52 > Betreff: Re: [rsyslog] queue configuration > > On Tue, 3 Nov 2009, Rainer Gerhards wrote: > >> David and all, >> >> I have created a new output plugin, omruleset, which permits to nest >> rulesets. What it does is copy a message over from one ruleset to another and >> make it process in parallel. Some more details in the doc: >> >> http://www.rsyslog.com/doc-omruleset.html > > this looks very interesting, however there is one thing mentioned here > that concerns me. > > it sounds like you are saying that if you have multiple rules that write > to the same file that you may get the content of various lines mixed > togeather. > > am I understanding this correctly? > > if so, is there any way to work around this? > > David Lang > >> It permits to do what you asked for below. It also permits to do many more >> things and create very advanced configurations. But one must be VERY careful >> to receive the intended results. >> >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Friday, October 23, 2009 10:23 PM >>> To: rsyslog-users >>> Subject: [rsyslog] queue configuration >>> >>> I know that you can create an action queue for a specific output. >>> >>> is there any way to create an action queue for multiple outputs? >>> >>> for example, in my configuration I have >>> >>> :fromhost, !isequal, "127.0.0.1" >>> /var/log/messages;TraditionalFormat >>> :fromhost, isequal, "127.0.0.1" >>> @192.168.1.1;TraditionalForwardFormat >>> *.* @192.168.1.115 >>> *.* @192.168.1.241 >>> *.* @192.168.1.242 >>> *.* @192.168.1.6 >>> *.* @192.168.1.7 >>> *.* @192.168.1.122 >>> :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 >>> & @192.168.1.1;fixsnareForwardFormat2 >>> & ~ >>> :syslogtag, startswith, "MSWinEventLog#011" >>> /var/log/messages;fixsnareFormat >>> & @192.168.1.1;fixsnareForwardFormat >>> & ~ >>> *.* /var/log/messages;TraditionalFormat >>> *.* @192.168.1.1 >>> >>> it would be nice to move the outbound relays off to a different queue, >>> and >>> to put the list of rules that have order dependancies (because they >>> output >>> in different formats to fix up formatting problem) to a seperate queue >>> to >>> spread the work across multiple processes and not slow down the basic >>> writing to file. >>> >>> but as far as I can tell I would have to create a queue for each >>> individual item, and I can't create a queue for the part of the config >>> where I need to discard. >>> >>> am I missing something (including a better way to do everything ;-) >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From epiphani at gmail.com Wed Nov 4 00:53:05 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 3 Nov 2009 18:53:05 -0500 Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> Message-ID: There is an action queue defined per rule set.. of which there are four. Here's the tricky part: input has been stopped. For days. The queues just seem to continue to flush very very slowly. I have a feeling that if I increased the traffic flow, the queues would flush faster. I'm not 100% sure about that though, and I can't test in this environment. Unfortunately, due to outside requirements, I'm going to have to pull rsyslog out of this deployment until we can sort out these problems... I'll work on replicating the issue in another environment, if I can. -Aaron On Tue, Nov 3, 2009 at 1:20 PM, wrote: > what is the configuration? does it have seperate action queues defined? > > if you have some way to stop the input, it would let the main queue drop > down from being full (and eliminate the input modules from needing to lock > it to try and deliver messages) > > I did see some times during my testing of 4.x stuff where the throughput > would drop drasticly when the queue was full and there ws more incoming > traffic hammering on it. it would drop from 500K logs/min to ~3k logs/min > with a trivial config, slowing down the input would let it get out of this > mode and speed up again after a little while. so much other stuff was > happening at the time that I don't think that I reported it. > > David Lang > > On Tue, 3 Nov 2009, Rainer Gerhards wrote: > >> Date: Tue, 3 Nov 2009 18:38:40 +0100 >> From: Rainer Gerhards >> Reply-To: rsyslog-users >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing bug (4.5.5) >> >> It really looks like a full queue. The 2-second wait is the default wait >> interval before discarding a message when a queue is full. So for some >> reason >> the action queue seems to think it is full, so that the main queue worker >> can >> no longer add any data to it. Out of the strace, I unfortunately do not >> see >> why this happens. >> >> If possible, it would be good to try the 4.5.6 release, as this may >> actually >> be caused by the memory bug that is solved in that release. If it doesn't >> help, we would need to think about a way to get more in-depth info when >> the >> problem happens. I have an idea, but need to check if it can be done. >> >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>> Sent: Tuesday, November 03, 2009 6:34 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>> >>> Here's one with times: >>> >>> 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >>> 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL >>> >>> 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> >>> 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, ? >>> 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, >>> 3889000}) = 0 <0.000047> >>> 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 >>> 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> >>> 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, >>> 4147000}) = 0 <0.000013> >>> 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} >>> >>> 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection >>> timed out) <2.002462> >>> 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 >>> WT-GARULF6"..., 250) = 250 <0.000305> >>> 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> >>> 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, >>> 7292000}) = 0 <0.000027> >>> 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL >>> >>> 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> >>> 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, ? >>> 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, >>> 414932000}) = 0 <0.000048> >>> 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 >>> 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> >>> 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, >>> 415191000}) = 0 <0.000013> >>> 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} >>> >>> 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection >>> timed out) <2.002220> >>> 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 >>> WT-GARULF6"..., 197) = 197 <0.000279> >>> 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> >>> 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, >>> 417992000}) = 0 <0.000027> >>> 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL >>> >>> 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> >>> 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, ? >>> 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, >>> 608226000}) = 0 <0.000047> >>> 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 >>> 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> >>> 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, >>> 608486000}) = 0 <0.000014> >>> 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} >>> >>> >>> It looks like the locks are waiting a -very- long time. >>> >>> -Aaron >>> >>> On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards >>> wrote: >>>> >>>> first shot at the data: this looks like a full queue where the >>> >>> enqueue >>>> >>>> operation is waiting on a drain that does not happen (fast enough). >>> >>> Of >>>> >>>> course, that doesn't explain why this happens... >>>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>>> Sent: Tuesday, November 03, 2009 6:27 PM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>>> >>>>> Ok - I captured an strace. ?This appears to be the action thread. >>>>> This specific set of calls took minutes. ?I'll re-run this with -t >>>>> and/or -T. >>>>> >>>>> Note that this syslog instance is not actually receiving any data >>> >>> right >>>>> >>>>> now. >>>>> >>>>> -Aaron >>>>> >>>>> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} >> >>> ...> >>>>> >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>> timed out) >>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} >> >>> ...> >>>>> >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>> timed out) >>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} >> >>> ...> >>>>> >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>> timed out) >>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource >>>>> temporarily unavailable) >>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} >> >>> ...> >>>>> >>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>> timed out) >>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 >>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 >>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL >>>>> >>>>> >>>>> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards >>>>> wrote: >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>> Sent: Tuesday, November 03, 2009 4:06 PM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>>>>> >>>>>>> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >>>>>>> >>>>>>>> This is still taking place this hour - though I obviously can't >>>>>>>> restart onto a newer version without losing this case. ?Is >>> >>> there >>>>>>>> >>>>>>>> anything I can do in the current configuration to try and debug >>>>> >>>>> this >>>>>>>> >>>>>>>> situation? >>>>>>> >>>>>>> if you can strace each thread for a few seconds it may give you >>> >>> an >>>>> >>>>> idea >>>>>>> >>>>>>> what's happening. >>>>>> >>>>>> This is a good suggestion. >>>>>> >>>>>> It would also potentially be enlightening to attach gdb to the >>>>> >>>>> process at >>>>>> >>>>>> various points in time and get a backtrace from all running >>> >>> threads. >>>>>> >>>>>> Other than that, there is little you can do on a running version. >>> >>> If >>>>> >>>>> it were >>>>>> >>>>>> compiled for debug, and the environment setup so that we could >>>>> >>>>> capture >>>>>> >>>>>> stdout/stderr, sending SIGUSR2 would yield to much the same >>>>> >>>>> information as >>>>>> >>>>>> the gdb backtrace BUT when runtime instrumentation is enabled, the >>>>> >>>>> build is >>>>>> >>>>>> operating at a third to a quarter of its normal speed. >>>>>> >>>>>>> in the meantime you need to stop the system from getting further >>>>>>> behind, >>>>>>> can you redirect or reconfigure the senders so that they are not >>>>>>> sending >>>>>>> new logs to this box so that it can dig itself out (stopping the >>>>> >>>>> input >>>>>>> >>>>>>> may >>>>>>> be enough by itself, rsyslog has historicly done a LOT of locking >>>>>>> around >>>>>>> the main queue, and if you have a full queue with more data >>> >>> trying >>>>> >>>>> to >>>>>>> >>>>>>> be >>>>>>> delivered it can really slow things down. I wouldn't expect it to >>> >>> be >>>>>>> >>>>>>> this >>>>>>> much, but it could be part of it) >>>>>>> >>>>>> >>>>>> This may clean up things, but I really doubt it, given the >>> >>> magnitude >>>>> >>>>> of >>>>>> >>>>>> delays. Also, Aaron runs 4.4.5, which already has lots of the >>> >>> locking >>>>>> >>>>>> removed/restructure (not to compare with the recent v5-devel, but >>>>> >>>>> much more >>>>>> >>>>>> effcient than early v4 and v3). >>>>>> >>>>>> Rainer >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>>> (We're up to 18:46:51 now!) >>>>>>>> >>>>>>>> -Aaron >>>>>>>> >>>>>>>> On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> mhhh... This is obviously not intended behavior. There are >>> >>> some >>>>>>> >>>>>>> rate-limiting >>>>>>>>> >>>>>>>>> features that can deliberately slow down a queue, but I guess >>> >>> you >>>>>>> >>>>>>> have not >>>>>>>>> >>>>>>>>> configured these. So there either is a bug that activates them >>> >>> at >>>>>>> >>>>>>> some point >>>>>>>>> >>>>>>>>> during processing (e.g. an invalid memory access could do >>> >>> that) >>>>> >>>>> or >>>>>>> >>>>>>> there is >>>>>>>>> >>>>>>>>> some other bug that causes the dequeue to happen very slowly. >>> >>> In >>>>> >>>>> any >>>>>>> >>>>>>> case, it >>>>>>>>> >>>>>>>>> is hard to guess. >>>>>>>>> >>>>>>>>> Given the volume of the messages, a debug log probably does >>> >>> not >>>>>>> >>>>>>> help. We >>>>>>>>> >>>>>>>>> could gain a little insight in at least the queue sizes that >>>>> >>>>> rsyslog >>>>>>> >>>>>>> sees via >>>>>>>>> >>>>>>>>> imdiag plus the (very rudamentary) v5 debug front-end (it >>> >>> doesn't >>>>>>> >>>>>>> require the >>>>>>>>> >>>>>>>>> engine to be v5!). I would need to spend at least a little >>> >>> work >>>>> >>>>> on >>>>>>> >>>>>>> the >>>>>>>>> >>>>>>>>> front-end to enable remote access, but that's not really a >>>>> >>>>> problem. >>>>>>>>> >>>>>>>>> One other thing is that I am still holding a v4-beta with >>>>> >>>>> additional >>>>>>> >>>>>>> fixes as >>>>>>>>> >>>>>>>>> I didn't want to put these in the middle of some other >>> >>> debugging. >>>>>>> >>>>>>> However, >>>>>>>>> >>>>>>>>> these fixes address potential memory problems, so the most >>>>>>> >>>>>>> appropriate course >>>>>>>>> >>>>>>>>> of action would be to give that version at least a try. It >>> >>> needs >>>>> >>>>> to >>>>>>> >>>>>>> be >>>>>>>>> >>>>>>>>> released in the next days in any case. >>>>>>>>> >>>>>>>>> I have uploaded that (pre-4.5.6) version so that you can give >>> >>> it >>>>> >>>>> a >>>>>>> >>>>>>> try if you >>>>>>>>> >>>>>>>>> like: >>>>>>>>> >>>>>>>>> http://www.rsyslog.com/download/rsyslog/pre/rsyslog- >>> >>> 4.5.6.tar.gz >>>>>>>>> >>>>>>>>> I think it would good if you could try it at least once, >>> >>> because >>>>> >>>>> I >>>>>>> >>>>>>> think any >>>>>>>>> >>>>>>>>> other troubleshooting will boil down to looking at the fixes >>> >>> this >>>>>>> >>>>>>> version >>>>>>>>> >>>>>>>>> contains. >>>>>>>>> >>>>>>>>> Rainer >>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>>>>>>>> Sent: Monday, November 02, 2009 11:52 PM >>>>>>>>>> To: rsyslog-users >>>>>>>>>> Subject: [rsyslog] Queuing bug (4.5.5) >>>>>>>>>> >>>>>>>>>> Greetings all, >>>>>>>>>> >>>>>>>>>> I appear to have run into an issue with 4.5.5 where queues >>> >>> are >>>>> >>>>> not >>>>>>>>>> >>>>>>>>>> being flushed in a timely manner. ?In this specific case, I >>> >>> have >>>>>>> >>>>>>> data >>>>>>>>>> >>>>>>>>>> from last wednesday that is being written to disk in small >>>>> >>>>> chunks >>>>>>>>>> >>>>>>>>>> today since last wednesday. ?Unfortunately I cannot be too >>>>> >>>>> specific >>>>>>> >>>>>>> in >>>>>>>>>> >>>>>>>>>> details, but here is what I can see: >>>>>>>>>> >>>>>>>>>> Using omfile+gzip, there appears to have been a decent burst >>> >>> in >>>>>>>>>> >>>>>>>>>> traffic sometime last wednesday. ?The rsyslog instance has >>> >>> grown >>>>> >>>>> to >>>>>>>>>> >>>>>>>>>> 1.8GB of memory and stayed there for a while. ?I have both >>> >>> main >>>>>>>>>> >>>>>>>>>> message and action queues defined using fixedarray, and I see >>> >>> no >>>>>>>>>> >>>>>>>>>> on-disk queues (appears to be entirely in memory). >>>>>>>>>> >>>>>>>>>> I've got templates writing out to filenames using an hourly >>>>>>> >>>>>>> timestamp >>>>>>>>>> >>>>>>>>>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some >>> >>> of >>>>>>> >>>>>>> those >>>>>>>>>> >>>>>>>>>> files, all of them less than 5k in size, there are between 5 >>> >>> and >>>>> >>>>> 15 >>>>>>>>>> >>>>>>>>>> lines of content, all of them from last wednesday, and within >>> >>> a >>>>> >>>>> few >>>>>>>>>> >>>>>>>>>> seconds of each other. ?It's almost like there was a >>> >>> significant >>>>>>> >>>>>>> queue >>>>>>>>>> >>>>>>>>>> built up, the hour rolled over, and only the first block of >>>>> >>>>> lines >>>>>>> >>>>>>> were >>>>>>>>>> >>>>>>>>>> pulled from the queue. ?Then the hour rolled over again, and >>>>>>> >>>>>>> another >>>>>>>>>> >>>>>>>>>> block of lines were pulled from the queue. ?Then the next >>> >>> hour, >>>>>>> >>>>>>> then >>>>>>>>>> >>>>>>>>>> another 5-15 lines. >>>>>>>>>> >>>>>>>>>> Is it possible that one of the queues still has a good chunk >>> >>> of >>>>>>> >>>>>>> data >>>>>>>>>> >>>>>>>>>> built up, and is flushing it out very slowly? ?It hasn't been >>>>>>>>>> consistantly at the top of the hour either, and not every >>> >>> hour. >>>>>>> >>>>>>> ?But >>>>>>>>>> >>>>>>>>>> the log lines themselves are sequentially written out, and >>>>> >>>>> usually >>>>>>> >>>>>>> the >>>>>>>>>> >>>>>>>>>> lines are within a few seconds of each other. >>>>>>>>>> >>>>>>>>>> For example: >>>>>>>>>> >>>>>>>>>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >>>>>>> >>>>>>> 18:46:34 >>>>>>>>>> >>>>>>>>>> and one 18:46:35 >>>>>>>>>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 >>> >>> 18:46:35, >>>>> >>>>> 14 >>>>>>>>>> >>>>>>>>>> lines Oct 21 18:46:36 >>>>>>>>>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 >>> >>> 18:46:36, >>>>> >>>>> 4 >>>>>>>>>> >>>>>>>>>> lines Oct 21 18:46:37 >>>>>>>>>> >>>>>>>>>> Thoughts? >>>>>>>>>> -Aaron >>>>>>>>>> _______________________________________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > From david at lang.hm Wed Nov 4 03:18:56 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 3 Nov 2009 18:18:56 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 3 Nov 2009, Aaron Wiebe wrote: > There is an action queue defined per rule set.. of which there are four. is it possible that something blocked one of the rule outputs? as I understand it, that would cause that action queue to fill up, and then the behavior of the main queue trying, then sleeping would make sense to me > Here's the tricky part: input has been stopped. For days. The > queues just seem to continue to flush very very slowly. I have a > feeling that if I increased the traffic flow, the queues would flush > faster. I'm not 100% sure about that though, and I can't test in this > environment. Unfortunately, due to outside requirements, I'm going to > have to pull rsyslog out of this deployment until we can sort out > these problems... did you configure this with HUPisRestart off? if so you may try sending it a HUP and see if that gets a few log messages out. if it does, hammer it with HUPs to drain the queue it is really a pain to run into problems only in production. David Lang > I'll work on replicating the issue in another environment, if I can. > > -Aaron > > On Tue, Nov 3, 2009 at 1:20 PM, wrote: >> what is the configuration? does it have seperate action queues defined? >> >> if you have some way to stop the input, it would let the main queue drop >> down from being full (and eliminate the input modules from needing to lock >> it to try and deliver messages) >> >> I did see some times during my testing of 4.x stuff where the throughput >> would drop drasticly when the queue was full and there ws more incoming >> traffic hammering on it. it would drop from 500K logs/min to ~3k logs/min >> with a trivial config, slowing down the input would let it get out of this >> mode and speed up again after a little while. so much other stuff was >> happening at the time that I don't think that I reported it. >> >> David Lang >> >> On Tue, 3 Nov 2009, Rainer Gerhards wrote: >> >>> Date: Tue, 3 Nov 2009 18:38:40 +0100 >>> From: Rainer Gerhards >>> Reply-To: rsyslog-users >>> To: rsyslog-users >>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>> >>> It really looks like a full queue. The 2-second wait is the default wait >>> interval before discarding a message when a queue is full. So for some >>> reason >>> the action queue seems to think it is full, so that the main queue worker >>> can >>> no longer add any data to it. Out of the strace, I unfortunately do not >>> see >>> why this happens. >>> >>> If possible, it would be good to try the 4.5.6 release, as this may >>> actually >>> be caused by the memory bug that is solved in that release. If it doesn't >>> help, we would need to think about a way to get more in-depth info when >>> the >>> problem happens. I have an idea, but need to check if it can be done. >>> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>> Sent: Tuesday, November 03, 2009 6:34 PM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>> >>>> Here's one with times: >>>> >>>> 26365 17:27:59.996584 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >>>> 26365 17:28:00.002436 futex(0x1bec924, FUTEX_WAIT, 121, NULL >>>> >>>> 26365 17:28:00.003810 <... futex resumed> ) = 0 <0.001356> >>>> 26365 17:28:00.003871 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 17:28:00.003934 <... clock_gettime resumed> {1257269280, >>>> 3889000}) = 0 <0.000047> >>>> 26365 17:28:00.004001 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>> 26365 17:28:00.004065 <... futex resumed> ) = 0 <0.000046> >>>> 26365 17:28:00.004129 clock_gettime(CLOCK_REALTIME, {1257269280, >>>> 4147000}) = 0 <0.000013> >>>> 26365 17:28:00.004185 futex(0x1bec924, FUTEX_WAIT, 123, {1, 999742000} >>>> >>>> 26365 17:28:02.006675 <... futex resumed> ) = -1 ETIMEDOUT (Connection >>>> timed out) <2.002462> >>>> 26365 17:28:02.006797 write(165, "Oct 29 10:01:42 R6002 >>>> WT-GARULF6"..., 250) = 250 <0.000305> >>>> 26365 17:28:02.007183 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000028> >>>> 26365 17:28:02.007264 clock_gettime(CLOCK_REALTIME, {1257269282, >>>> 7292000}) = 0 <0.000027> >>>> 26365 17:28:02.007349 futex(0x1bec924, FUTEX_WAIT, 125, NULL >>>> >>>> 26365 17:30:15.414854 <... futex resumed> ) = 0 <133.407486> >>>> 26365 17:30:15.414914 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 17:30:15.414978 <... clock_gettime resumed> {1257269415, >>>> 414932000}) = 0 <0.000048> >>>> 26365 17:30:15.415044 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>> 26365 17:30:15.415108 <... futex resumed> ) = 0 <0.000047> >>>> 26365 17:30:15.415173 clock_gettime(CLOCK_REALTIME, {1257269415, >>>> 415191000}) = 0 <0.000013> >>>> 26365 17:30:15.415229 futex(0x1bec924, FUTEX_WAIT, 127, {1, 999741000} >>>> >>>> 26365 17:30:17.417472 <... futex resumed> ) = -1 ETIMEDOUT (Connection >>>> timed out) <2.002220> >>>> 26365 17:30:17.417540 write(165, "Oct 29 10:01:42 R6002 >>>> WT-GARULF6"..., 197) = 197 <0.000279> >>>> 26365 17:30:17.417886 futex(0x1bec8c8, FUTEX_WAKE, 1) = 0 <0.000027> >>>> 26365 17:30:17.417965 clock_gettime(CLOCK_REALTIME, {1257269417, >>>> 417992000}) = 0 <0.000027> >>>> 26365 17:30:17.418047 futex(0x1bec924, FUTEX_WAIT, 129, NULL >>>> >>>> 26365 17:31:15.608148 <... futex resumed> ) = 0 <58.190082> >>>> 26365 17:31:15.608209 clock_gettime(CLOCK_REALTIME, ? >>>> 26365 17:31:15.608272 <... clock_gettime resumed> {1257269475, >>>> 608226000}) = 0 <0.000047> >>>> 26365 17:31:15.608339 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>> 26365 17:31:15.608403 <... futex resumed> ) = 0 <0.000048> >>>> 26365 17:31:15.608468 clock_gettime(CLOCK_REALTIME, {1257269475, >>>> 608486000}) = 0 <0.000014> >>>> 26365 17:31:15.608524 futex(0x1bec924, FUTEX_WAIT, 131, {1, 999740000} >>>> >>>> >>>> It looks like the locks are waiting a -very- long time. >>>> >>>> -Aaron >>>> >>>> On Tue, Nov 3, 2009 at 12:30 PM, Rainer Gerhards >>>> wrote: >>>>> >>>>> first shot at the data: this looks like a full queue where the >>>> >>>> enqueue >>>>> >>>>> operation is waiting on a drain that does not happen (fast enough). >>>> >>>> Of >>>>> >>>>> course, that doesn't explain why this happens... >>>>> >>>>> Rainer >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>>>> Sent: Tuesday, November 03, 2009 6:27 PM >>>>>> To: rsyslog-users >>>>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>>>> >>>>>> Ok - I captured an strace. ?This appears to be the action thread. >>>>>> This specific set of calls took minutes. ?I'll re-run this with -t >>>>>> and/or -T. >>>>>> >>>>>> Note that this syslog instance is not actually receiving any data >>>> >>>> right >>>>>> >>>>>> now. >>>>>> >>>>>> -Aaron >>>>>> >>>>>> 26365 --- SIGSTOP (Stopped (signal)) @ 0 (0) --- >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 97, NULL >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 165944000}) = 0 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268800, 166035000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 99, {1, 999909000} >>> >>>> ...> >>>>>> >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>>> timed out) >>>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 229) = 229 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268802, 173123000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 101, NULL >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439872000}) = 0 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268893, 439999000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 103, {1, 999873000} >>> >>>> ...> >>>>>> >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>>> timed out) >>>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 248) = 248 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268895, 443096000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 105, NULL >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>>> 26365 <... clock_gettime resumed> {1257268950, 612585000}) = 0 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268950, 612757000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 107, {1, 999828000} >>> >>>> ...> >>>>>> >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>>> timed out) >>>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 180) = 180 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268952, 619980000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 109, NULL >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAIT, 2, NULL >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 EAGAIN (Resource >>>>>> temporarily unavailable) >>>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>>> 26365 <... clock_gettime resumed> {1257268956, 628783000}) = 0 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1 >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, ? >>>>>> 26365 <... clock_gettime resumed> {1257268956, 628953000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 111, {1, 999830000} >>> >>>> ...> >>>>>> >>>>>> 26365 <... futex resumed> ) ? ? ? ? ? ? = -1 ETIMEDOUT (Connection >>>>>> timed out) >>>>>> 26365 write(165, "Oct 29 10:01:42 R6002 WT-GARULF6"..., 335) = 335 >>>>>> 26365 futex(0x1bec8c8, FUTEX_WAKE, 1) ? = 0 >>>>>> 26365 clock_gettime(CLOCK_REALTIME, {1257268958, 636092000}) = 0 >>>>>> 26365 futex(0x1bec924, FUTEX_WAIT, 113, NULL >>>>>> >>>>>> >>>>>> On Tue, Nov 3, 2009 at 10:25 AM, Rainer Gerhards >>>>>> wrote: >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>>> Sent: Tuesday, November 03, 2009 4:06 PM >>>>>>>> To: rsyslog-users >>>>>>>> Subject: Re: [rsyslog] Queuing bug (4.5.5) >>>>>>>> >>>>>>>> On Tue, 3 Nov 2009, Aaron Wiebe wrote: >>>>>>>> >>>>>>>>> This is still taking place this hour - though I obviously can't >>>>>>>>> restart onto a newer version without losing this case. ?Is >>>> >>>> there >>>>>>>>> >>>>>>>>> anything I can do in the current configuration to try and debug >>>>>> >>>>>> this >>>>>>>>> >>>>>>>>> situation? >>>>>>>> >>>>>>>> if you can strace each thread for a few seconds it may give you >>>> >>>> an >>>>>> >>>>>> idea >>>>>>>> >>>>>>>> what's happening. >>>>>>> >>>>>>> This is a good suggestion. >>>>>>> >>>>>>> It would also potentially be enlightening to attach gdb to the >>>>>> >>>>>> process at >>>>>>> >>>>>>> various points in time and get a backtrace from all running >>>> >>>> threads. >>>>>>> >>>>>>> Other than that, there is little you can do on a running version. >>>> >>>> If >>>>>> >>>>>> it were >>>>>>> >>>>>>> compiled for debug, and the environment setup so that we could >>>>>> >>>>>> capture >>>>>>> >>>>>>> stdout/stderr, sending SIGUSR2 would yield to much the same >>>>>> >>>>>> information as >>>>>>> >>>>>>> the gdb backtrace BUT when runtime instrumentation is enabled, the >>>>>> >>>>>> build is >>>>>>> >>>>>>> operating at a third to a quarter of its normal speed. >>>>>>> >>>>>>>> in the meantime you need to stop the system from getting further >>>>>>>> behind, >>>>>>>> can you redirect or reconfigure the senders so that they are not >>>>>>>> sending >>>>>>>> new logs to this box so that it can dig itself out (stopping the >>>>>> >>>>>> input >>>>>>>> >>>>>>>> may >>>>>>>> be enough by itself, rsyslog has historicly done a LOT of locking >>>>>>>> around >>>>>>>> the main queue, and if you have a full queue with more data >>>> >>>> trying >>>>>> >>>>>> to >>>>>>>> >>>>>>>> be >>>>>>>> delivered it can really slow things down. I wouldn't expect it to >>>> >>>> be >>>>>>>> >>>>>>>> this >>>>>>>> much, but it could be part of it) >>>>>>>> >>>>>>> >>>>>>> This may clean up things, but I really doubt it, given the >>>> >>>> magnitude >>>>>> >>>>>> of >>>>>>> >>>>>>> delays. Also, Aaron runs 4.4.5, which already has lots of the >>>> >>>> locking >>>>>>> >>>>>>> removed/restructure (not to compare with the recent v5-devel, but >>>>>> >>>>>> much more >>>>>>> >>>>>>> effcient than early v4 and v3). >>>>>>> >>>>>>> Rainer >>>>>>>> >>>>>>>> David Lang >>>>>>>> >>>>>>>>> (We're up to 18:46:51 now!) >>>>>>>>> >>>>>>>>> -Aaron >>>>>>>>> >>>>>>>>> On Tue, Nov 3, 2009 at 1:46 AM, Rainer Gerhards >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> mhhh... This is obviously not intended behavior. There are >>>> >>>> some >>>>>>>> >>>>>>>> rate-limiting >>>>>>>>>> >>>>>>>>>> features that can deliberately slow down a queue, but I guess >>>> >>>> you >>>>>>>> >>>>>>>> have not >>>>>>>>>> >>>>>>>>>> configured these. So there either is a bug that activates them >>>> >>>> at >>>>>>>> >>>>>>>> some point >>>>>>>>>> >>>>>>>>>> during processing (e.g. an invalid memory access could do >>>> >>>> that) >>>>>> >>>>>> or >>>>>>>> >>>>>>>> there is >>>>>>>>>> >>>>>>>>>> some other bug that causes the dequeue to happen very slowly. >>>> >>>> In >>>>>> >>>>>> any >>>>>>>> >>>>>>>> case, it >>>>>>>>>> >>>>>>>>>> is hard to guess. >>>>>>>>>> >>>>>>>>>> Given the volume of the messages, a debug log probably does >>>> >>>> not >>>>>>>> >>>>>>>> help. We >>>>>>>>>> >>>>>>>>>> could gain a little insight in at least the queue sizes that >>>>>> >>>>>> rsyslog >>>>>>>> >>>>>>>> sees via >>>>>>>>>> >>>>>>>>>> imdiag plus the (very rudamentary) v5 debug front-end (it >>>> >>>> doesn't >>>>>>>> >>>>>>>> require the >>>>>>>>>> >>>>>>>>>> engine to be v5!). I would need to spend at least a little >>>> >>>> work >>>>>> >>>>>> on >>>>>>>> >>>>>>>> the >>>>>>>>>> >>>>>>>>>> front-end to enable remote access, but that's not really a >>>>>> >>>>>> problem. >>>>>>>>>> >>>>>>>>>> One other thing is that I am still holding a v4-beta with >>>>>> >>>>>> additional >>>>>>>> >>>>>>>> fixes as >>>>>>>>>> >>>>>>>>>> I didn't want to put these in the middle of some other >>>> >>>> debugging. >>>>>>>> >>>>>>>> However, >>>>>>>>>> >>>>>>>>>> these fixes address potential memory problems, so the most >>>>>>>> >>>>>>>> appropriate course >>>>>>>>>> >>>>>>>>>> of action would be to give that version at least a try. It >>>> >>>> needs >>>>>> >>>>>> to >>>>>>>> >>>>>>>> be >>>>>>>>>> >>>>>>>>>> released in the next days in any case. >>>>>>>>>> >>>>>>>>>> I have uploaded that (pre-4.5.6) version so that you can give >>>> >>>> it >>>>>> >>>>>> a >>>>>>>> >>>>>>>> try if you >>>>>>>>>> >>>>>>>>>> like: >>>>>>>>>> >>>>>>>>>> http://www.rsyslog.com/download/rsyslog/pre/rsyslog- >>>> >>>> 4.5.6.tar.gz >>>>>>>>>> >>>>>>>>>> I think it would good if you could try it at least once, >>>> >>>> because >>>>>> >>>>>> I >>>>>>>> >>>>>>>> think any >>>>>>>>>> >>>>>>>>>> other troubleshooting will boil down to looking at the fixes >>>> >>>> this >>>>>>>> >>>>>>>> version >>>>>>>>>> >>>>>>>>>> contains. >>>>>>>>>> >>>>>>>>>> Rainer >>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >>>>>>>>>>> Sent: Monday, November 02, 2009 11:52 PM >>>>>>>>>>> To: rsyslog-users >>>>>>>>>>> Subject: [rsyslog] Queuing bug (4.5.5) >>>>>>>>>>> >>>>>>>>>>> Greetings all, >>>>>>>>>>> >>>>>>>>>>> I appear to have run into an issue with 4.5.5 where queues >>>> >>>> are >>>>>> >>>>>> not >>>>>>>>>>> >>>>>>>>>>> being flushed in a timely manner. ?In this specific case, I >>>> >>>> have >>>>>>>> >>>>>>>> data >>>>>>>>>>> >>>>>>>>>>> from last wednesday that is being written to disk in small >>>>>> >>>>>> chunks >>>>>>>>>>> >>>>>>>>>>> today since last wednesday. ?Unfortunately I cannot be too >>>>>> >>>>>> specific >>>>>>>> >>>>>>>> in >>>>>>>>>>> >>>>>>>>>>> details, but here is what I can see: >>>>>>>>>>> >>>>>>>>>>> Using omfile+gzip, there appears to have been a decent burst >>>> >>>> in >>>>>>>>>>> >>>>>>>>>>> traffic sometime last wednesday. ?The rsyslog instance has >>>> >>>> grown >>>>>> >>>>>> to >>>>>>>>>>> >>>>>>>>>>> 1.8GB of memory and stayed there for a while. ?I have both >>>> >>>> main >>>>>>>>>>> >>>>>>>>>>> message and action queues defined using fixedarray, and I see >>>> >>>> no >>>>>>>>>>> >>>>>>>>>>> on-disk queues (appears to be entirely in memory). >>>>>>>>>>> >>>>>>>>>>> I've got templates writing out to filenames using an hourly >>>>>>>> >>>>>>>> timestamp >>>>>>>>>>> >>>>>>>>>>> (filenames like: ?[token]-[host]-YYYYMMDD-HH.txt.gz) ?In some >>>> >>>> of >>>>>>>> >>>>>>>> those >>>>>>>>>>> >>>>>>>>>>> files, all of them less than 5k in size, there are between 5 >>>> >>>> and >>>>>> >>>>>> 15 >>>>>>>>>>> >>>>>>>>>>> lines of content, all of them from last wednesday, and within >>>> >>>> a >>>>>> >>>>>> few >>>>>>>>>>> >>>>>>>>>>> seconds of each other. ?It's almost like there was a >>>> >>>> significant >>>>>>>> >>>>>>>> queue >>>>>>>>>>> >>>>>>>>>>> built up, the hour rolled over, and only the first block of >>>>>> >>>>>> lines >>>>>>>> >>>>>>>> were >>>>>>>>>>> >>>>>>>>>>> pulled from the queue. ?Then the hour rolled over again, and >>>>>>>> >>>>>>>> another >>>>>>>>>>> >>>>>>>>>>> block of lines were pulled from the queue. ?Then the next >>>> >>>> hour, >>>>>>>> >>>>>>>> then >>>>>>>>>>> >>>>>>>>>>> another 5-15 lines. >>>>>>>>>>> >>>>>>>>>>> Is it possible that one of the queues still has a good chunk >>>> >>>> of >>>>>>>> >>>>>>>> data >>>>>>>>>>> >>>>>>>>>>> built up, and is flushing it out very slowly? ?It hasn't been >>>>>>>>>>> consistantly at the top of the hour either, and not every >>>> >>>> hour. >>>>>>>> >>>>>>>> ?But >>>>>>>>>>> >>>>>>>>>>> the log lines themselves are sequentially written out, and >>>>>> >>>>>> usually >>>>>>>> >>>>>>>> the >>>>>>>>>>> >>>>>>>>>>> lines are within a few seconds of each other. >>>>>>>>>>> >>>>>>>>>>> For example: >>>>>>>>>>> >>>>>>>>>>> syslog-myhost-20091102-18.txt.gz: ?3 lines, 2 with TS Oct 21 >>>>>>>> >>>>>>>> 18:46:34 >>>>>>>>>>> >>>>>>>>>>> and one 18:46:35 >>>>>>>>>>> syslog-myhost-20091102-19.txt.gz: ?17 lines, 3 Oct 21 >>>> >>>> 18:46:35, >>>>>> >>>>>> 14 >>>>>>>>>>> >>>>>>>>>>> lines Oct 21 18:46:36 >>>>>>>>>>> syslog-myhost-20091102-20.txt.gz: ?12 lines, 8 Oct 21 >>>> >>>> 18:46:36, >>>>>> >>>>>> 4 >>>>>>>>>>> >>>>>>>>>>> lines Oct 21 18:46:37 >>>>>>>>>>> >>>>>>>>>>> Thoughts? >>>>>>>>>>> -Aaron >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> rsyslog mailing list >>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>> http://www.rsyslog.com >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Nov 4 07:22:01 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 07:22:01 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5? References: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103323@GRFEXC.intern.adiscon.com> Hi, that actually could be a regression caused by the rewrite of omfile. I'll look at it as soon as I have time, but it probably is a good idea to file a bug tracker with bugzilla. I first need to finish what I am currently implementing (custom parser interface) and then I need to look into which bugs are open and have priority. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luis Fernando Mu?oz Mej?as > Sent: Tuesday, November 03, 2009 7:05 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Output to named pipes not working on v4.5.5? > > Hello, world > > I must be doing something really dumb. I need to send some output to a > named pipe that I have already created, but rsyslog is refusing to > open it. > > My configuration states: > > *.* |/var/log/external/all/unknown;RSYSLOG_FileFormat > > And, of course, the named pipe exists: > > # ls -Z /var/log/external/all/unknown > > prw-r----- root logreaders user_u:object_r:var_log_t > /var/log/external/all/unknown > > But when I start the syslog daemon it complains with this message: > > rsyslogd-2039: Could no open output file > '|/var/log/external/all/unknown' [try http://www.rsyslog.com/e/2039 ] > > It may be a SELinux problem, but the logs show nothing about any > denied requests. And the context for the named pipe is the exact same > I have for the log files, which lie in the same directory as well. > > Am I the only one experiencing such problems? > > Thanks. > -- > Luis Fernando Mu?oz Mej?as > Luis.Fernando.Munoz.Mejias at cern.ch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Nov 4 07:34:43 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 07:34:43 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> > it is really a pain to run into problems only in production. The problem with hunting such bugs is that we cannot get the diagnostic information we need. I have thought about a solution. Could you please comment if this one would be acceptable for your environment: I add the capability to send the debug log out via UDP syslog (not via the usual channels, but rather via a special send function inside the debug printf system). Then, when a problem occurs, we could enable the debug printf (which is present in production builds as well) and make it output the data to a remote server (or a special listener, even netcat, running on the local machine). I could turn on/off this via a signal (as we currently do for sending debug message to stdout). Alternatively, one could load imdiag into a suspect instance and use that to gain more in-depth information. However, I am not sure if that would be suitable for use in production, at least it is questionable from a security POV. Any feedback is appreciated. While I like to have better ability to dig into problems as they occur, I don't think it makes sense to invest time into coding something if it is unclear whether or not it could be utilized... So unless I get sufficient positive feedback, I'll stay away from implementing that. Rainer From correo at miguelangelnieto.net Wed Nov 4 10:16:42 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Wed, 4 Nov 2009 10:16:42 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory Message-ID: Hi, I have a problem with the attached client-configuration. When I stop the server, the client computer hangs-up some minutes later and didn't write logs on $WorkDirectory /var/log/queue. Where is the mistake? Thank you and sorry for my bad english :P -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From epiphani at gmail.com Wed Nov 4 14:09:29 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Wed, 4 Nov 2009 08:09:29 -0500 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> Message-ID: On Wed, Nov 4, 2009 at 1:34 AM, Rainer Gerhards wrote: > > I add the capability to send the debug log out via UDP syslog (not via the > usual channels, but rather via a special send function inside the debug > printf system). Then, when a problem occurs, we could enable the debug printf > (which is present in production builds as well) and make it output the data > to a remote server (or a special listener, even netcat, running on the local > machine). > > I could turn on/off this via a signal (as we currently do for sending debug > message to stdout). That could work - I could see that being an acceptable step for some of our environments here. > Alternatively, one could load imdiag into a suspect instance and use that to > gain more in-depth information. However, I am not sure if that would be > suitable for use in production, at least it is questionable from a security > POV. Is there more docs on imdiag and the enhancements you've made to it recently kicking around? I'll see if I can't make use if it in my efforts to reproduce the issue (which probably won't start until next week). -Aaron From rgerhards at hq.adiscon.com Wed Nov 4 14:13:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 14:13:56 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103333@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Wednesday, November 04, 2009 2:09 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) / Problem in production > > On Wed, Nov 4, 2009 at 1:34 AM, Rainer Gerhards > wrote: > > > > I add the capability to send the debug log out via UDP syslog (not > via the > > usual channels, but rather via a special send function inside the > debug > > printf system). Then, when a problem occurs, we could enable the > debug printf > > (which is present in production builds as well) and make it output > the data > > to a remote server (or a special listener, even netcat, running on > the local > > machine). > > > > I could turn on/off this via a signal (as we currently do for sending > debug > > message to stdout). > > That could work - I could see that being an acceptable step for some > of our environments here. OK, that sounds good. Will see what I can do (I think this could be generally useful). > > > Alternatively, one could load imdiag into a suspect instance and use > that to > > gain more in-depth information. However, I am not sure if that would > be > > suitable for use in production, at least it is questionable from a > security > > POV. > > Is there more docs on imdiag and the enhancements you've made to it > recently kicking around? I'll see if I can't make use if it in my > efforts to reproduce the issue (which probably won't start until next > week). No, actually not. Initially it was intended to be a thing mostly for Rainer and the testbench ;) I'll see that I do a write-up on production troubleshooting and include it. Any more feedback from anyone on this issue is appreciated. Rainer > > -Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From Luis.Fernando.Munoz.Mejias at cern.ch Wed Nov 4 14:14:34 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?utf-8?q?Mu=C3=B1oz_Mej=C3=ADas?=) Date: Wed, 4 Nov 2009 14:14:34 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5? In-Reply-To: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> References: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <200911041414.34449.Luis.Fernando.Munoz.Mejias@cern.ch> So, this has two parts: 1) A SELinux problem on RHEL5 and similar (but the logs show nothing!!). It seems the syslog context is not allowed to access FIFOs. I'll have to fix that. 2) Even when there are no permission problems, rsyslog 4.5.5 just doesn't write to a named pipe. From the following consecutive lines: *.* |/var/log/external/fifo *.* /var/log/external/file the file receives data, the FIFO doesn't. I'll paste all this to bugzilla. Cheers. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From david at lang.hm Wed Nov 4 14:28:57 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 05:28:57 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 4 Nov 2009, Rainer Gerhards wrote: >> it is really a pain to run into problems only in production. > > The problem with hunting such bugs is that we cannot get the diagnostic > information we need. I have thought about a solution. Could you please > comment if this one would be acceptable for your environment: > > I add the capability to send the debug log out via UDP syslog (not via the > usual channels, but rather via a special send function inside the debug > printf system). Then, when a problem occurs, we could enable the debug printf > (which is present in production builds as well) and make it output the data > to a remote server (or a special listener, even netcat, running on the local > machine). > > I could turn on/off this via a signal (as we currently do for sending debug > message to stdout). that would be a LOT easier than having to run in debug mode the entire time. would this have similar performance impact to running in debug mode? or does this need less serialization, and therefor less impact? > Alternatively, one could load imdiag into a suspect instance and use that to > gain more in-depth information. However, I am not sure if that would be > suitable for use in production, at least it is questionable from a security > POV. is loading this on the fly even possible? david Lang > Any feedback is appreciated. While I like to have better ability to dig into > problems as they occur, I don't think it makes sense to invest time into > coding something if it is unclear whether or not it could be utilized... So > unless I get sufficient positive feedback, I'll stay away from implementing > that. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Nov 4 14:32:22 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 05:32:22 -0800 (PST) Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > I have a problem with the attached client-configuration. When I stop > the server, the client computer hangs-up some minutes later and didn't > write logs on $WorkDirectory /var/log/queue. this list strips attachments, please re-send with the config in the body of the message. david Lang -------------- next part -------------- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Nov 4 14:33:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 14:33:53 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103334@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, November 04, 2009 2:29 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) / Problem in production > > On Wed, 4 Nov 2009, Rainer Gerhards wrote: > > >> it is really a pain to run into problems only in production. > > > > The problem with hunting such bugs is that we cannot get the > diagnostic > > information we need. I have thought about a solution. Could you > please > > comment if this one would be acceptable for your environment: > > > > I add the capability to send the debug log out via UDP syslog (not > via the > > usual channels, but rather via a special send function inside the > debug > > printf system). Then, when a problem occurs, we could enable the > debug printf > > (which is present in production builds as well) and make it output > the data > > to a remote server (or a special listener, even netcat, running on > the local > > machine). > > > > I could turn on/off this via a signal (as we currently do for sending > debug > > message to stdout). > > that would be a LOT easier than having to run in debug mode the entire > time. The problem is that you would not get the information BEFORE you turned in on. In theory, it should even be possible with the current system and a file, but I need to check this out (I never used it that way, but the engine probably is capable of doing it). > would this have similar performance impact to running in debug mode? or > does this need less serialization, and therefor less impact? Think of it as turning on debug mode on the fly. That exactly is what it does. So before the signal, there is no impact, but after it is the same impact as usual. > > Alternatively, one could load imdiag into a suspect instance and use > that to > > gain more in-depth information. However, I am not sure if that would > be > > suitable for use in production, at least it is questionable from a > security > > POV. > > is loading this on the fly even possible? NO, that's the point. If we intend to pursue that path, imdiag always needs to be loaded. That does not have any performance impact (but a security impact!). Once imdiag is loaded, it could be used to do all sorts of "interesting" things (most of which need to be implemented first...). Rainer > > david Lang > > > Any feedback is appreciated. While I like to have better ability to > dig into > > problems as they occur, I don't think it makes sense to invest time > into > > coding something if it is unclear whether or not it could be > utilized... So > > unless I get sufficient positive feedback, I'll stay away from > implementing > > that. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Nov 4 14:55:20 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 05:55:20 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103334@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103334@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 4 Nov 2009, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >>> >>> I could turn on/off this via a signal (as we currently do for sending >> debug >>> message to stdout). >> >> that would be a LOT easier than having to run in debug mode the entire >> time. > > The problem is that you would not get the information BEFORE you turned in > on. In theory, it should even be possible with the current system and a file, > but I need to check this out (I never used it that way, but the engine > probably is capable of doing it). yes, this would not help see what triggered the problem, but could be a huge help in figuring out what the current state of things are. going to a file could work as well. in either case a config option would have to be created to specify the destination. >> would this have similar performance impact to running in debug mode? or >> does this need less serialization, and therefor less impact? > > Think of it as turning on debug mode on the fly. That exactly is what it > does. So before the signal, there is no impact, but after it is the same > impact as usual. even in a high volume production environment this can be tolorated for short time periods. currently there is no way (short of stop, reconfigure, start, stop, reconfigure, start) to try and gather this info, and each stop will loose logs, so this ability would be much better. >>> Alternatively, one could load imdiag into a suspect instance and use >> that to >>> gain more in-depth information. However, I am not sure if that would >> be >>> suitable for use in production, at least it is questionable from a >> security >>> POV. >> >> is loading this on the fly even possible? > > NO, that's the point. If we intend to pursue that path, imdiag always needs > to be loaded. That does not have any performance impact (but a security > impact!). Once imdiag is loaded, it could be used to do all sorts of > "interesting" things (most of which need to be implemented first...). it all depends on what imdiag can do, and where the control signals come from. if the contol is via normal syslog messages it is a problem, on the other hand if it creates a new socket that it listens on that can only be written to by root it's probably far less of a problem. I guess the real response is that I don't know enough about imdiag to know the risks. David Lang From correo at miguelangelnieto.net Wed Nov 4 15:00:35 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Wed, 4 Nov 2009 15:00:35 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # kernel logging (formerly provided by rklogd) $WorkDirectory /var/log/queue $MainMsgQueueFileName mainq $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "TVC" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "TVB" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "TTD" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "KCD" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "LPT" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "ABT" @@10.10.0.100 & ~ $ActionQueueType LinkedList $ActionQueueFileName dbq $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 :msg, contains, "XET" @@10.10.0.100 & ~ *.* /var/log/syslog kern.* /dev/console *.info;mail.none;authpriv.none;cron.none -/var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* -/var/log/cron uucp,news.crit -/var/log/spooler local7.* /var/log/boot.log 2009/11/4 : > On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > >> I have a problem with the attached client-configuration. When I stop >> the server, the client computer hangs-up some minutes later and didn't >> write logs on $WorkDirectory /var/log/queue. > > this list strips attachments, please re-send with the config in the body of > the message. > > david Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From rgerhards at hq.adiscon.com Wed Nov 4 15:11:58 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 15:11:58 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103315@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710331F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103320@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103324@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103334@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> > > The problem is that you would not get the information BEFORE you > turned in > > on. In theory, it should even be possible with the current system and > a file, > > but I need to check this out (I never used it that way, but the > engine > > probably is capable of doing it). > > yes, this would not help see what triggered the problem, but could be a > huge help in figuring out what the current state of things are. definitely > going to a file could work as well. in either case a config option > would > have to be created to specify the destination. traditionally, this is done via environment options, please see: http://www.rsyslog.com/doc-debug.html It may very well work just be specifying a file and sending SIGUSR1 - but I have never tried that in a production run (which also means I *never* tested it). When I added this mechanism, I selected envrionment variables because that enables to change the settings without touching the config file. Obviously it wouldn't be much of a problem to create config options for them as well. > > >> would this have similar performance impact to running in debug mode? > or > >> does this need less serialization, and therefor less impact? > > > > Think of it as turning on debug mode on the fly. That exactly is what > it > > does. So before the signal, there is no impact, but after it is the > same > > impact as usual. > > even in a high volume production environment this can be tolorated for > short time periods. currently there is no way (short of stop, > reconfigure, > start, stop, reconfigure, start) to try and gather this info, and each > stop will loose logs, so this ability would be much better. > > >>> Alternatively, one could load imdiag into a suspect instance and > use > >> that to > >>> gain more in-depth information. However, I am not sure if that > would > >> be > >>> suitable for use in production, at least it is questionable from a > >> security > >>> POV. > >> > >> is loading this on the fly even possible? > > > > NO, that's the point. If we intend to pursue that path, imdiag always > needs > > to be loaded. That does not have any performance impact (but a > security > > impact!). Once imdiag is loaded, it could be used to do all sorts of > > "interesting" things (most of which need to be implemented first...). > > it all depends on what imdiag can do, and where the control signals > come > from. if the contol is via normal syslog messages it is a problem, on > the > other hand if it creates a new socket that it listens on that can only > be > written to by root it's probably far less of a problem. > > I guess the real response is that I don't know enough about imdiag to > know > the risks. Most of that needs to be written, so we can specify what is acceptable. So far it listens to a network socket. This was intended so that testcases can be build without the need to fiddle with local socket creation policies. One potential was would also to selectively enable/disable imdiag functionality inside the config file. The GUI front-end I (barely) started to write talks to imdiag, but this is not really anything decent so far. In the long term, I'd like to be able to pull things like queue status, modules loaded, action status and the like from a running instance as well as inject not only messages but change some settings on the fly. My initial need was to provide a consistent rsyslog/UDP monitoring solution (for performance testing) as well as problem analysis. Rainer > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Nov 4 15:13:37 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 15:13:37 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103336@GRFEXC.intern.adiscon.com> just a quick look: the file name is always "dbq" and that will quickly result in an abort when actions go to the queue. Make sure the file names are different. Could be the source of your problem, but not sure... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Miguel Angel Nieto > Sent: Wednesday, November 04, 2009 3:01 PM > To: rsyslog-users > Subject: Re: [rsyslog] computer hang-up and WorkDirectory > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > $WorkDirectory /var/log/queue > $MainMsgQueueFileName mainq > > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TVC" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TVB" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TTD" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "KCD" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "LPT" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "ABT" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "XET" @@10.10.0.100 > & ~ > > *.* /var/log/syslog > kern.* /dev/console > *.info;mail.none;authpriv.none;cron.none - > /var/log/messages > authpriv.* /var/log/secure > mail.* - > /var/log/maillog > cron.* -/var/log/cron > uucp,news.crit - > /var/log/spooler > local7.* > /var/log/boot.log > > > 2009/11/4 : > > On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > > > >> I have a problem with the attached client-configuration. When I stop > >> the server, the client computer hangs-up some minutes later and > didn't > >> write logs on $WorkDirectory /var/log/queue. > > > > this list strips attachments, please re-send with the config in the > body of > > the message. > > > > david Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > > > > -- > Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que > hablar. Si quer?an decirme algo, tendr?an que escribirlo en un > papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que > hablar el resto de mi vida. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Nov 4 15:27:03 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 06:27:03 -0800 (PST) Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: ok, looking at this I don't see that you have any commands that would use the work directory. now when you say the client computer locks up do you mean the following? you have a server writing logs you have a seperate client sending logs to the server you shut down the server later the client machine stops responding. is this config for the client or for the server? one possible explination for the freeze you are seeing is that if you have the client configured to send via TCP (the @@ option) and the server does not accept the message, the client will queue the message, when the client queue fills up it will not accept any more messages. many processes (including login) will block until syslog accepts the message causeing the machine to 'freeze' or 'lock up' does this match what you are seeing? if so, turning the server back on should un-freeze the client machines. if this is the case you need to decide your priorities how critical is it to get the logs off the machine? in some cases they are a real security issue and you must get them off (in which case you really should be using relp, not tcp, but that's a different discussion that rainer did a write-up on), and your only real answer is to setup multiple servers so that one is always up. in other cases you are willing to spill over to disk and risk having an intruder tamper with the logs before they get sent off to another machine and set the main queu type to disk assisted mode in other cases you are willing to loose logs rather than freezing the machine and can configure rsyslog to accept messages, even when it can't do anything with them to avoid this sort of lockup. Daivd Lang On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > $WorkDirectory /var/log/queue > $MainMsgQueueFileName mainq > > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TVC" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TVB" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "TTD" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "KCD" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "LPT" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "ABT" @@10.10.0.100 > & ~ > $ActionQueueType LinkedList > $ActionQueueFileName dbq > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionResumeRetryCount -1 > :msg, contains, "XET" @@10.10.0.100 > & ~ > > *.* /var/log/syslog > kern.* /dev/console > *.info;mail.none;authpriv.none;cron.none -/var/log/messages > authpriv.* /var/log/secure > mail.* -/var/log/maillog > cron.* -/var/log/cron > uucp,news.crit -/var/log/spooler > local7.* /var/log/boot.log > > > 2009/11/4 : >> On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: >> >>> I have a problem with the attached client-configuration. When I stop >>> the server, the client computer hangs-up some minutes later and didn't >>> write logs on $WorkDirectory /var/log/queue. >> >> this list strips attachments, please re-send with the config in the body of >> the message. >> >> david Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > > > > From david at lang.hm Wed Nov 4 15:38:54 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 06:38:54 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 4 Nov 2009, Rainer Gerhards wrote: >>> The problem is that you would not get the information BEFORE you >> turned in >>> on. In theory, it should even be possible with the current system and >> a file, >>> but I need to check this out (I never used it that way, but the >> engine >>> probably is capable of doing it). >> >> yes, this would not help see what triggered the problem, but could be a >> huge help in figuring out what the current state of things are. > > definitely > >> going to a file could work as well. in either case a config option >> would >> have to be created to specify the destination. > > traditionally, this is done via environment options, please see: > > http://www.rsyslog.com/doc-debug.html > > It may very well work just be specifying a file and sending SIGUSR1 - but I > have never tried that in a production run (which also means I *never* tested > it). > > When I added this mechanism, I selected envrionment variables because that > enables to change the settings without touching the config file. Obviously it > wouldn't be much of a problem to create config options for them as well. given that in a production environment, rsyslog is started from an init script of some sort, setting environment variables before it starts is just editing a different config file. it would probably be much better to have it in the rsyslog config than to hunt down the correct (distro specific) init file and edit it to set an environemnt variable there. >>> NO, that's the point. If we intend to pursue that path, imdiag always >> needs >>> to be loaded. That does not have any performance impact (but a >> security >>> impact!). Once imdiag is loaded, it could be used to do all sorts of >>> "interesting" things (most of which need to be implemented first...). >> >> it all depends on what imdiag can do, and where the control signals >> come >> from. if the contol is via normal syslog messages it is a problem, on >> the >> other hand if it creates a new socket that it listens on that can only >> be >> written to by root it's probably far less of a problem. >> >> I guess the real response is that I don't know enough about imdiag to >> know >> the risks. > > Most of that needs to be written, so we can specify what is acceptable. So > far it listens to a network socket. This was intended so that testcases can > be build without the need to fiddle with local socket creation policies. One > potential was would also to selectively enable/disable imdiag functionality > inside the config file. > > The GUI front-end I (barely) started to write talks to imdiag, but this is > not really anything decent so far. In the long term, I'd like to be able to > pull things like queue status, modules loaded, action status and the like > from a running instance as well as inject not only messages but change some > settings on the fly. My initial need was to provide a consistent rsyslog/UDP > monitoring solution (for performance testing) as well as problem analysis. in a production environement you may not be able to use a GUI, and a network socket (on anything other than localhost) is definantly a problem. if you have it create a network listener on localhost that can be connected to via telnet it's not nearly as big of a security problem. then it can be scripted or a user can cut-n-paste to it while still allowing a nice GUI (or text based equivalent) to be created as time permits. but for now just use a simple line-based interface and it would be usable. if the listening address and port are configurable you can have the address default to 127.0.0.1 (relativly safe), but be configurable to 0.0.0.0 so that you can hit it over the network with other machines as needed. David Lang From rgerhards at hq.adiscon.com Wed Nov 4 15:57:37 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 15:57:37 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com> > given that in a production environment, rsyslog is started from an init > script of some sort, setting environment variables before it starts is > just editing a different config file. it would probably be much better > to > have it in the rsyslog config than to hunt down the correct (distro > specific) init file and edit it to set an environemnt variable there. I'll see if there is anything that prevents this, if not, I'll simply add the relevant statements. > > The GUI front-end I (barely) started to write talks to imdiag, but > this is > > not really anything decent so far. In the long term, I'd like to be > able to > > pull things like queue status, modules loaded, action status and the > like > > from a running instance as well as inject not only messages but > change some > > settings on the fly. My initial need was to provide a consistent > rsyslog/UDP > > monitoring solution (for performance testing) as well as problem > analysis. > > in a production environement you may not be able to use a GUI, and a > network socket (on anything other than localhost) is definantly a > problem. > > if you have it create a network listener on localhost that can be > connected to via telnet it's not nearly as big of a security problem. > then > it can be scripted or a user can cut-n-paste to it while still allowing > a > nice GUI (or text based equivalent) to be created as time permits. but > for > now just use a simple line-based interface and it would be usable. > > if the listening address and port are configurable you can have the > address default to 127.0.0.1 (relativly safe), but be configurable to > 0.0.0.0 so that you can hit it over the network with other machines as > needed. That's along the lines after which the current imdiag is designed. However, I was thinking to exchange some data via XML streams as it can be very lengthy (think of a complete queue stats dump for all queues in a system). XML would probably not work very well with the frontend, something readable would probably not work very well with the GUI - maybe the request should just specify the output format ;) Rainer From david at lang.hm Wed Nov 4 16:01:34 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 07:01:34 -0800 (PST) Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 4 Nov 2009, Rainer Gerhards wrote: >>> The GUI front-end I (barely) started to write talks to imdiag, but >> this is >>> not really anything decent so far. In the long term, I'd like to be >> able to >>> pull things like queue status, modules loaded, action status and the >> like >>> from a running instance as well as inject not only messages but >> change some >>> settings on the fly. My initial need was to provide a consistent >> rsyslog/UDP >>> monitoring solution (for performance testing) as well as problem >> analysis. >> >> in a production environement you may not be able to use a GUI, and a >> network socket (on anything other than localhost) is definantly a >> problem. >> >> if you have it create a network listener on localhost that can be >> connected to via telnet it's not nearly as big of a security problem. >> then >> it can be scripted or a user can cut-n-paste to it while still allowing >> a >> nice GUI (or text based equivalent) to be created as time permits. but >> for >> now just use a simple line-based interface and it would be usable. >> >> if the listening address and port are configurable you can have the >> address default to 127.0.0.1 (relativly safe), but be configurable to >> 0.0.0.0 so that you can hit it over the network with other machines as >> needed. > > That's along the lines after which the current imdiag is designed. However, I > was thinking to exchange some data via XML streams as it can be very lengthy > (think of a complete queue stats dump for all queues in a system). XML would > probably not work very well with the frontend, something readable would > probably not work very well with the GUI - maybe the request should just > specify the output format ;) that can work, however XML output may not be too bad, people entering commands manually are probably not going to be asking for that much data. the biggest problem I see with XML is the need to send requests and responses via e-mail for debugging. if one end uses a HTML enabled e-mail client it may not work well to paste XML text into it. David Lang From rgerhards at hq.adiscon.com Wed Nov 4 16:06:36 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 16:06:36 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103338@GRFEXC.intern.adiscon.com> > that can work, however XML output may not be too bad, people entering > commands manually are probably not going to be asking for that much > data. If I look at time constraints, I will probably not able to define a big CLI with a real language. So I think you might send something like "queuestatus" and it will reply with all status information on all queues it has. > the biggest problem I see with XML is the need to send requests and > responses via e-mail for debugging. if one end uses a HTML enabled e- > mail > client it may not work well to paste XML text into it. That's a very good point and I really should begin to think about this right now. Some kind of "debug package" that's easily mailable would not be a bad thing ;) Rainer From correo at miguelangelnieto.net Wed Nov 4 16:14:03 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Wed, 4 Nov 2009 16:14:03 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: Hi, 2009/11/4 : > ok, looking at this I don't see that you have any commands that would use > the work directory. Ok, here I have the first bug. I supposed that rsyslog will use work directory to save queues of all TCP forwards. What I missed to get this functionality? > now when you say the client computer locks up do you mean the following? > > you have a server writing logs > you have a seperate client sending logs to the server > you shut down the server > later the client machine stops responding. The client stops responding. > is this config for the client or for the server? Client. > one possible explination for the freeze you are seeing is that if you have > the client configured to send via TCP (the @@ option) and the server does > not accept the message, the client will queue the message, when the client > queue fills up it will not accept any more messages. many processes > (including login) will block until syslog accepts the message causeing the > machine to 'freeze' or 'lock up' > > does this match what you are seeing? Yes. So, activating the disk queue should fix this problem. Is it true? I would set a limit of 5GB. > if this is the case you need to decide your priorities > > how critical is it to get the logs off the machine? We have a lot of computers logging to a centralized rsyslog server. In each computer runs a application logging diferent activities (webcams, doors, etc.) We need all the data in one computer to analize it. > in other cases you are willing to loose logs rather than freezing the > machine and can configure rsyslog to accept messages, even when it can't > do anything with them to avoid this sort of lockup. How Can I do that? Thank you! > Daivd Lang > > > On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > >> $ModLoad immark.so # provides --MARK-- message capability >> $ModLoad imuxsock.so # provides support for local system logging (e.g. >> via logger command) >> $ModLoad imklog.so # kernel logging (formerly provided by rklogd) >> >> $WorkDirectory /var/log/queue >> $MainMsgQueueFileName mainq >> >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TVC" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TVB" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TTD" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "KCD" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "LPT" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "ABT" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "XET" @@10.10.0.100 >> & ~ >> >> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? /var/log/syslog >> kern.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? /dev/console >> *.info;mail.none;authpriv.none;cron.none ? ? ? ? ? ? ? ?-/var/log/messages >> authpriv.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/secure >> mail.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/maillog >> cron.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/cron >> uucp,news.crit ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/spooler >> local7.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/boot.log >> >> >> 2009/11/4 ?: >>> On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: >>> >>>> I have a problem with the attached client-configuration. When I stop >>>> the server, the client computer hangs-up some minutes later and didn't >>>> write logs on $WorkDirectory /var/log/queue. >>> >>> this list strips attachments, please re-send with the config in the body of >>> the message. >>> >>> david Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> >> >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From mbiebl at gmail.com Wed Nov 4 19:30:20 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 4 Nov 2009 19:30:20 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 Message-ID: Hi Rainer, hi list I tried the latest v5 stable release. First thing I noticed is, that it still uses -c 4 as native mode. I also noticed problems with logging to a pipe. The default rsyslog.conf file on Debian has a daemon.*;mail.*;\ news.err;\ *.=debug;*.=info;\ *.=notice;*.=warn |/dev/xconsole But I get the following error: 9042.679173747:b7607b70: file to log to: |/dev/xconsole 9042.679183525:b7607b70: doWrite, pData->pStrm 0x814d8c0, lenBuf 131 9042.679194699:b7607b70: strm 0x814d8c0: file -1 flush, buflen 208 9042.679225429:b7607b70: strm 0x814d8c0: open error 2, file '|/dev/xconsole' I've put a complete debug log + config files at http://debs.michaelbiebl.de/rsyslog/v5 Rainer, maybe you can take a look. Thanks, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Wed Nov 4 21:17:09 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Nov 2009 21:17:09 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Michael, That is very probably the same regression that Luis Fernando mentioned for 4.5.5. All in all, I would be quite careful with 5.2.0, I really have the impression that folks simply didn't try it out. But as I wrote, I need to somehow get the process started to come to make v5 mainstream. It is really important to me to get rid of v4-devel, as it takes considerable time (and bug potential!) to have two branches that are in development state. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Wednesday, November 04, 2009 7:30 PM > To: rsyslog-users > Subject: [rsyslog] rsyslog v5 5.2.0 > > Hi Rainer, hi list > > I tried the latest v5 stable release. > First thing I noticed is, that it still uses -c 4 as native mode. > I also noticed problems with logging to a pipe. The default > rsyslog.conf file on Debian has a > daemon.*;mail.*;\ > news.err;\ > *.=debug;*.=info;\ > *.=notice;*.=warn |/dev/xconsole > > But I get the following error: > > 9042.679173747:b7607b70: file to log to: |/dev/xconsole > 9042.679183525:b7607b70: doWrite, pData->pStrm 0x814d8c0, lenBuf 131 > 9042.679194699:b7607b70: strm 0x814d8c0: file -1 flush, buflen 208 > 9042.679225429:b7607b70: strm 0x814d8c0: open error 2, file > '|/dev/xconsole' > > I've put a complete debug log + config files at > http://debs.michaelbiebl.de/rsyslog/v5 > > Rainer, maybe you can take a look. > > Thanks, > Michael > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From correo at miguelangelnieto.net Wed Nov 4 22:32:43 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Wed, 4 Nov 2009 22:32:43 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: Hi again, I simplified the rsyslog config file: $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # kernel logging (formerly provided by rklogd) $WorkDirectory /var/log/queue $ActionQueueType LinkedList $ActionQueueFileName prueba $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on *.* @@10.10.0.210 & ~ *.* /var/log/messages kern.* /dev/console *.info;mail.none;authpriv.none;cron.none -/var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* -/var/log/cron uucp,news.crit -/var/log/spooler local7.* /var/log/boot.log Now I have disk queue, but rsyslog only logs up to 3000 entries. 0719.224892620:imuxsock.c: --------imuxsock calling select, active file descriptors (max 3): 3 0719.233534436:imuxsock.c: Message from UNIX socket: 0000003 0719.233573550:imuxsock.c: logmsg: flags 4, from 'linux-92wq', msg Nov 4 22:38:39 punisher: pruebaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa 0719.233589475:imuxsock.c: Message has legacy syslog format. 0719.233643675:imuxsock.c: main queue: entry added, size now 3000 entries 0719.233659599:imuxsock.c: main queue: EnqueueMsg signaled condition (0) 0719.233674127:imuxsock.c: wtpAdviseMaxWorkers signals busy 0719.233688096:imuxsock.c: --------imuxsock calling select, active file descriptors (max 3): 3 0719.244104262:imuxsock.c: Message from UNIX socket: 0000003 0719.244145331:imuxsock.c: logmsg: flags 4, from 'linux-92wq', msg Nov 4 22:38:39 punisher: pruebaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa 0719.244161256:imuxsock.c: Message has legacy syslog format. 0719.244186680:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0720.818283853:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0721.819871573:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0722.828967380:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0723.832482270:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0724.839541947:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0725.848415367:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0726.855937980:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0727.858592935:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. 0728.862998771:imuxsock.c: main queue: enqueueMsg: LightDelay mark reached for light delayble message - blocking a bit. Rsyslog 3.19.7 (Suse) 2009/11/4 : > ok, looking at this I don't see that you have any commands that would use > the work directory. > > now when you say the client computer locks up do you mean the following? > > you have a server writing logs > you have a seperate client sending logs to the server > you shut down the server > later the client machine stops responding. > > is this config for the client or for the server? > > one possible explination for the freeze you are seeing is that if you have > the client configured to send via TCP (the @@ option) and the server does > not accept the message, the client will queue the message, when the client > queue fills up it will not accept any more messages. many processes > (including login) will block until syslog accepts the message causeing the > machine to 'freeze' or 'lock up' > > does this match what you are seeing? > > if so, turning the server back on should un-freeze the client machines. > > > if this is the case you need to decide your priorities > > how critical is it to get the logs off the machine? in some cases they are > a real security issue and you must get them off (in which case you really > should be using relp, not tcp, but that's a different discussion that > rainer did a write-up on), and your only real answer is to setup multiple > servers so that one is always up. > > in other cases you are willing to spill over to disk and risk having an > intruder tamper with the logs before they get sent off to another machine > and set the main queu type to disk assisted mode > > in other cases you are willing to loose logs rather than freezing the > machine and can configure rsyslog to accept messages, even when it can't > do anything with them to avoid this sort of lockup. > > Daivd Lang > > > On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: > >> $ModLoad immark.so # provides --MARK-- message capability >> $ModLoad imuxsock.so # provides support for local system logging (e.g. >> via logger command) >> $ModLoad imklog.so # kernel logging (formerly provided by rklogd) >> >> $WorkDirectory /var/log/queue >> $MainMsgQueueFileName mainq >> >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TVC" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TVB" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "TTD" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "KCD" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "LPT" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "ABT" @@10.10.0.100 >> & ~ >> $ActionQueueType LinkedList >> $ActionQueueFileName dbq >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionResumeRetryCount -1 >> :msg, contains, "XET" @@10.10.0.100 >> & ~ >> >> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? /var/log/syslog >> kern.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? /dev/console >> *.info;mail.none;authpriv.none;cron.none ? ? ? ? ? ? ? ?-/var/log/messages >> authpriv.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/secure >> mail.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/maillog >> cron.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/cron >> uucp,news.crit ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/spooler >> local7.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/boot.log >> >> >> 2009/11/4 ?: >>> On Wed, 4 Nov 2009, Miguel Angel Nieto wrote: >>> >>>> I have a problem with the attached client-configuration. When I stop >>>> the server, the client computer hangs-up some minutes later and didn't >>>> write logs on $WorkDirectory /var/log/queue. >>> >>> this list strips attachments, please re-send with the config in the body of >>> the message. >>> >>> david Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> >> >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From tbergfeld at hq.adiscon.com Thu Nov 5 08:08:00 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Thu, 5 Nov 2009 08:08:00 +0100 Subject: [rsyslog] rsyslog 5.3.4 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103340@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.3.4, a member of the v5-devel branch. Today's release offers a number of important improvements. I would like to highlight the ability to nest rulesets [1] (I already announced that) and the ability to define custom parsers [2]. Let me elaborate a bit on the later. In the syslog world exists a myriad of incompatible and invalidly formatted messages. We have had numerous support requests on how to parse this or that malformed message format. With custom parsers, we now have a vehicel to integrate all these invalid formats in an elegant way that is also offering high performance ([2] has a concrete sample). The core idea now is that parsers are plugins just like input and output modules. It is relatively easy to write such a parser (roughly a day, I expect) and custom parsers can be integrated into rsyslogd in a way that they only affect messages received on specific port. So far, I have not actually created a custom parser, but I hope that over time many of them will become available. My hope here is that we actually can build a directory, which other folks can browse so that they are able to find a solution to the mess their vendor has provided ;) This release also introduces the capability to run rulesets off their own queue (also already mentioned on the mailing list). Plus, it contains the usual set of bug fixes. We plan to make this version the basis for the next v5-stable. [1] http://www.rsyslog.com/doc-omruleset.html [2] http://www.rsyslog.com/doc-rsconf1_rulesetparser.html ChangeLog: http://www.rsyslog.com/Article423.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-185.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From david at lang.hm Thu Nov 5 08:53:12 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 4 Nov 2009 23:53:12 -0800 (PST) Subject: [rsyslog] rsyslog 5.3.4 (devel) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103340@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103340@GRFEXC.intern.adiscon.com> Message-ID: yOn Thu, 5 Nov 2009, Tom Bergfeld wrote: > We have just released rsyslog 5.3.4, a member of the v5-devel branch. > Today's release offers a number of important improvements. I would like to > highlight the ability to nest rulesets [1] (I already announced that) and the > ability to define custom parsers [2]. > > Let me elaborate a bit on the later. In the syslog world exists a myriad of > incompatible and invalidly formatted messages. We have had numerous support > requests on how to parse this or that malformed message format. With custom > parsers, we now have a vehicel to integrate all these invalid formats in an > elegant way that is also offering high performance ([2] has a concrete > sample). The core idea now is that parsers are plugins just like input and > output modules. It is relatively easy to write such a parser (roughly a day, > I expect) and custom parsers can be integrated into rsyslogd in a way that > they only affect messages received on specific port. So far, I have not > actually created a custom parser, but I hope that over time many of them will > become available. My hope here is that we actually can build a directory, > which other folks can browse so that they are able to find a solution to the > mess their vendor has provided ;) this sounds very interesting. however, reading the link I'm a little confused. on the one hand it talks a lot about the priority of parsers, but then it talks about binding different parsers to different ports. It's not clear if these are two different ways to use parsers, or how these two would interact. where can these parsers be used? obviously the imudp module can use them. can all input modules use them? what are the limits of the parser? a couple of extreme examples: could you implement relp as a parser for imtcp, or since relp sends replies that can't be done (limiting it to different ways of processing a message that's already been received) could a parser detect that it had a piece of a multi-line messsage and stash the piece until it receives the rest of the message so that it could submit the complete message? a coouple questions from trying to look through the code at almost 3am local time if it can easily be done, may I suggest changing the santization flag from a yes/no boolean to an enum so that there can be more than one sanitation routine do you have the default parser broken out as a seperate file tha canbe used as an example? David Lang > This release also introduces the capability to run rulesets off their own > queue (also already mentioned on the mailing list). Plus, it contains the > usual set of bug fixes. > > We plan to make this version the basis for the next v5-stable. > > > [1] http://www.rsyslog.com/doc-omruleset.html > [2] http://www.rsyslog.com/doc-rsconf1_rulesetparser.html > > > > ChangeLog: > > http://www.rsyslog.com/Article423.phtml > > Download: > > http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-185.phtml > > As always, feedback is appreciated. > > Best regards, > Tom Bergfeld > -- > Support > ======= > > Improving rsyslog is costly, but you can help! We are looking for > organizations that find rsyslog useful and wish to contribute back. You can > contribute by reporting bugs, improve the software, or donate money or > equipment. > > Commercial support contracts for rsyslog are available, and they help finance > continued maintenance. Adiscon GmbH, a privately held German company, is > currently funding rsyslog development. We are always looking for interesting > development projects. For details on how to help, please see > http://www.rsyslog.com/doc-how2help.html . > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Nov 5 09:29:26 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 09:29:26 +0100 Subject: [rsyslog] rsyslog 5.3.4 (devel) released References: <9B6E2A8877C38245BFB15CC491A11DA7103340@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103342@GRFEXC.intern.adiscon.com> Good questions, thanks! :) I'll add some of the answers to the doc, what probably makes most sense. Just one thing: These are message parsers, not frame parsers. So you cannot parse RELP out of plain tcp, simply because that would be a frame parser (and even more, as these are totally different protocols). Message parsers work on the raw message, once it is extracted from the transport framing. I don't see any use case for permitting different frame parsers, as the framing is always bound to a specific protocol and a protocol has many more attributes than just the framing. The right way to implement a new protocol is via an input/output plugins. Currently, all message parsers are built into the core, but this is just a delivery mechanism (like with omfile, ...). The code is the same for loadable parsers. The two parsers I currently provide can be seen here: http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/pmrfc3164.c;h=5b684af56e 6d5e8ea2bcd616a06cc39e4cbb4a09;hb=6d9c54c7a2d4f07b0414082ef9681bd197ed6bde http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/pmrfc5424.c;h=07994adeba fb7e40d597e417dd1215874972971c;hb=6d9c54c7a2d4f07b0414082ef9681bd197ed6bde Rest comes either via doc or with a later reply... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, November 05, 2009 8:53 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 5.3.4 (devel) released > > yOn Thu, 5 Nov 2009, Tom Bergfeld wrote: > > > We have just released rsyslog 5.3.4, a member of the v5-devel branch. > > Today's release offers a number of important improvements. I would > like to > > highlight the ability to nest rulesets [1] (I already announced that) > and the > > ability to define custom parsers [2]. > > > > Let me elaborate a bit on the later. In the syslog world exists a > myriad of > > incompatible and invalidly formatted messages. We have had numerous > support > > requests on how to parse this or that malformed message format. With > custom > > parsers, we now have a vehicel to integrate all these invalid formats > in an > > elegant way that is also offering high performance ([2] has a > concrete > > sample). The core idea now is that parsers are plugins just like > input and > > output modules. It is relatively easy to write such a parser (roughly > a day, > > I expect) and custom parsers can be integrated into rsyslogd in a way > that > > they only affect messages received on specific port. So far, I have > not > > actually created a custom parser, but I hope that over time many of > them will > > become available. My hope here is that we actually can build a > directory, > > which other folks can browse so that they are able to find a solution > to the > > mess their vendor has provided ;) > > this sounds very interesting. however, reading the link I'm a little > confused. > > on the one hand it talks a lot about the priority of parsers, but then > it > talks about binding different parsers to different ports. It's not > clear > if these are two different ways to use parsers, or how these two would > interact. > > where can these parsers be used? > > obviously the imudp module can use them. can all input modules use > them? > > what are the limits of the parser? > > a couple of extreme examples: > > could you implement relp as a parser for imtcp, or since relp sends > replies that can't be done (limiting it to different ways of processing > a > message that's already been received) > > could a parser detect that it had a piece of a multi-line messsage and > stash the piece until it receives the rest of the message so that it > could > submit the complete message? > > > a coouple questions from trying to look through the code at almost 3am > local time > > if it can easily be done, may I suggest changing the santization flag > from > a yes/no boolean to an enum so that there can be more than one > sanitation > routine > > do you have the default parser broken out as a seperate file tha canbe > used as an example? > > David Lang > > > This release also introduces the capability to run rulesets off their > own > > queue (also already mentioned on the mailing list). Plus, it contains > the > > usual set of bug fixes. > > > > We plan to make this version the basis for the next v5-stable. > > > > > > [1] http://www.rsyslog.com/doc-omruleset.html > > [2] http://www.rsyslog.com/doc-rsconf1_rulesetparser.html > > > > > > > > ChangeLog: > > > > http://www.rsyslog.com/Article423.phtml > > > > Download: > > > > http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid- > 185.phtml > > > > As always, feedback is appreciated. > > > > Best regards, > > Tom Bergfeld > > -- > > Support > > ======= > > > > Improving rsyslog is costly, but you can help! We are looking for > > organizations that find rsyslog useful and wish to contribute back. > You can > > contribute by reporting bugs, improve the software, or donate money > or > > equipment. > > > > Commercial support contracts for rsyslog are available, and they help > finance > > continued maintenance. Adiscon GmbH, a privately held German > company, is > > currently funding rsyslog development. We are always looking for > interesting > > development projects. For details on how to help, please see > > http://www.rsyslog.com/doc-how2help.html . > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Nov 5 09:36:14 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 09:36:14 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103343@GRFEXC.intern.adiscon.com> > 0726.855937980:imuxsock.c: main queue: enqueueMsg: LightDelay mark > reached for light delayble message - blocking a bit. > 0727.858592935:imuxsock.c: main queue: enqueueMsg: LightDelay mark > reached for light delayble message - blocking a bit. > 0728.862998771:imuxsock.c: main queue: enqueueMsg: LightDelay mark > reached for light delayble message - blocking a bit. > > Rsyslog 3.19.7 (Suse) The version is the issue - that's an outdated beta version ;) Some versions had unix sockets flagged as sources that could lightly be delayed, resulting in what you see. That was made optional in later builds (and I guess nobody has turned it on since then ;)). Cure: upgrade to the current (3.22.1) v3-stable. Rainer From rgerhards at hq.adiscon.com Thu Nov 5 12:47:39 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 12:47:39 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5/5.2.0? References: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> <200911041414.34449.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103353@GRFEXC.intern.adiscon.com> Looks like a trivial regression, fix here: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=a5cd509be736fcff3c8ae310 4712d2fe85776416 I'll now do some more checks and see if I can include an automatted test for the pipe functionality so that a similar problem won't slip through in the future :) I will later apply the same fix to 5.2.0, it is the same problem. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luis Fernando Mu?oz Mej?as > Sent: Wednesday, November 04, 2009 2:15 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Output to named pipes not working on v4.5.5? > > So, this has two parts: > > 1) A SELinux problem on RHEL5 and similar (but the logs show > nothing!!). It seems the syslog context is not allowed to access > FIFOs. I'll have to fix that. > > 2) Even when there are no permission problems, rsyslog 4.5.5 just > doesn't write to a named pipe. From the following consecutive lines: > > *.* |/var/log/external/fifo > *.* /var/log/external/file > > the file receives data, the FIFO doesn't. > > I'll paste all this to bugzilla. > > Cheers. > -- > Luis Fernando Mu?oz Mej?as > Luis.Fernando.Munoz.Mejias at cern.ch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From martinmie at PartyGaming.com Thu Nov 5 13:51:38 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 5 Nov 2009 13:51:38 +0100 Subject: [rsyslog] rsyslog 5.2.0 meets signal 11. Message-ID: <0B1B9163138571439EAF804646F3F6AA08A586E8@GIBSVWIN004X.partygaming.local> Hi, Yesterday I successfully upgraded our logservers to the stable rsyslog 5.2.0. This morning I found out that the rsyslog process on the primary server was dead. Digging a bit in the logs I found this: -- # grep stop logserver.20091104.log 2009-11-04T07:32:01.970288-05:00 logserver kernel: Kernel logging (proc) stopped. -- According to the audit.log, it received a SIGSEGV (kill -11): -- type=ANOM_ABEND msg=audit(1257340663.009:19009): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=25881 comm="rsyslogd" sig=11 -- ...so, the application died on nov. 4th 2009 @ 8:17:43 EST -- # date -d @1257340663.009 Wed Nov 4 08:17:43 EST 2009 -- Has this been observed for this v5 stable branch?? And is there any fix? And, auditd was needed in order to determine the reason why rsyslogd was stopped. Could it be possible to have a more verbose message when the process stops due to an anomaly? Cheers, Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From Luis.Fernando.Munoz.Mejias at cern.ch Thu Nov 5 15:49:40 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?utf-8?q?Mu=C3=B1oz_Mej=C3=ADas?=) Date: Thu, 5 Nov 2009 15:49:40 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5/5.2.0? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103353@GRFEXC.intern.adiscon.com> References: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch> <200911041414.34449.Luis.Fernando.Munoz.Mejias@cern.ch> <9B6E2A8877C38245BFB15CC491A11DA7103353@GRFEXC.intern.adiscon.com> Message-ID: <200911051549.40352.Luis.Fernando.Munoz.Mejias@cern.ch> Rainer, > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=a5cd509be736fcff3c8ae3 >10 4712d2fe85776416 I've cherry-picked this commit and I confirm it's fixed on v4.5.X. Thanks a lot for the quick action. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From rgerhards at hq.adiscon.com Thu Nov 5 15:51:15 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 15:51:15 +0100 Subject: [rsyslog] Output to named pipes not working on v4.5.5/5.2.0? References: <200911031904.36361.Luis.Fernando.Munoz.Mejias@cern.ch><200911041414.34449.Luis.Fernando.Munoz.Mejias@cern.ch><9B6E2A8877C38245BFB15CC491A11DA7103353@GRFEXC.intern.adiscon.com> <200911051549.40352.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103357@GRFEXC.intern.adiscon.com> Thanks for the feedback, much appreciated. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luis Fernando Mu?oz Mej?as > Sent: Thursday, November 05, 2009 3:50 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Output to named pipes not working on > v4.5.5/5.2.0? > > Rainer, > > > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=a5cd509be736fcff3c > 8ae3 > >10 4712d2fe85776416 > > I've cherry-picked this commit and I confirm it's fixed on > v4.5.X. Thanks a lot for the quick action. > -- > Luis Fernando Mu?oz Mej?as > Luis.Fernando.Munoz.Mejias at cern.ch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Thu Nov 5 16:50:16 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 5 Nov 2009 16:50:16 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: 2009/11/4 Rainer Gerhards : > Michael, > > That is very probably the same regression that Luis Fernando mentioned for > 4.5.5. All in all, I would be quite careful with 5.2.0, I really have the > impression that folks simply didn't try it out. But as I wrote, I need to > somehow get the process started to come to make v5 mainstream. It is really > important to me to get rid of v4-devel, as it takes considerable time (and > bug potential!) to have two branches that are in development state. I can confirm that a5cd509be736fcff3c8ae3104712d2fe85776416 fixes this issue for me using 5.2.0. What about the -c compat flag? Is it intentional that there is no -c 5? I will probably upload a 5.2.* to unstable (or experimental) soon, which should give it some wider exposure. Cheers, Michael > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Michael Biebl >> Sent: Wednesday, November 04, 2009 7:30 PM >> To: rsyslog-users >> Subject: [rsyslog] rsyslog v5 5.2.0 >> >> Hi Rainer, hi list >> >> I tried the latest v5 stable release. >> First thing I noticed is, that it still uses -c 4 as native mode. >> I also noticed problems with logging to a pipe. The default >> rsyslog.conf file on Debian has a >> daemon.*;mail.*;\ >> ? ? ? ? news.err;\ >> ? ? ? ? *.=debug;*.=info;\ >> ? ? ? ? *.=notice;*.=warn ? ? ? |/dev/xconsole >> >> But I get the following error: >> >> 9042.679173747:b7607b70: file to log to: |/dev/xconsole >> 9042.679183525:b7607b70: doWrite, pData->pStrm 0x814d8c0, lenBuf 131 >> 9042.679194699:b7607b70: strm 0x814d8c0: file -1 flush, buflen 208 >> 9042.679225429:b7607b70: strm 0x814d8c0: open error 2, file >> '|/dev/xconsole' >> >> I've put a complete debug log + config files at >> http://debs.michaelbiebl.de/rsyslog/v5 >> >> Rainer, maybe you can take a look. >> >> Thanks, >> Michael >> >> -- >> Why is it that all of the instruments seeking intelligent life in the >> universe are pointed away from Earth? >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Nov 5 16:53:34 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 16:53:34 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, November 05, 2009 4:50 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog v5 5.2.0 > > 2009/11/4 Rainer Gerhards : > > Michael, > > > > That is very probably the same regression that Luis Fernando > mentioned for > > 4.5.5. All in all, I would be quite careful with 5.2.0, I really have > the > > impression that folks simply didn't try it out. But as I wrote, I > need to > > somehow get the process started to come to make v5 mainstream. It is > really > > important to me to get rid of v4-devel, as it takes considerable time > (and > > bug potential!) to have two branches that are in development state. > > I can confirm that a5cd509be736fcff3c8ae3104712d2fe85776416 fixes this > issue for me using 5.2.0. > What about the -c compat flag? Is it intentional that there is no -c 5? > I will probably upload a 5.2.* to unstable (or experimental) soon, > which should give it some wider exposure. I'll look into that, soon. My hopes are to get past 5.2.0 as quickly as possible. I have done a lot of work on the queue engine in the recent development version and there are definitely a couple of issues in 5.2.0 which cannot be fixed without upgrading the 5.3 version. That just FYI. I will probably not work on any queue-related issues in 5.2 but rather recommend to move on to a newer version. However, for normal operations that do not require intensive use of the queues, 5.2.0 should work OK and be faster than pervious versions. Rainer > Cheers, > Michael > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Michael > Biebl > >> Sent: Wednesday, November 04, 2009 7:30 PM > >> To: rsyslog-users > >> Subject: [rsyslog] rsyslog v5 5.2.0 > >> > >> Hi Rainer, hi list > >> > >> I tried the latest v5 stable release. > >> First thing I noticed is, that it still uses -c 4 as native mode. > >> I also noticed problems with logging to a pipe. The default > >> rsyslog.conf file on Debian has a > >> daemon.*;mail.*;\ > >> ? ? ? ? news.err;\ > >> ? ? ? ? *.=debug;*.=info;\ > >> ? ? ? ? *.=notice;*.=warn ? ? ? |/dev/xconsole > >> > >> But I get the following error: > >> > >> 9042.679173747:b7607b70: file to log to: |/dev/xconsole > >> 9042.679183525:b7607b70: doWrite, pData->pStrm 0x814d8c0, lenBuf 131 > >> 9042.679194699:b7607b70: strm 0x814d8c0: file -1 flush, buflen 208 > >> 9042.679225429:b7607b70: strm 0x814d8c0: open error 2, file > >> '|/dev/xconsole' > >> > >> I've put a complete debug log + config files at > >> http://debs.michaelbiebl.de/rsyslog/v5 > >> > >> Rainer, maybe you can take a look. > >> > >> Thanks, > >> Michael > >> > >> -- > >> Why is it that all of the instruments seeking intelligent life in > the > >> universe are pointed away from Earth? > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Thu Nov 5 17:02:32 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 5 Nov 2009 17:02:32 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710335A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710335A@GRFEXC.intern.adiscon.com> Message-ID: Another issue I noticed with 5.2.0 (+said patch) is, that it no longer shuts down cleany on SIGTERM/SIGINT If I run it with -c 4 -d and press STRG+C, I get ... 6895.847489380:b7692940: Terminating outputs... 6895.847516758:b7692940: destructing ruleset 0x89c7648, name 0x89c7678 6895.847553634:b7692940: strm 0x89c97b8: file -1 flush, buflen 0 6895.847595539:b7692940: strm 0x89ca1d8: file 5 flush, buflen 0 6895.847627386:b7692940: strm 0x89ca1d8: file 5 closing 6895.847688567:b7692940: strm 0x89cac48: file -1 flush, buflen 0 6895.847728796:b7692940: strm 0x89cb680: file 6 flush, buflen 0 6895.847758967:b7692940: strm 0x89cb680: file 6 closing 6895.847805062:b7692940: strm 0x89cc108: file -1 flush, buflen 0 6895.847846129:b7692940: strm 0x89ccb78: file -1 flush, buflen 0 6895.847886916:b7692940: strm 0x89cd5c0: file -1 flush, buflen 0 6895.847927145:b7692940: strm 0x89ce060: file -1 flush, buflen 0 6895.847967653:b7692940: strm 0x89cea98: file -1 flush, buflen 0 6895.848007882:b7692940: strm 0x89cf500: file -1 flush, buflen 0 6895.848048110:b7692940: strm 0x89cff68: file -1 flush, buflen 0 6895.848088897:b7692940: strm 0x89d09d8: file -1 flush, buflen 0 6895.848129126:b7692940: strm 0x89d1478: file -1 flush, buflen 0 6895.848169355:b7692940: strm 0x89d1ee8: file -1 flush, buflen 0 6895.848220758:b7692940: strm 0x89d2990: file 7 flush, buflen 0 6895.848251488:b7692940: strm 0x89d2990: file 7 closing 6895.848300098:b7692940: strm 0x89d38c0: file -1 flush, buflen 77 pressing STRG+C a second time will then terminate rsyslogd. From mrdemeanour at jackpot.uk.net Thu Nov 5 17:14:05 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Thu, 05 Nov 2009 16:14:05 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls Message-ID: <4AF2F9CD.9000402@jackpot.uk.net> Hi, I'm running a central rsyslog server with a couple of remote WAN (internet) clients and several remote LAN clients. Traffic is low - of the order of 10,000 messages per day. Internet clients communicate with the server using gnutls. LAN clients are currently using UDP. The server writes client logs to mysql, and also writes messages of local origin to disk. After some interval x::x <= 24h, the server consumes all memory (and swap) in the system. It looks as if the OS then tries to evict rsyslogd, and hangs - there's what looks like a stacktrace displayed on the (dead) console. I've stopped using gnutls for now, because the server machine is also a NFS file-server, and other systems depend on it not shutting down like this. It seems the problem doesn't occur with plain tcp. # uname -a Linux prajna 2.6.30-2-686 #1 SMP Sat Sep 26 01:16:22 UTC 2009 i686 GNU/Linux # rsyslogd -v rsyslogd 4.4.2, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: Yes FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: Yes FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: Yes Runtime Instrumentation (slow code): No Here's some log output from around the time of one of these incidents; I don't know if it helps, but the data on the console after the incident looks a lot like the end of this set of entries. Nov 4 08:10:05 prajna rsyslogd: [origin software="rsyslogd" swVersion="4.4.2" x-pid="15425" x-info="http://www.rsyslog.com"] (re)start Nov 4 08:10:05 prajna kernel: [47633.000709] Killed process 13665 (rsyslogd) Nov 4 08:10:05 prajna kernel: [47633.000612] 191505 pages non-shared Nov 4 08:10:05 prajna kernel: [47633.000636] Out of memory: kill process 13665 (rsyslogd) score 14137 or a child Nov 4 08:10:05 prajna kernel: [47633.000575] 2870 pages reserved Nov 4 08:10:05 prajna kernel: [47633.000594] 221 pages shared Nov 4 08:10:05 prajna kernel: [47633.000529] 196608 pages RAM Nov 4 08:10:05 prajna kernel: [47633.000557] 0 pages HighMem Nov 4 08:10:05 prajna kernel: [47632.970129] Free swap = 0kB Nov 4 08:10:05 prajna kernel: [47632.970148] Total swap = 1951856kB Nov 4 08:10:05 prajna kernel: [47632.970081] 0 pages in swap cache Nov 4 08:10:05 prajna kernel: [47632.970103] Swap cache stats: add 497948, delete 497948, find 68248/68574 Nov 4 08:10:05 prajna kernel: [47632.970061] 209 total pagecache pages Nov 4 08:10:05 prajna kernel: [47632.969982] Normal: 6*4kB 1*8kB 1*16kB 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3440kB Nov 4 08:10:05 prajna kernel: [47632.969873] lowmem_reserve[]: 0 0 0 0 Nov 4 08:10:05 prajna kernel: [47632.969905] DMA: 1*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3052kB Nov 4 08:10:05 prajna kernel: [47632.969763] lowmem_reserve[]: 0 746 746 746 Nov 4 08:10:05 prajna kernel: [47632.969804] Normal free:3456kB min:3460kB low:4324kB high:5188kB active_anon:366232kB inactive_anon:366324kB active_file:608kB inactive_file:156kB unevictable:64kB present:764032kB pages_scanned:1512018 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47632.968951] free:1627 slab:1995 mapped:1 pagetables:944 bounce:0 Nov 4 08:10:05 prajna kernel: [47632.969024] DMA free:3052kB min:68kB low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB active_file:0kB inactive_file:16kB unevictable:0kB present:15868kB pages_scanned:15745 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47632.968946] inactive_file:43 unevictable:16 dirty:0 writeback:0 unstable:0 Nov 4 08:10:05 prajna kernel: [47632.968909] CPU 0: hi: 186, btch: 31 usd: 171 Nov 4 08:10:05 prajna kernel: [47632.968941] Active_anon:92646 active_file:152 inactive_anon:92691 Nov 4 08:10:05 prajna kernel: [47632.968889] Normal per-cpu: Nov 4 08:10:05 prajna kernel: [47632.968827] Mem-Info: Nov 4 08:10:05 prajna kernel: [47632.968846] DMA per-cpu: Nov 4 08:10:05 prajna kernel: [47632.968866] CPU 0: hi: 0, btch: 1 usd: 0 Nov 4 08:10:05 prajna kernel: [47632.968803] [] ? do_page_fault+0x0/0x1e7 Nov 4 08:10:05 prajna kernel: [47632.968731] [] ? do_page_fault+0x0/0x1e7 Nov 4 08:10:05 prajna kernel: [47632.968774] [] ? error_code+0x6d/0x74 Nov 4 08:10:05 prajna kernel: [47632.968702] [] ? do_page_fault+0x1d8/0x1e7 Nov 4 08:10:05 prajna kernel: [47632.968623] [] ? handle_mm_fault+0x2bb/0x652 Nov 4 08:10:05 prajna kernel: [47632.968658] [] ? fd_install+0x1e/0x3c Nov 4 08:10:05 prajna kernel: [47632.968592] [] ? irq_exit+0x31/0x53 Nov 4 08:10:05 prajna kernel: [47632.968549] [] ? __do_fault+0x47/0x355 Nov 4 08:10:05 prajna kernel: [47632.968508] [] ? kunmap_atomic+0x63/0x72 Nov 4 08:10:05 prajna kernel: [47632.968442] [] ? do_page_cache_readahead+0x3d/0x47 Nov 4 08:10:05 prajna kernel: [47632.968474] [] ? filemap_fault+0x154/0x315 Nov 4 08:10:05 prajna kernel: [47632.968410] [] ? __do_page_cache_readahead+0x98/0x16b Nov 4 08:10:05 prajna kernel: [47632.968376] [] ? __alloc_pages_internal+0x2ee/0x39d Nov 4 08:10:05 prajna kernel: [47632.968342] [] ? out_of_memory+0x5a/0x7c Nov 4 08:10:05 prajna kernel: [47632.968311] [] ? __out_of_memory+0x33/0x10b Nov 4 08:10:05 prajna kernel: [47632.968231] Call Trace: Nov 4 08:10:05 prajna kernel: [47632.968279] [] ? oom_kill_process+0x79/0x1f7 Nov 4 08:10:05 prajna kernel: [47632.968173] hald-addon-stor cpuset=/ mems_allowed=0 Nov 4 08:10:05 prajna kernel: [47632.968204] Pid: 1965, comm: hald-addon-stor Not tainted 2.6.30-2-686 #1 Nov 4 08:10:05 prajna kernel: [47632.968114] hald-addon-stor invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 Nov 4 08:10:05 prajna kernel: [47631.204598] Out of memory: kill process 6420 (rsyslogd) score 21205 or a child Nov 4 08:10:05 prajna kernel: [47631.204667] Killed process 6420 (rsyslogd) Nov 4 08:10:05 prajna kernel: [47631.204555] 207 pages shared Nov 4 08:10:05 prajna kernel: [47631.204573] 191533 pages non-shared Nov 4 08:10:05 prajna kernel: [47631.204518] 0 pages HighMem Nov 4 08:10:05 prajna kernel: [47631.204536] 2870 pages reserved Nov 4 08:10:05 prajna kernel: [47631.174161] Total swap = 1951856kB Nov 4 08:10:05 prajna kernel: [47631.204489] 196608 pages RAM Nov 4 08:10:05 prajna kernel: [47631.174142] Free swap = 0kB Nov 4 08:10:05 prajna kernel: [47631.174093] 14 pages in swap cache Nov 4 08:10:05 prajna kernel: [47631.174116] Swap cache stats: add 497939, delete 497925, find 68248/68574 Nov 4 08:10:05 prajna kernel: [47631.173995] Normal: 4*4kB 1*8kB 1*16kB 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3432kB Nov 4 08:10:05 prajna kernel: [47631.174073] 230 total pagecache pages Nov 4 08:10:05 prajna kernel: [47631.173917] DMA: 1*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3052kB Nov 4 08:10:05 prajna kernel: [47631.173817] Normal free:3448kB min:3460kB low:4324kB high:5188kB active_anon:366252kB inactive_anon:366360kB active_file:280kB inactive_file:584kB unevictable:64kB present:764032kB pages_scanned:1830820 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47631.173885] lowmem_reserve[]: 0 0 0 0 Nov 4 08:10:05 prajna kernel: [47631.173775] lowmem_reserve[]: 0 746 746 746 Nov 4 08:10:05 prajna kernel: [47631.173710] DMA free:3052kB min:68kB low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB active_file:0kB inactive_file:0kB unevictable:0kB present:15868kB pages_scanned:15745 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47631.173637] free:1625 slab:2000 mapped:1 pagetables:944 bounce:0 Nov 4 08:10:05 prajna kernel: [47631.173627] Active_anon:92651 active_file:70 inactive_anon:92700 Nov 4 08:10:05 prajna kernel: [47631.173632] inactive_file:146 unevictable:16 dirty:0 writeback:0 unstable:0 Nov 4 08:10:05 prajna kernel: [47631.173575] Normal per-cpu: Nov 4 08:10:05 prajna kernel: [47631.173596] CPU 0: hi: 186, btch: 31 usd: 154 Nov 4 08:10:05 prajna kernel: [47631.173532] DMA per-cpu: Nov 4 08:10:05 prajna kernel: [47631.173552] CPU 0: hi: 0, btch: 1 usd: 0 Nov 4 08:10:05 prajna kernel: [47631.173514] Mem-Info: Nov 4 08:10:05 prajna kernel: [47631.173461] [] ? error_code+0x6d/0x74 Nov 4 08:10:05 prajna kernel: [47631.173490] [] ? do_page_fault+0x0/0x1e7 Nov 4 08:10:05 prajna kernel: [47631.173420] [] ? do_page_fault+0x0/0x1e7 Nov 4 08:10:05 prajna kernel: [47631.173348] [] ? sys_socketcall+0x103/0x19f Nov 4 08:10:05 prajna kernel: [47631.173390] [] ? do_page_fault+0x1d8/0x1e7 Nov 4 08:10:05 prajna kernel: [47631.173319] [] ? sys_recv+0x19/0x1d Nov 4 08:10:05 prajna kernel: [47631.173280] [] ? handle_mm_fault+0x2bb/0x652 Nov 4 08:10:05 prajna kernel: [47631.173208] [] ? kunmap_atomic+0x63/0x72 Nov 4 08:10:05 prajna kernel: [47631.173248] [] ? __do_fault+0x47/0x355 Nov 4 08:10:05 prajna kernel: [47631.173173] [] ? filemap_fault+0x154/0x315 Nov 4 08:10:05 prajna kernel: [47631.173110] [] ? __do_page_cache_readahead+0x98/0x16b Nov 4 08:10:05 prajna kernel: [47631.173142] [] ? do_page_cache_readahead+0x3d/0x47 Nov 4 08:10:05 prajna kernel: [47631.173011] [] ? __out_of_memory+0x33/0x10b Nov 4 08:10:05 prajna kernel: [47631.173042] [] ? out_of_memory+0x5a/0x7c Nov 4 08:10:05 prajna kernel: [47631.173076] [] ? __alloc_pages_internal+0x2ee/0x39d Nov 4 08:10:05 prajna kernel: [47631.172979] [] ? oom_kill_process+0x79/0x1f7 Nov 4 08:10:05 prajna kernel: [47631.172935] Call Trace: Nov 4 08:10:05 prajna kernel: [47631.172909] Pid: 13665, comm: rsyslogd Not tainted 2.6.30-2-686 #1 Nov 4 08:10:05 prajna kernel: [47631.172829] rsyslogd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 Nov 4 08:10:05 prajna kernel: [47631.172881] rsyslogd cpuset=/ mems_allowed=0 Nov 4 08:10:05 prajna kernel: [47631.136403] Killed process 13664 (rsyslogd) Nov 4 08:10:05 prajna kernel: [47631.136310] 191499 pages non-shared Nov 4 08:10:05 prajna kernel: [47631.136334] Out of memory: kill process 13664 (rsyslogd) score 42411 or a child Nov 4 08:10:05 prajna kernel: [47631.136272] 2870 pages reserved Nov 4 08:10:05 prajna kernel: [47631.136291] 203 pages shared Nov 4 08:10:05 prajna kernel: [47631.136226] 196608 pages RAM Nov 4 08:10:05 prajna kernel: [47631.136254] 0 pages HighMem Nov 4 08:10:05 prajna kernel: [47631.105740] Total swap = 1951856kB Nov 4 08:10:05 prajna kernel: [47631.105722] Free swap = 0kB Nov 4 08:10:05 prajna kernel: [47631.105696] Swap cache stats: add 497920, delete 497920, find 68247/68572 Nov 4 08:10:05 prajna kernel: [47631.105674] 0 pages in swap cache Nov 4 08:10:05 prajna kernel: [47631.105653] 152 total pagecache pages Nov 4 08:10:05 prajna kernel: [47631.105574] Normal: 33*4kB 2*8kB 1*16kB 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3556kB Nov 4 08:10:05 prajna kernel: [47631.105496] DMA: 1*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3052kB Nov 4 08:10:05 prajna kernel: [47631.105465] lowmem_reserve[]: 0 0 0 0 Nov 4 08:10:05 prajna kernel: [47631.105396] Normal free:3556kB min:3460kB low:4324kB high:5188kB active_anon:366288kB inactive_anon:366268kB active_file:280kB inactive_file:324kB unevictable:64kB present:764032kB pages_scanned:1830304 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47631.105355] lowmem_reserve[]: 0 746 746 746 Nov 4 08:10:05 prajna kernel: [47631.105289] DMA free:3052kB min:68kB low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB active_file:0kB inactive_file:0kB unevictable:0kB present:15868kB pages_scanned:15745 all_unreclaimable? yes Nov 4 08:10:05 prajna kernel: [47631.105217] free:1652 slab:2000 mapped:1 pagetables:944 bounce:0 Nov 4 08:10:05 prajna kernel: [47631.105207] Active_anon:92660 active_file:70 inactive_anon:92677 Nov 4 08:10:05 prajna kernel: [47631.105212] inactive_file:81 unevictable:16 dirty:0 writeback:0 unstable:0 Nov 4 08:10:05 prajna kernel: [47631.105155] Normal per-cpu: Nov 4 08:10:05 prajna kernel: [47631.105175] CPU 0: hi: 186, btch: 31 usd: 161 Nov 4 08:10:05 prajna kernel: [47631.105132] CPU 0: hi: 0, btch: 1 usd: 0 Nov 4 08:10:05 prajna kernel: [47631.105093] Mem-Info: Nov 4 08:10:05 prajna kernel: [47631.105112] DMA per-cpu: Nov 4 08:10:05 prajna kernel: [47631.105069] [] ? do_page_fault+0x0/0x1e7 Nov 4 08:10:05 prajna kernel: imklog 4.4.2, log source = /proc/kmsg started. Nov 4 08:10:05 prajna kernel: [] ? error_code+0x6d/0x74 From rgerhards at hq.adiscon.com Thu Nov 5 17:17:16 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:17:16 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710335A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335B@GRFEXC.intern.adiscon.com> Hi Michael, Is this a standard debian config? I ask because it works well for me... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, November 05, 2009 5:03 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog v5 5.2.0 > > Another issue I noticed with 5.2.0 (+said patch) is, that it no longer > shuts down cleany on SIGTERM/SIGINT > If I run it with -c 4 -d and press STRG+C, I get > ... > 6895.847489380:b7692940: Terminating outputs... > 6895.847516758:b7692940: destructing ruleset 0x89c7648, name 0x89c7678 > 6895.847553634:b7692940: strm 0x89c97b8: file -1 flush, buflen 0 > 6895.847595539:b7692940: strm 0x89ca1d8: file 5 flush, buflen 0 > 6895.847627386:b7692940: strm 0x89ca1d8: file 5 closing > 6895.847688567:b7692940: strm 0x89cac48: file -1 flush, buflen 0 > 6895.847728796:b7692940: strm 0x89cb680: file 6 flush, buflen 0 > 6895.847758967:b7692940: strm 0x89cb680: file 6 closing > 6895.847805062:b7692940: strm 0x89cc108: file -1 flush, buflen 0 > 6895.847846129:b7692940: strm 0x89ccb78: file -1 flush, buflen 0 > 6895.847886916:b7692940: strm 0x89cd5c0: file -1 flush, buflen 0 > 6895.847927145:b7692940: strm 0x89ce060: file -1 flush, buflen 0 > 6895.847967653:b7692940: strm 0x89cea98: file -1 flush, buflen 0 > 6895.848007882:b7692940: strm 0x89cf500: file -1 flush, buflen 0 > 6895.848048110:b7692940: strm 0x89cff68: file -1 flush, buflen 0 > 6895.848088897:b7692940: strm 0x89d09d8: file -1 flush, buflen 0 > 6895.848129126:b7692940: strm 0x89d1478: file -1 flush, buflen 0 > 6895.848169355:b7692940: strm 0x89d1ee8: file -1 flush, buflen 0 > 6895.848220758:b7692940: strm 0x89d2990: file 7 flush, buflen 0 > 6895.848251488:b7692940: strm 0x89d2990: file 7 closing > 6895.848300098:b7692940: strm 0x89d38c0: file -1 flush, buflen 77 > > > pressing STRG+C a second time will then terminate rsyslogd. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Nov 5 17:23:32 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:23:32 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335C@GRFEXC.intern.adiscon.com> Michael, > I can confirm that a5cd509be736fcff3c8ae3104712d2fe85776416 fixes this > issue for me using 5.2.0. > What about the -c compat flag? Is it intentional that there is no -c 5? > I will probably upload a 5.2.* to unstable (or experimental) soon, > which should give it some wider exposure. one more side-note: to do this, do you need a "stable" tag attached to the version by the upstream? I am asking because it would be much more useful to have wider exposure for 5.3.4, whereas 5.2.0 will probably not provide that much useful feedback (the most interesting pieces inside the queue engine already have been rewritten in 5.3). Still, it is good to see that the stable tag now draws attention to the version, I begin to think that nobody ever tried the beta ;) Maybe I should attach stable tags earlier in the future. Rainer From david at lang.hm Thu Nov 5 17:19:17 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 5 Nov 2009 08:19:17 -0800 (PST) Subject: [rsyslog] rsyslog v5 5.2.0 In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 5 Nov 2009, Michael Biebl wrote: > 2009/11/4 Rainer Gerhards : >> Michael, >> >> That is very probably the same regression that Luis Fernando mentioned for >> 4.5.5. All in all, I would be quite careful with 5.2.0, I really have the >> impression that folks simply didn't try it out. But as I wrote, I need to >> somehow get the process started to come to make v5 mainstream. It is really >> important to me to get rid of v4-devel, as it takes considerable time (and >> bug potential!) to have two branches that are in development state. > > I can confirm that a5cd509be736fcff3c8ae3104712d2fe85776416 fixes this > issue for me using 5.2.0. > What about the -c compat flag? Is it intentional that there is no -c 5? are you sure there isn't? I've been running dev versions of 5.x and hav eneeded to use -c5 with them instead of -c4 > I will probably upload a 5.2.* to unstable (or experimental) soon, > which should give it some wider exposure. I would not use 5.2.0, it had many problems under load (and problems where invalid syslog messages could cause it to crash) I'm currently running a slightly later git snapshot reliably under fairly heavy load (>10k logs/sec sustained for hours at a time). unfortunantly I am 3 timezones away from the office or I would upgrade to the 5.3 that was just released. I will proabaly do so tuesday when I am back in the office. David Lang > Cheers, > Michael > > >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com >>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Michael Biebl >>> Sent: Wednesday, November 04, 2009 7:30 PM >>> To: rsyslog-users >>> Subject: [rsyslog] rsyslog v5 5.2.0 >>> >>> Hi Rainer, hi list >>> >>> I tried the latest v5 stable release. >>> First thing I noticed is, that it still uses -c 4 as native mode. >>> I also noticed problems with logging to a pipe. The default >>> rsyslog.conf file on Debian has a >>> daemon.*;mail.*;\ >>> ? ? ? ? news.err;\ >>> ? ? ? ? *.=debug;*.=info;\ >>> ? ? ? ? *.=notice;*.=warn ? ? ? |/dev/xconsole >>> >>> But I get the following error: >>> >>> 9042.679173747:b7607b70: file to log to: |/dev/xconsole >>> 9042.679183525:b7607b70: doWrite, pData->pStrm 0x814d8c0, lenBuf 131 >>> 9042.679194699:b7607b70: strm 0x814d8c0: file -1 flush, buflen 208 >>> 9042.679225429:b7607b70: strm 0x814d8c0: open error 2, file >>> '|/dev/xconsole' >>> >>> I've put a complete debug log + config files at >>> http://debs.michaelbiebl.de/rsyslog/v5 >>> >>> Rainer, maybe you can take a look. >>> >>> Thanks, >>> Michael >>> >>> -- >>> Why is it that all of the instruments seeking intelligent life in the >>> universe are pointed away from Earth? >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > > > From rgerhards at hq.adiscon.com Thu Nov 5 17:29:52 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:29:52 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335D@GRFEXC.intern.adiscon.com> > > I will probably upload a 5.2.* to unstable (or experimental) soon, > > which should give it some wider exposure. > > I would not use 5.2.0, it had many problems under load (and problems > where > invalid syslog messages could cause it to crash) The invalid message problem should be fixed in 5.2.0, this was merged in to all v5 versions from the v4 fix. But I won't touch any queue problems, and I am sure there exist some during shutdown. > I'm currently running a slightly later git snapshot reliably under > fairly > heavy load (>10k logs/sec sustained for hours at a time). unfortunantly > I > am 3 timezones away from the office or I would upgrade to the 5.3 that > was > just released. I will proabaly do so tuesday when I am back in the > office. Looking forward to the results. I intend to promote that version to beta soon and try to push it through a very short beta cycle. I do not plan any more enhancements within the short-term timeframe. I looks like the advanced features begin to get used and so finally bug reports come in, so it is bug fixing and instrumentation time :) Rainer From correo at miguelangelnieto.net Thu Nov 5 17:31:33 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Thu, 5 Nov 2009 17:31:33 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103343@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103343@GRFEXC.intern.adiscon.com> Message-ID: Thank you! It works now! :) This is the final rsyslog.conf ############## $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # kernel logging (formerly provided by rklogd) $WorkDirectory /var/log/queue $ActionQueueType LinkedList $ActionQueueFileName SL $ActionQueueMaxDiskSpace 5g $ActionQueueMaxFileSize 10m $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 if $msg contains 'TL' or $msg contains 'CD' or $msg contains 'TR' then @@192.168.1.222 & ~ *.* /var/log/syslog kern.* /dev/console *.info;mail.none;authpriv.none;cron.none -/var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* -/var/log/cron uucp,news.crit -/var/log/spooler local7.* /var/log/boot.log ############# 2009/11/5 Rainer Gerhards : >> 0726.855937980:imuxsock.c: main queue: enqueueMsg: LightDelay mark >> reached for light delayble message - blocking a bit. >> 0727.858592935:imuxsock.c: main queue: enqueueMsg: LightDelay mark >> reached for light delayble message - blocking a bit. >> 0728.862998771:imuxsock.c: main queue: enqueueMsg: LightDelay mark >> reached for light delayble message - blocking a bit. >> >> Rsyslog 3.19.7 (Suse) > > The version is the issue - that's an outdated beta version ;) Some versions > had unix sockets flagged as sources that could lightly be delayed, resulting > in what you see. That was made optional in later builds (and I guess nobody > has turned it on since then ;)). > > Cure: upgrade to the current (3.22.1) v3-stable. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From rgerhards at hq.adiscon.com Thu Nov 5 17:37:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:37:44 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> Can you send me your rsyslog.conf, so that I can run it under the memory debugger in my lab. I'll also take this as a motivation to finally add multi-daemon tests to the testbench (what may take me a little while...). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Thursday, November 05, 2009 5:14 PM > To: rsyslog at lists.adiscon.com >> rsyslog-users > Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Hi, > > I'm running a central rsyslog server with a couple of remote WAN > (internet) clients and several remote LAN clients. Traffic is low - of > the order of 10,000 messages per day. Internet clients communicate with > the server using gnutls. LAN clients are currently using UDP. The > server > writes client logs to mysql, and also writes messages of local origin > to > disk. > > After some interval x::x <= 24h, the server consumes all memory (and > swap) in the system. It looks as if the OS then tries to evict > rsyslogd, > and hangs - there's what looks like a stacktrace displayed on the > (dead) > console. > > I've stopped using gnutls for now, because the server machine is also a > NFS file-server, and other systems depend on it not shutting down like > this. It seems the problem doesn't occur with plain tcp. > > # uname -a > Linux prajna 2.6.30-2-686 #1 SMP Sat Sep 26 01:16:22 UTC 2009 i686 > GNU/Linux > > # rsyslogd -v > rsyslogd 4.4.2, compiled with: > FEATURE_REGEXP: Yes > FEATURE_LARGEFILE: Yes > FEATURE_NETZIP (message compression): Yes > GSSAPI Kerberos 5 support: Yes > FEATURE_DEBUG (debug build, slow code): No > Atomic operations supported: Yes > Runtime Instrumentation (slow code): No > > Here's some log output from around the time of one of these incidents; > I > don't know if it helps, but the data on the console after the incident > looks a lot like the end of this set of entries. > > Nov 4 08:10:05 prajna rsyslogd: [origin software="rsyslogd" > swVersion="4.4.2" x-pid="15425" x-info="http://www.rsyslog.com"] > (re)start > Nov 4 08:10:05 prajna kernel: [47633.000709] Killed process 13665 > (rsyslogd) > Nov 4 08:10:05 prajna kernel: [47633.000612] 191505 pages non-shared > Nov 4 08:10:05 prajna kernel: [47633.000636] Out of memory: kill > process 13665 (rsyslogd) score 14137 or a child > Nov 4 08:10:05 prajna kernel: [47633.000575] 2870 pages reserved > Nov 4 08:10:05 prajna kernel: [47633.000594] 221 pages shared > Nov 4 08:10:05 prajna kernel: [47633.000529] 196608 pages RAM > Nov 4 08:10:05 prajna kernel: [47633.000557] 0 pages HighMem > Nov 4 08:10:05 prajna kernel: [47632.970129] Free swap = 0kB > Nov 4 08:10:05 prajna kernel: [47632.970148] Total swap = 1951856kB > Nov 4 08:10:05 prajna kernel: [47632.970081] 0 pages in swap cache > Nov 4 08:10:05 prajna kernel: [47632.970103] Swap cache stats: add > 497948, delete 497948, find 68248/68574 > Nov 4 08:10:05 prajna kernel: [47632.970061] 209 total pagecache pages > Nov 4 08:10:05 prajna kernel: [47632.969982] Normal: 6*4kB 1*8kB > 1*16kB > 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = > 3440kB > Nov 4 08:10:05 prajna kernel: [47632.969873] lowmem_reserve[]: 0 0 0 0 > Nov 4 08:10:05 prajna kernel: [47632.969905] DMA: 1*4kB 1*8kB 0*16kB > 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = > 3052kB > Nov 4 08:10:05 prajna kernel: [47632.969763] lowmem_reserve[]: 0 746 > 746 746 > Nov 4 08:10:05 prajna kernel: [47632.969804] Normal free:3456kB > min:3460kB low:4324kB high:5188kB active_anon:366232kB > inactive_anon:366324kB active_file:608kB inactive_file:156kB > unevictable:64kB present:764032kB pages_scanned:1512018 > all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47632.968951] free:1627 slab:1995 > mapped:1 pagetables:944 bounce:0 > Nov 4 08:10:05 prajna kernel: [47632.969024] DMA free:3052kB min:68kB > low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB > active_file:0kB inactive_file:16kB unevictable:0kB present:15868kB > pages_scanned:15745 all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47632.968946] inactive_file:43 > unevictable:16 dirty:0 writeback:0 unstable:0 > Nov 4 08:10:05 prajna kernel: [47632.968909] CPU 0: hi: 186, btch: > 31 usd: 171 > Nov 4 08:10:05 prajna kernel: [47632.968941] Active_anon:92646 > active_file:152 inactive_anon:92691 > Nov 4 08:10:05 prajna kernel: [47632.968889] Normal per-cpu: > Nov 4 08:10:05 prajna kernel: [47632.968827] Mem-Info: > Nov 4 08:10:05 prajna kernel: [47632.968846] DMA per-cpu: > Nov 4 08:10:05 prajna kernel: [47632.968866] CPU 0: hi: 0, btch: > 1 usd: 0 > Nov 4 08:10:05 prajna kernel: [47632.968803] [] ? > do_page_fault+0x0/0x1e7 > Nov 4 08:10:05 prajna kernel: [47632.968731] [] ? > do_page_fault+0x0/0x1e7 > Nov 4 08:10:05 prajna kernel: [47632.968774] [] ? > error_code+0x6d/0x74 > Nov 4 08:10:05 prajna kernel: [47632.968702] [] ? > do_page_fault+0x1d8/0x1e7 > Nov 4 08:10:05 prajna kernel: [47632.968623] [] ? > handle_mm_fault+0x2bb/0x652 > Nov 4 08:10:05 prajna kernel: [47632.968658] [] ? > fd_install+0x1e/0x3c > Nov 4 08:10:05 prajna kernel: [47632.968592] [] ? > irq_exit+0x31/0x53 > Nov 4 08:10:05 prajna kernel: [47632.968549] [] ? > __do_fault+0x47/0x355 > Nov 4 08:10:05 prajna kernel: [47632.968508] [] ? > kunmap_atomic+0x63/0x72 > Nov 4 08:10:05 prajna kernel: [47632.968442] [] ? > do_page_cache_readahead+0x3d/0x47 > Nov 4 08:10:05 prajna kernel: [47632.968474] [] ? > filemap_fault+0x154/0x315 > Nov 4 08:10:05 prajna kernel: [47632.968410] [] ? > __do_page_cache_readahead+0x98/0x16b > Nov 4 08:10:05 prajna kernel: [47632.968376] [] ? > __alloc_pages_internal+0x2ee/0x39d > Nov 4 08:10:05 prajna kernel: [47632.968342] [] ? > out_of_memory+0x5a/0x7c > Nov 4 08:10:05 prajna kernel: [47632.968311] [] ? > __out_of_memory+0x33/0x10b > Nov 4 08:10:05 prajna kernel: [47632.968231] Call Trace: > Nov 4 08:10:05 prajna kernel: [47632.968279] [] ? > oom_kill_process+0x79/0x1f7 > Nov 4 08:10:05 prajna kernel: [47632.968173] hald-addon-stor cpuset=/ > mems_allowed=0 > Nov 4 08:10:05 prajna kernel: [47632.968204] Pid: 1965, comm: > hald-addon-stor Not tainted 2.6.30-2-686 #1 > Nov 4 08:10:05 prajna kernel: [47632.968114] hald-addon-stor invoked > oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 > Nov 4 08:10:05 prajna kernel: [47631.204598] Out of memory: kill > process 6420 (rsyslogd) score 21205 or a child > Nov 4 08:10:05 prajna kernel: [47631.204667] Killed process 6420 > (rsyslogd) > Nov 4 08:10:05 prajna kernel: [47631.204555] 207 pages shared > Nov 4 08:10:05 prajna kernel: [47631.204573] 191533 pages non-shared > Nov 4 08:10:05 prajna kernel: [47631.204518] 0 pages HighMem > Nov 4 08:10:05 prajna kernel: [47631.204536] 2870 pages reserved > Nov 4 08:10:05 prajna kernel: [47631.174161] Total swap = 1951856kB > Nov 4 08:10:05 prajna kernel: [47631.204489] 196608 pages RAM > Nov 4 08:10:05 prajna kernel: [47631.174142] Free swap = 0kB > Nov 4 08:10:05 prajna kernel: [47631.174093] 14 pages in swap cache > Nov 4 08:10:05 prajna kernel: [47631.174116] Swap cache stats: add > 497939, delete 497925, find 68248/68574 > Nov 4 08:10:05 prajna kernel: [47631.173995] Normal: 4*4kB 1*8kB > 1*16kB > 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = > 3432kB > Nov 4 08:10:05 prajna kernel: [47631.174073] 230 total pagecache pages > Nov 4 08:10:05 prajna kernel: [47631.173917] DMA: 1*4kB 1*8kB 0*16kB > 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = > 3052kB > Nov 4 08:10:05 prajna kernel: [47631.173817] Normal free:3448kB > min:3460kB low:4324kB high:5188kB active_anon:366252kB > inactive_anon:366360kB active_file:280kB inactive_file:584kB > unevictable:64kB present:764032kB pages_scanned:1830820 > all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47631.173885] lowmem_reserve[]: 0 0 0 0 > Nov 4 08:10:05 prajna kernel: [47631.173775] lowmem_reserve[]: 0 746 > 746 746 > Nov 4 08:10:05 prajna kernel: [47631.173710] DMA free:3052kB min:68kB > low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB > active_file:0kB inactive_file:0kB unevictable:0kB present:15868kB > pages_scanned:15745 all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47631.173637] free:1625 slab:2000 > mapped:1 pagetables:944 bounce:0 > Nov 4 08:10:05 prajna kernel: [47631.173627] Active_anon:92651 > active_file:70 inactive_anon:92700 > Nov 4 08:10:05 prajna kernel: [47631.173632] inactive_file:146 > unevictable:16 dirty:0 writeback:0 unstable:0 > Nov 4 08:10:05 prajna kernel: [47631.173575] Normal per-cpu: > Nov 4 08:10:05 prajna kernel: [47631.173596] CPU 0: hi: 186, btch: > 31 usd: 154 > Nov 4 08:10:05 prajna kernel: [47631.173532] DMA per-cpu: > Nov 4 08:10:05 prajna kernel: [47631.173552] CPU 0: hi: 0, btch: > 1 usd: 0 > Nov 4 08:10:05 prajna kernel: [47631.173514] Mem-Info: > Nov 4 08:10:05 prajna kernel: [47631.173461] [] ? > error_code+0x6d/0x74 > Nov 4 08:10:05 prajna kernel: [47631.173490] [] ? > do_page_fault+0x0/0x1e7 > Nov 4 08:10:05 prajna kernel: [47631.173420] [] ? > do_page_fault+0x0/0x1e7 > Nov 4 08:10:05 prajna kernel: [47631.173348] [] ? > sys_socketcall+0x103/0x19f > Nov 4 08:10:05 prajna kernel: [47631.173390] [] ? > do_page_fault+0x1d8/0x1e7 > Nov 4 08:10:05 prajna kernel: [47631.173319] [] ? > sys_recv+0x19/0x1d > Nov 4 08:10:05 prajna kernel: [47631.173280] [] ? > handle_mm_fault+0x2bb/0x652 > Nov 4 08:10:05 prajna kernel: [47631.173208] [] ? > kunmap_atomic+0x63/0x72 > Nov 4 08:10:05 prajna kernel: [47631.173248] [] ? > __do_fault+0x47/0x355 > Nov 4 08:10:05 prajna kernel: [47631.173173] [] ? > filemap_fault+0x154/0x315 > Nov 4 08:10:05 prajna kernel: [47631.173110] [] ? > __do_page_cache_readahead+0x98/0x16b > Nov 4 08:10:05 prajna kernel: [47631.173142] [] ? > do_page_cache_readahead+0x3d/0x47 > Nov 4 08:10:05 prajna kernel: [47631.173011] [] ? > __out_of_memory+0x33/0x10b > Nov 4 08:10:05 prajna kernel: [47631.173042] [] ? > out_of_memory+0x5a/0x7c > Nov 4 08:10:05 prajna kernel: [47631.173076] [] ? > __alloc_pages_internal+0x2ee/0x39d > Nov 4 08:10:05 prajna kernel: [47631.172979] [] ? > oom_kill_process+0x79/0x1f7 > Nov 4 08:10:05 prajna kernel: [47631.172935] Call Trace: > Nov 4 08:10:05 prajna kernel: [47631.172909] Pid: 13665, comm: > rsyslogd > Not tainted 2.6.30-2-686 #1 > Nov 4 08:10:05 prajna kernel: [47631.172829] rsyslogd invoked > oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 > Nov 4 08:10:05 prajna kernel: [47631.172881] rsyslogd cpuset=/ > mems_allowed=0 > Nov 4 08:10:05 prajna kernel: [47631.136403] Killed process 13664 > (rsyslogd) > Nov 4 08:10:05 prajna kernel: [47631.136310] 191499 pages non-shared > Nov 4 08:10:05 prajna kernel: [47631.136334] Out of memory: kill > process 13664 (rsyslogd) score 42411 or a child > Nov 4 08:10:05 prajna kernel: [47631.136272] 2870 pages reserved > Nov 4 08:10:05 prajna kernel: [47631.136291] 203 pages shared > Nov 4 08:10:05 prajna kernel: [47631.136226] 196608 pages RAM > Nov 4 08:10:05 prajna kernel: [47631.136254] 0 pages HighMem > Nov 4 08:10:05 prajna kernel: [47631.105740] Total swap = 1951856kB > Nov 4 08:10:05 prajna kernel: [47631.105722] Free swap = 0kB > Nov 4 08:10:05 prajna kernel: [47631.105696] Swap cache stats: add > 497920, delete 497920, find 68247/68572 > Nov 4 08:10:05 prajna kernel: [47631.105674] 0 pages in swap cache > Nov 4 08:10:05 prajna kernel: [47631.105653] 152 total pagecache pages > Nov 4 08:10:05 prajna kernel: [47631.105574] Normal: 33*4kB 2*8kB > 1*16kB 0*32kB 1*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB > = 3556kB > Nov 4 08:10:05 prajna kernel: [47631.105496] DMA: 1*4kB 1*8kB 0*16kB > 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = > 3052kB > Nov 4 08:10:05 prajna kernel: [47631.105465] lowmem_reserve[]: 0 0 0 0 > Nov 4 08:10:05 prajna kernel: [47631.105396] Normal free:3556kB > min:3460kB low:4324kB high:5188kB active_anon:366288kB > inactive_anon:366268kB active_file:280kB inactive_file:324kB > unevictable:64kB present:764032kB pages_scanned:1830304 > all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47631.105355] lowmem_reserve[]: 0 746 > 746 746 > Nov 4 08:10:05 prajna kernel: [47631.105289] DMA free:3052kB min:68kB > low:84kB high:100kB active_anon:4352kB inactive_anon:4440kB > active_file:0kB inactive_file:0kB unevictable:0kB present:15868kB > pages_scanned:15745 all_unreclaimable? yes > Nov 4 08:10:05 prajna kernel: [47631.105217] free:1652 slab:2000 > mapped:1 pagetables:944 bounce:0 > Nov 4 08:10:05 prajna kernel: [47631.105207] Active_anon:92660 > active_file:70 inactive_anon:92677 > Nov 4 08:10:05 prajna kernel: [47631.105212] inactive_file:81 > unevictable:16 dirty:0 writeback:0 unstable:0 > Nov 4 08:10:05 prajna kernel: [47631.105155] Normal per-cpu: > Nov 4 08:10:05 prajna kernel: [47631.105175] CPU 0: hi: 186, btch: > 31 usd: 161 > Nov 4 08:10:05 prajna kernel: [47631.105132] CPU 0: hi: 0, btch: > 1 usd: 0 > Nov 4 08:10:05 prajna kernel: [47631.105093] Mem-Info: > Nov 4 08:10:05 prajna kernel: [47631.105112] DMA per-cpu: > Nov 4 08:10:05 prajna kernel: [47631.105069] [] ? > do_page_fault+0x0/0x1e7 > Nov 4 08:10:05 prajna kernel: imklog 4.4.2, log source = /proc/kmsg > started. > Nov 4 08:10:05 prajna kernel: [] ? error_code+0x6d/0x74 > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mrdemeanour at jackpot.uk.net Thu Nov 5 17:41:15 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Thu, 05 Nov 2009 16:41:15 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> References: <4AF2F9CD.9000402@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> Message-ID: <4AF3002B.9060302@jackpot.uk.net> Rainer Gerhards wrote: > Can you send me your rsyslog.conf, so that I can run it under the memory > debugger in my lab. I'll also take this as a motivation to finally add > multi-daemon tests to the testbench (what may take me a little while...). This is the server config (some of the remarks are misleading). # /etc/rsyslog.conf Configuration file for rsyslog v3. # # For more information see # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html # $DebugPrintTemplateList on # $ActionFileDefaultTemplate mysql-template ################# #### MODULES #### ################# $ModLoad imuxsock # provides support for local system logging # $ModLoad ommysql.so # provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 $ModLoad imklog # provides kernel logging support (previously done by rklogd) # provides TCP syslog reception $ModLoad imtcp # make gtls driver the default # $DefaultNetstreamDriver gtls $DefaultNetstreamDriver ptcp # certificate files $DefaultNetstreamDriverCAFile /etc/rsyslog.d/.ssl/gnu-ca-cert.pem $DefaultNetstreamDriverCertFile /etc/rsyslog.d/.ssl/saraha-rsyslog-cert.pem $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/.ssl/saraha-rsyslog-key.pem # $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode # $InputTCPServerStreamDriverAuthMode anon # client is NOT authenticated $InputTCPServerRun 10514 # start up listener at port 10514 $ModLoad MySQL ########################### ########################### #### GLOBAL DIRECTIVES #### ########################### # # Use default timestamp format. # To enable high precision timestamps, comment out the following line. # $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # # Set the default permissions for all log files. # $FileOwner root $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $WorkDirectory /var/log/rsyslog $template mysql-template, "insert into logs(host, facility, priority, level, tag, datetime, msg) values ('%source%', '%syslogfacility-text%', '%syslogpriority-text%', '%syslogseverity-text%', '%programname%', '%timereported:::date-mysql%', '%msg%')", sql, mysql # $template DEBUG,"Debug line with all properties:\nFROMHOST: '%FROMHOST%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nrawmsg: '%rawmsg%'\n\n" ############### #### RULES #### ############### #Discard some dross messages # authpriv.info ~ :HOSTNAME, isequal, "last" ~ #Discard router access messages for the script on Prajna that collects the router logs. if $msg contains 'User logged in on TELNET (192.168.1.2)' then ~ if $msg contains 'User logged out on TELNET (192.168.1.2)' then ~ # Log everything else to mysql. $ActionQueueType LinkedList # Number of elements... $ActionQueueSize 100 # $ActionQueueFileName mysql # $ActionQueueMaxDiskSpace 1M # $ActionQueueHighWaterMark 40 # $ActionQueueLowWaterMark 5 *.* >127.0.0.1,syslog,syslog,syslog;mysql-template $ActionExecOnlyWhenPreviousIsSuspended on & ~ $ActionExecOnlyWhenPreviousIsSuspended off #Log local stuff ONLY to /var/log/syslog :HOSTNAME, isequal, "prajna" -/var/log/syslog -- Jack. From rgerhards at hq.adiscon.com Thu Nov 5 17:42:01 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:42:01 +0100 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com><200910201639.43143.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> <200911031158.02666.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710335F@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, November 03, 2009 11:58 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Hello Rainer, > > is there some news about this issue? Unfortuantely not at the moment (I guess you see it is a busy time). Could you give 5.3.4 a try? It would be very interesting (even for a v4-fix) to see if the problem persists or not... Rainer > > TIA > -Marc > > Am Dienstag, 20. Oktober 2009 18:38:16 schrieb Rainer Gerhards: > > thanks! > > > > Interesting, I see that only one of the messages that should be in > the .qi > > are actually present. I wonder if this is related to the fact that > ompgsql > > emits an error message itself while being down (the others don't do > this > > AFIK). Will try to dig down to this (but I have to do so as time > permits). > > > > Rainer > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Nov 5 17:45:07 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:45:07 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> Thanks, but I wasn't specific enough. For TLS, I also need to client config, because I need two machines to reproduce any issues (these two instances are also the challenge for the current testbench, what requires hopefully fewer than I expect changes ;)). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Thursday, November 05, 2009 5:41 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Rainer Gerhards wrote: > > Can you send me your rsyslog.conf, so that I can run it under the > memory > > debugger in my lab. I'll also take this as a motivation to finally > add > > multi-daemon tests to the testbench (what may take me a little > while...). > > This is the server config (some of the remarks are misleading). > > # /etc/rsyslog.conf Configuration file for rsyslog v3. > # > # For more information see > # /usr/share/doc/rsyslog- > doc/html/rsyslog_conf.html > > # $DebugPrintTemplateList on > # $ActionFileDefaultTemplate mysql-template > ################# > #### MODULES #### > ################# > > $ModLoad imuxsock # provides support for local system logging > > # $ModLoad ommysql.so > > # provides UDP syslog reception > $ModLoad imudp > $UDPServerRun 514 > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > > # provides TCP syslog reception > $ModLoad imtcp > > # make gtls driver the default > # $DefaultNetstreamDriver gtls > $DefaultNetstreamDriver ptcp > > # certificate files > $DefaultNetstreamDriverCAFile /etc/rsyslog.d/.ssl/gnu-ca-cert.pem > $DefaultNetstreamDriverCertFile /etc/rsyslog.d/.ssl/saraha-rsyslog- > cert.pem > $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/.ssl/saraha-rsyslog- > key.pem > > # $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode > # $InputTCPServerStreamDriverAuthMode anon # client is NOT > authenticated > $InputTCPServerRun 10514 # start up listener at port 10514 > > $ModLoad MySQL > > ########################### > ########################### > #### GLOBAL DIRECTIVES #### > ########################### > > # > # Use default timestamp format. > # To enable high precision timestamps, comment out the following line. > # > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > # > # Set the default permissions for all log files. > # > $FileOwner root > $FileGroup adm > $FileCreateMode 0640 > $DirCreateMode 0755 > $WorkDirectory /var/log/rsyslog > > $template mysql-template, "insert into logs(host, facility, priority, > level, tag, datetime, msg) values ('%source%', '%syslogfacility-text%', > '%syslogpriority-text%', '%syslogseverity-text%', '%programname%', > '%timereported:::date-mysql%', '%msg%')", sql, mysql > > # $template DEBUG,"Debug line with all properties:\nFROMHOST: > '%FROMHOST%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag > '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', > PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', > STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nrawmsg: > '%rawmsg%'\n\n" > > ############### > #### RULES #### > ############### > #Discard some dross messages > # authpriv.info ~ > :HOSTNAME, isequal, "last" ~ > > #Discard router access messages for the script on Prajna that collects > the router logs. > if $msg contains 'User logged in on TELNET (192.168.1.2)' then ~ > if $msg contains 'User logged out on TELNET (192.168.1.2)' then ~ > > # Log everything else to mysql. > $ActionQueueType LinkedList > # Number of elements... > $ActionQueueSize 100 > # $ActionQueueFileName mysql > # $ActionQueueMaxDiskSpace 1M > # $ActionQueueHighWaterMark 40 > # $ActionQueueLowWaterMark 5 > > > *.* > >127.0.0.1,syslog,syslog,syslog;mysql-template > > $ActionExecOnlyWhenPreviousIsSuspended on > & ~ > $ActionExecOnlyWhenPreviousIsSuspended off > > #Log local stuff ONLY to /var/log/syslog > :HOSTNAME, isequal, "prajna" -/var/log/syslog > > > -- > Jack. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Thu Nov 5 17:50:14 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 5 Nov 2009 17:50:14 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: 2009/11/5 : > On Thu, 5 Nov 2009, Michael Biebl wrote: > >> What about the -c compat flag? Is it intentional that there is no -c 5? > > are you sure there isn't? I've been running dev versions of 5.x and hav > eneeded to use -c5 with them instead of -c4 Well, at least with 5.2.0, rsyslogd does not complain if I start it with -c4 (complain in the sense, that rsyslog says it's running in non-native mode) >> I will probably upload a 5.2.* to unstable (or experimental) soon, >> which should give it some wider exposure. > > I would not use 5.2.0, it had many problems under load (and problems where > invalid syslog messages could cause it to crash) > > I'm currently running a slightly later git snapshot reliably under fairly > heavy load (>10k logs/sec sustained for hours at a time). unfortunantly I am > 3 timezones away from the office or I would upgrade to the 5.3 that was just > released. I will proabaly do so tuesday when I am back in the office. That's good to know. It seems the 5.2.* series is some kind of neglected so maybe it's best to just skip that branch. and wait for 5.4.* or stable 5.3.* beta releases. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Thu Nov 5 17:51:05 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 5 Nov 2009 17:51:05 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710335B@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710335A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710335B@GRFEXC.intern.adiscon.com> Message-ID: 2009/11/5 Rainer Gerhards : > Hi Michael, > > Is this a standard debian config? I ask because it works well for me... It's a standard Debian config as shipped with version 4.4.1-1 > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Michael Biebl >> Sent: Thursday, November 05, 2009 5:03 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] rsyslog v5 5.2.0 >> >> Another issue I noticed with 5.2.0 (+said patch) is, that it no longer >> shuts down cleany on SIGTERM/SIGINT >> If I run it with -c 4 -d and press STRG+C, I get >> ... >> 6895.847489380:b7692940: Terminating outputs... >> 6895.847516758:b7692940: destructing ruleset 0x89c7648, name 0x89c7678 >> 6895.847553634:b7692940: strm 0x89c97b8: file -1 flush, buflen 0 >> 6895.847595539:b7692940: strm 0x89ca1d8: file 5 flush, buflen 0 >> 6895.847627386:b7692940: strm 0x89ca1d8: file 5 closing >> 6895.847688567:b7692940: strm 0x89cac48: file -1 flush, buflen 0 >> 6895.847728796:b7692940: strm 0x89cb680: file 6 flush, buflen 0 >> 6895.847758967:b7692940: strm 0x89cb680: file 6 closing >> 6895.847805062:b7692940: strm 0x89cc108: file -1 flush, buflen 0 >> 6895.847846129:b7692940: strm 0x89ccb78: file -1 flush, buflen 0 >> 6895.847886916:b7692940: strm 0x89cd5c0: file -1 flush, buflen 0 >> 6895.847927145:b7692940: strm 0x89ce060: file -1 flush, buflen 0 >> 6895.847967653:b7692940: strm 0x89cea98: file -1 flush, buflen 0 >> 6895.848007882:b7692940: strm 0x89cf500: file -1 flush, buflen 0 >> 6895.848048110:b7692940: strm 0x89cff68: file -1 flush, buflen 0 >> 6895.848088897:b7692940: strm 0x89d09d8: file -1 flush, buflen 0 >> 6895.848129126:b7692940: strm 0x89d1478: file -1 flush, buflen 0 >> 6895.848169355:b7692940: strm 0x89d1ee8: file -1 flush, buflen 0 >> 6895.848220758:b7692940: strm 0x89d2990: file 7 flush, buflen 0 >> 6895.848251488:b7692940: strm 0x89d2990: file 7 closing >> 6895.848300098:b7692940: strm 0x89d38c0: file -1 flush, buflen 77 >> >> >> pressing STRG+C a second time will then terminate rsyslogd. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Nov 5 17:54:10 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 5 Nov 2009 17:54:10 +0100 Subject: [rsyslog] rsyslog v5 5.2.0 References: <9B6E2A8877C38245BFB15CC491A11DA710333A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103361@GRFEXC.intern.adiscon.com> > That's good to know. It seems the 5.2.* series is some kind of > neglected so maybe it's best to just skip that branch. and wait for > 5.4.* or stable 5.3.* beta releases. As I said on-list (and within the announcement), I needed to get a 5.2.0 version out. The problem was that there was so few feedback on 5.1.x that it looked like it worked. However, I knew that there are problems with queue termination. That all is solved (hopefully ;)) in the newer v5 versions, but I can't renumber them to a lower version to make them 5.2.x stable). Still it would be good if we could push out some of the new 5.3.x builds, maybe after they become beta, because otherwise we may end up with the same problem... (no testers, no feedback, no bugs, but still not working while I pursue for other things, thus never arriving at a really stable build...) Rainer From mrdemeanour at jackpot.uk.net Thu Nov 5 18:02:23 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Thu, 05 Nov 2009 17:02:23 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> Message-ID: <4AF3051F.5020501@jackpot.uk.net> Rainer Gerhards wrote: > Thanks, but I wasn't specific enough. For TLS, I also need to client config, > because I need two machines to reproduce any issues (these two instances are > also the challenge for the current testbench, what requires hopefully fewer > than I expect changes ;)). Sorry, Rainer. Anyway, I sent you the *current* config, i.e. using ptls. Here are the two configs using gnutls. But NOTE: I'm not using your default MySQL schema; you can't just drop this into your testlab. It should work if you ignore the custom MySQL template. I *really* doubt this has anything to do with MySQL - I've been using this MySQL setup for a year. Also note that there's nothing included from /etc/rsyslog.d - that directory is empty. These are the complete configs. There are a bunch of *Queue* directives in these files, both active and commented-out; I started playing around with queues to see if I could straighten it out that way, but it didn't work. That is, the problem should occur with a default queueing setup. ====== Start Server ========= # /etc/rsyslog.conf Configuration file for rsyslog v3. # # For more information see # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html # $DebugPrintTemplateList on # $ActionFileDefaultTemplate mysql-template ################# #### MODULES #### ################# $ModLoad imuxsock # provides support for local system logging # $ModLoad ommysql.so # provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 $ModLoad imklog # provides kernel logging support (previously done by rklogd) # provides TCP syslog reception $ModLoad imtcp # make gtls driver the default $DefaultNetstreamDriver gtls # $DefaultNetstreamDriver ptcp # certificate files $DefaultNetstreamDriverCAFile /etc/rsyslog.d/.ssl/gnu-ca-cert.pem $DefaultNetstreamDriverCertFile /etc/rsyslog.d/.ssl/saraha-rsyslog-cert.pem $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/.ssl/saraha-rsyslog-key.pem $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode $InputTCPServerStreamDriverAuthMode anon # client is NOT authenticated $InputTCPServerRun 10514 # start up listener at port 10514 $ModLoad MySQL ########################### #### GLOBAL DIRECTIVES #### ########################### # # Use default timestamp format. # To enable high precision timestamps, comment out the following line. # $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # # Set the default permissions for all log files. # $FileOwner root $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $WorkDirectory /var/log/rsyslog $template mysql-template, "insert into logs(host, facility, priority, level, tag, datetime, msg) values ('%source%', '%syslogfacility-text%', '%syslogpriority-text%', '%syslogseverity-text%', '%programname%', '%timereported:::date-mysql%', '%msg%')", sql, mysql # $template DEBUG,"Debug line with all properties:\nFROMHOST: '%FROMHOST%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nrawmsg: '%rawmsg%'\n\n" ############### #### RULES #### ############### #Discard some dross messages # authpriv.info ~ :HOSTNAME, isequal, "last" ~ #Discard router access messages for the script on Prajna that collects the router logs. if $msg contains 'User logged in on TELNET (192.168.1.2)' then ~ if $msg contains 'User logged out on TELNET (192.168.1.2)' then ~ # Log everything else to mysql. $ActionQueueType LinkedList # Number of elements... $ActionQueueSize 100 # $ActionQueueFileName mysql # $ActionQueueMaxDiskSpace 1M # $ActionQueueHighWaterMark 40 # $ActionQueueLowWaterMark 5 *.* >127.0.0.1,syslog,syslog,syslog;mysql-template $ActionExecOnlyWhenPreviousIsSuspended on & ~ $ActionExecOnlyWhenPreviousIsSuspended off #Log local stuff ONLY to /var/log/syslog :HOSTNAME, isequal, "prajna" -/var/log/syslog ====== End Server ========= ====== Start Client ========= # /etc/rsyslog.conf Configuration file for rsyslog v3. # # For more information see # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html ################# #### MODULES #### ################# $ModLoad imuxsock # provides support for local system logging $ModLoad imklog # provides kernel logging support (previously done by rklogd) #$ModLoad immark # provides --MARK-- message capability # provides UDP syslog reception #$ModLoad imudp #$UDPServerRun 514 # provides TCP syslog reception # $ModLoad imtcp # $InputTCPServerRun 514 # certificate files - just CA for a client $DefaultNetstreamDriverCAFile /etc/rsyslog.d/.ssl/gnu-ca-cert.pem # set up the action $DefaultNetstreamDriver gtls # use gtls netstream driver # $DefaultNetstreamDriver ptcp # use default netstream driver $ActionSendStreamDriverMode 1 # require TLS for the connection $ActionSendStreamDriverAuthMode anon # server is NOT authenticated ########################### #### GLOBAL DIRECTIVES #### ########################### # # Use traditional timestamp format. # To enable high precision timestamps, comment out the following line. # $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # # Set the default permissions for all log files. # $FileOwner root $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 # # Include all config files in /etc/rsyslog.d/ # $IncludeConfig /etc/rsyslog.d/*.conf ############### #### RULES #### ############### #Encrypted TCP log to database on Prajna *.* @@87.194.213.229:10514 # *.* -/var/log/syslog ====== End Client ========= > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour >> Sent: Thursday, November 05, 2009 5:41 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls >> >> Rainer Gerhards wrote: >>> Can you send me your rsyslog.conf, so that I can run it under the >> memory >>> debugger in my lab. I'll also take this as a motivation to finally >> add >>> multi-daemon tests to the testbench (what may take me a little >> while...). >> >> This is the server config (some of the remarks are misleading). >> >> # /etc/rsyslog.conf Configuration file for rsyslog v3. >> # >> # For more information see >> # /usr/share/doc/rsyslog- >> doc/html/rsyslog_conf.html >> >> # $DebugPrintTemplateList on >> # $ActionFileDefaultTemplate mysql-template >> ################# >> #### MODULES #### >> ################# >> >> $ModLoad imuxsock # provides support for local system logging >> >> # $ModLoad ommysql.so >> >> # provides UDP syslog reception >> $ModLoad imudp >> $UDPServerRun 514 >> $ModLoad imklog # provides kernel logging support (previously done by >> rklogd) >> >> # provides TCP syslog reception >> $ModLoad imtcp >> >> # make gtls driver the default >> # $DefaultNetstreamDriver gtls >> $DefaultNetstreamDriver ptcp >> >> # certificate files >> $DefaultNetstreamDriverCAFile /etc/rsyslog.d/.ssl/gnu-ca-cert.pem >> $DefaultNetstreamDriverCertFile /etc/rsyslog.d/.ssl/saraha-rsyslog- >> cert.pem >> $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/.ssl/saraha-rsyslog- >> key.pem >> >> # $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode >> # $InputTCPServerStreamDriverAuthMode anon # client is NOT >> authenticated >> $InputTCPServerRun 10514 # start up listener at port 10514 >> >> $ModLoad MySQL >> >> ########################### >> ########################### >> #### GLOBAL DIRECTIVES #### >> ########################### >> >> # >> # Use default timestamp format. >> # To enable high precision timestamps, comment out the following line. >> # >> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >> >> # >> # Set the default permissions for all log files. >> # >> $FileOwner root >> $FileGroup adm >> $FileCreateMode 0640 >> $DirCreateMode 0755 >> $WorkDirectory /var/log/rsyslog >> >> $template mysql-template, "insert into logs(host, facility, priority, >> level, tag, datetime, msg) values ('%source%', '%syslogfacility-text%', >> '%syslogpriority-text%', '%syslogseverity-text%', '%programname%', >> '%timereported:::date-mysql%', '%msg%')", sql, mysql >> >> # $template DEBUG,"Debug line with all properties:\nFROMHOST: >> '%FROMHOST%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag >> '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', >> PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', >> STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nrawmsg: >> '%rawmsg%'\n\n" >> >> ############### >> #### RULES #### >> ############### >> #Discard some dross messages >> # authpriv.info ~ >> :HOSTNAME, isequal, "last" ~ >> >> #Discard router access messages for the script on Prajna that collects >> the router logs. >> if $msg contains 'User logged in on TELNET (192.168.1.2)' then ~ >> if $msg contains 'User logged out on TELNET (192.168.1.2)' then ~ >> >> # Log everything else to mysql. >> $ActionQueueType LinkedList >> # Number of elements... >> $ActionQueueSize 100 >> # $ActionQueueFileName mysql >> # $ActionQueueMaxDiskSpace 1M >> # $ActionQueueHighWaterMark 40 >> # $ActionQueueLowWaterMark 5 >> >> >> *.* >> >127.0.0.1,syslog,syslog,syslog;mysql-template >> >> $ActionExecOnlyWhenPreviousIsSuspended on >> & ~ >> $ActionExecOnlyWhenPreviousIsSuspended off >> >> #Log local stuff ONLY to /var/log/syslog >> :HOSTNAME, isequal, "prajna" -/var/log/syslog >> >> >> -- >> Jack. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Thu Nov 5 23:22:04 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Thu, 5 Nov 2009 23:22:04 +0100 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710335F@GRFEXC.intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200911031158.02666.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA710335F@GRFEXC.intern.adiscon.com> Message-ID: <200911052322.04174.marc.schiffbauer@mightycare.de> Hello Rainer, I will give it a try and send you the feedback as soon as I am back from holiday. This will be in the beginning of december... Thanks -Marc Am Donnerstag, 5. November 2009 17:42:01 schrieb Rainer Gerhards: > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > > Sent: Tuesday, November 03, 2009 11:58 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > > Hello Rainer, > > > > is there some news about this issue? > > Unfortuantely not at the moment (I guess you see it is a busy time). Could > you give 5.3.4 a try? It would be very interesting (even for a v4-fix) to > see if the problem persists or not... > > Rainer > From rgerhards at hq.adiscon.com Fri Nov 6 11:54:50 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 6 Nov 2009 11:54:50 +0100 Subject: [rsyslog] message parser info - was: rsyslog 5.3.4 (devel) released References: <9B6E2A8877C38245BFB15CC491A11DA7103340@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710336D@GRFEXC.intern.adiscon.com> David, all, I hope I have answered all questions in this new document: http://www.rsyslog.com/doc-messageparser.html I would also appreciate feedback on that part: ====== Note that it is currently under evaluation if rsyslog will support binding parser chains to specific inputs directly, without depending on the ruleset. There are some concerns that this may not be necessary but adds considerable complexity to the configuration. So this may or may not be possible in the future. In any case, if we decide to add it, input modules need to support it, so this functionality would require some time to implement. ====== If I implement this, a listener would probably have a parser chain assigned, it not, the parser chain from the ruleset is used, if not, the parser chain from the default ruleset is used. I it (fairly...) trivial to add this capability, but I am really concerned that the config options get more and more complex... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, November 05, 2009 8:53 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 5.3.4 (devel) released > > yOn Thu, 5 Nov 2009, Tom Bergfeld wrote: > > > We have just released rsyslog 5.3.4, a member of the v5-devel branch. > > Today's release offers a number of important improvements. I would > like to > > highlight the ability to nest rulesets [1] (I already announced that) > and the > > ability to define custom parsers [2]. > > > > Let me elaborate a bit on the later. In the syslog world exists a > myriad of > > incompatible and invalidly formatted messages. We have had numerous > support > > requests on how to parse this or that malformed message format. With > custom > > parsers, we now have a vehicel to integrate all these invalid formats > in an > > elegant way that is also offering high performance ([2] has a > concrete > > sample). The core idea now is that parsers are plugins just like > input and > > output modules. It is relatively easy to write such a parser (roughly > a day, > > I expect) and custom parsers can be integrated into rsyslogd in a way > that > > they only affect messages received on specific port. So far, I have > not > > actually created a custom parser, but I hope that over time many of > them will > > become available. My hope here is that we actually can build a > directory, > > which other folks can browse so that they are able to find a solution > to the > > mess their vendor has provided ;) > > this sounds very interesting. however, reading the link I'm a little > confused. > > on the one hand it talks a lot about the priority of parsers, but then > it > talks about binding different parsers to different ports. It's not > clear > if these are two different ways to use parsers, or how these two would > interact. > > where can these parsers be used? > > obviously the imudp module can use them. can all input modules use > them? > > what are the limits of the parser? > > a couple of extreme examples: > > could you implement relp as a parser for imtcp, or since relp sends > replies that can't be done (limiting it to different ways of processing > a > message that's already been received) > > could a parser detect that it had a piece of a multi-line messsage and > stash the piece until it receives the rest of the message so that it > could > submit the complete message? > > > a coouple questions from trying to look through the code at almost 3am > local time > > if it can easily be done, may I suggest changing the santization flag > from > a yes/no boolean to an enum so that there can be more than one > sanitation > routine > > do you have the default parser broken out as a seperate file tha canbe > used as an example? > > David Lang > > > This release also introduces the capability to run rulesets off their > own > > queue (also already mentioned on the mailing list). Plus, it contains > the > > usual set of bug fixes. > > > > We plan to make this version the basis for the next v5-stable. > > > > > > [1] http://www.rsyslog.com/doc-omruleset.html > > [2] http://www.rsyslog.com/doc-rsconf1_rulesetparser.html > > > > > > > > ChangeLog: > > > > http://www.rsyslog.com/Article423.phtml > > > > Download: > > > > http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid- > 185.phtml > > > > As always, feedback is appreciated. > > > > Best regards, > > Tom Bergfeld > > -- > > Support > > ======= > > > > Improving rsyslog is costly, but you can help! We are looking for > > organizations that find rsyslog useful and wish to contribute back. > You can > > contribute by reporting bugs, improve the software, or donate money > or > > equipment. > > > > Commercial support contracts for rsyslog are available, and they help > finance > > continued maintenance. Adiscon GmbH, a privately held German > company, is > > currently funding rsyslog development. We are always looking for > interesting > > development projects. For details on how to help, please see > > http://www.rsyslog.com/doc-how2help.html . > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From correo at miguelangelnieto.net Fri Nov 6 12:31:04 2009 From: correo at miguelangelnieto.net (Miguel Angel Nieto) Date: Fri, 6 Nov 2009 12:31:04 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: I made rpm packages with the stable v3 of rsyslog for suse 10.3. If anyone need them, tell me. > in other cases you are willing to loose logs rather than freezing the > machine and can configure rsyslog to accept messages, even when it can't > do anything with them to avoid this sort of lockup. How can I do to tell rsyslog to accept all messages and not use any queue? -- Lo que har?a ser?a hacerme pasar por sordomudo y as? no tendr?a que hablar. Si quer?an decirme algo, tendr?an que escribirlo en un papelito y ense??rmelo. Al final se hartar?an y ya no tendr?a que hablar el resto de mi vida. From rgerhards at hq.adiscon.com Fri Nov 6 14:27:20 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 6 Nov 2009 14:27:20 +0100 Subject: [rsyslog] potential server problems on rsyslog.com Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103373@GRFEXC.intern.adiscon.com> Hi all, I was just told that http://www.rsyslog.com, associated services and some other sites (like the knowledge base) will probably experience some temporary outage within the next couple of hours. Looks like the sysadmins need to fix something urgently. So if you got a connection problem, please retry a little bit later. Thanks, Rainer From jbondc at openmv.com Fri Nov 6 16:18:55 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Fri, 6 Nov 2009 10:18:55 -0500 Subject: [rsyslog] Postgresql module: standard_conforming_strings Message-ID: <002801ca5ef4$74e7cd20$5eb76760$@com> I have two questions: 1) How does rsyslog espace SQL commands in the plugin ompgsql.c? I've been staring at the code but I can't figure out where the backslash escaping happens. My problem is I've set: standard_conforming_strings=On This means that the backslash espace ' gfgfdg \' ' is ignored and causes errors. There are many ways to fix this. a) Have rsyslog issue (SET standard_conforming_strings = off;) for postgresql 8.2+ (quick fix) b) Change default sql espacing to use doubles quotes '' --- so 'test \' ' becomes 'test '' ' http://www.postgresql.org/docs/8.1/static/libpq-exec.html#LIBPQ-EXEC-ESCAPE- STRING 2) Is there an SVN repository of rsyslog? Or does anyone know of a good way or using GIT on windows (turtoisegit is currently buggy) ? From rgerhards at hq.adiscon.com Fri Nov 6 16:45:33 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 6 Nov 2009 16:45:33 +0100 Subject: [rsyslog] Postgresql module: standard_conforming_strings References: <002801ca5ef4$74e7cd20$5eb76760$@com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103377@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Jonathan Bond-Caron > Sent: Friday, November 06, 2009 4:19 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Postgresql module: standard_conforming_strings > > I have two questions: > > > > 1) How does rsyslog espace SQL commands in the plugin ompgsql.c? > > > > I've been staring at the code but I can't figure out where the > backslash > escaping happens. > > > > My problem is I've set: standard_conforming_strings=On > > This means that the backslash espace ' gfgfdg \' ' is ignored and > causes > errors. There are many ways to fix this. > > > > a) Have rsyslog issue (SET standard_conforming_strings = off;) for > postgresql 8.2+ (quick fix) > > b) Change default sql espacing to use doubles quotes '' --- so > 'test \' > ' becomes 'test '' ' > > http://www.postgresql.org/docs/8.1/static/libpq-exec.html#LIBPQ-EXEC- > ESCAPE- > STRING > This is controlled via the template. For postgre SQL, I think it needs the STDSQL option. I will check if ompgsql requests that option. $template name,"insert ...",STDSQL The string must be sanitized before it is passed down to the module, because the module does not know any longer what a field is. > > > 2) Is there an SVN repository of rsyslog? No > Or does anyone know of a > good > way or using GIT on windows (turtoisegit is currently buggy) ? > My co-workers use http://code.google.com/p/msysgit/ and don't complain (too much ;)) about it. HTH Rainer > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From epiphani at gmail.com Fri Nov 6 19:18:37 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Fri, 6 Nov 2009 13:18:37 -0500 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103338@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103338@GRFEXC.intern.adiscon.com> Message-ID: Wanted to point out that I've observed this now in a second environment - so we can eliminate the possibility that this is specific to that situation. I'll be working to reproduce the issue in a separate system next week, and hopefully I can get something reproducable so I can test the updated version. Thanks! -Aaron On Wed, Nov 4, 2009 at 10:06 AM, Rainer Gerhards wrote: > >> that can work, however XML output may not be too bad, people entering >> commands manually are probably not going to be asking for that much >> data. > > If I look at time constraints, I will probably not able to define a big CLI > with a real language. So I think you might send something like "queuestatus" > and it will reply with all status information on all queues it has. > >> the biggest problem I see with XML is the need to send requests and >> responses via e-mail for debugging. if one end uses a HTML enabled e- >> mail >> client it may not work well to paste XML text into it. > > That's a very good point and I really should begin to think about this right > now. Some kind of "debug package" that's easily mailable would not be a bad > thing ;) > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Nov 6 19:36:03 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 6 Nov 2009 19:36:03 +0100 Subject: [rsyslog] Queuing bug (4.5.5) / Problem in production References: <9B6E2A8877C38245BFB15CC491A11DA7103335@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103337@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103338@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710337D@GRFEXC.intern.adiscon.com> excellent news. I have reviewed the debug log handling. It looks like we can use the current version, mabe just slightly modified, to generate debug output when a problem occurs by sending SIGUSR1 to it. I hope to be able to complete testing on monday. If it works out like I expect, I'll do a write-up of how it can be done. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Friday, November 06, 2009 7:19 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing bug (4.5.5) / Problem in production > > Wanted to point out that I've observed this now in a second > environment - so we can eliminate the possibility that this is > specific to that situation. > > I'll be working to reproduce the issue in a separate system next week, > and hopefully I can get something reproducable so I can test the > updated version. > > Thanks! > -Aaron > > On Wed, Nov 4, 2009 at 10:06 AM, Rainer Gerhards > wrote: > > > >> that can work, however XML output may not be too bad, people > entering > >> commands manually are probably not going to be asking for that much > >> data. > > > > If I look at time constraints, I will probably not able to define a > big CLI > > with a real language. So I think you might send something like > "queuestatus" > > and it will reply with all status information on all queues it has. > > > >> the biggest problem I see with XML is the need to send requests and > >> responses via e-mail for debugging. if one end uses a HTML enabled > e- > >> mail > >> client it may not work well to paste XML text into it. > > > > That's a very good point and I really should begin to think about > this right > > now. Some kind of "debug package" that's easily mailable would not be > a bad > > thing ;) > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Nov 6 19:47:41 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 6 Nov 2009 10:47:41 -0800 (PST) Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: On Fri, 6 Nov 2009, Miguel Angel Nieto wrote: >> in other cases you are willing to loose logs rather than freezing the >> machine and can configure rsyslog to accept messages, even when it can't >> do anything with them to avoid this sort of lockup. > > How can I do to tell rsyslog to accept all messages and not use any queue? you cannot tell rsyslog to not use a queue. the fundamental architecture of rsyslog is that the input modules receive messages, parse minimal info out of them, and put them in a queue.worker threads then pull messages from this queue and send them to their destination. this queue can be in ram, ram + disk file, or disk-only classic syslog doesn't use any queue and must fully process each message before receiving the next message. syslog-ng has the ability to delay writing the logs to disk a bit to do them in batches, but otherwise is the same. in rsyslog, significantly more time is spent in the output side of things than in the input side. historicly, syslog was intended to be a reliable logging mechanism, so applications do not complete their write until the syslog daemon has the log safe on disk. the performance of doing this is so horrible that everybody has some way of relaxing this requirement (the - option in sysklogd, the batch option in syslog-ng, the memory queue in rsyslog), but all of these options only allow a finite number of items to be buffered before the syslog daemon will stop to wait for them to really get to the destination. this is why your systems locked up, they were trying to write to rsyslog on the client, rsyslog on the client was configured to not consider the logs processed until they were acknowledged by the TCP stack of the server. so when the server is not available the clients keep accepting messages and queue them up, but when the queue filled up they stopped accepting new messages. the reason to use TCP for syslog instead of UDP is so that you can have the log server push back on the sender and tell the sender that it can't process the log message right now, the sender should hang on to it and tray again to send it later. if you are really willing to loose logs if the log server is down or backed up, why not just use UDP for the logs? look at the MainMsgQueue* options for setting high watermark, low watermark, discard options, etc if you want to change how rsyslog acts when the queue fills up. I haven't used these, so all I can do is to point you in the right direction. David Lang From mrdemeanour at jackpot.uk.net Fri Nov 6 20:10:15 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 06 Nov 2009 19:10:15 +0000 Subject: [rsyslog] computer hang-up and WorkDirectory In-Reply-To: References: Message-ID: <4AF47497.4070407@jackpot.uk.net> david at lang.hm wrote: > On Fri, 6 Nov 2009, Miguel Angel Nieto wrote: > >>> in other cases you are willing to loose logs rather than freezing >>> the machine and can configure rsyslog to accept messages, even >>> when it can't do anything with them to avoid this sort of lockup. >>> >> How can I do to tell rsyslog to accept all messages and not use any >> queue? > > you cannot tell rsyslog to not use a queue. Is that really true? "Direct queues are non-queuing queues. A queue in direct mode does neither queue nor buffer any of the queue elements but rather passes the element directly (and immediately) from the producer to the consumer. This sounds strange, but there is a good reason for this queue type." [...] "To create a direct queue, use the "$QueueType Direct" config directive." e.g. (I suppose): $MainQueueType Direct http://www.rsyslog.com/doc-queues.html -- Jack. From ryan.b.lynch at gmail.com Fri Nov 6 22:02:40 2009 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Fri, 6 Nov 2009 16:02:40 -0500 Subject: [rsyslog] What backslash-escape sequences are supported in templates? Message-ID: <115906d10911061302h56d1b56cs50247fd010de57a5@mail.gmail.com> The man page gives '\7' (bell) as an example (in the section 'Templates'), and then it says "The set in rsyslog is a bit restricted currently." I couldn't find any more info in the online documentation, in either of the 'Templates' or 'Property Replacer' sections. I just tried to use '\t' (tab) in a log format template, though, and it didn't work--it prints as just 't'. Is there a doc that lists what escapes are and aren't accepted? If not, can somebody who knows the answer enlighten me? Ryan B. Lynch ryan.b.lynch at gmail.com From philr at jaspers.co.nz Sat Nov 7 21:37:59 2009 From: philr at jaspers.co.nz (Phil Reilly) Date: Sun, 08 Nov 2009 09:37:59 +1300 Subject: [rsyslog] Alerting rules via a database Message-ID: <4AF5DAA7.5010001@jaspers.co.nz> I attempting to allow for flexible rule matches on Syslogs from a web front end (rather than entires into the rsyslog config files) I want to get regexp filters from a db to alert upon messages. Not sure the best way to achieve this. I've so far though of. * Outputting to a pipe and runing it via an alerting script. * Having file watch the messages. * Recieving the messages then passing them to rsyslog (yuck) Can the rule engine allow for match rules outside of the config? is there an elegant way of doing this? From david at lang.hm Sun Nov 8 08:44:36 2009 From: david at lang.hm (david at lang.hm) Date: Sat, 7 Nov 2009 23:44:36 -0800 (PST) Subject: [rsyslog] Alerting rules via a database In-Reply-To: <4AF5DAA7.5010001@jaspers.co.nz> References: <4AF5DAA7.5010001@jaspers.co.nz> Message-ID: On Sun, 8 Nov 2009, Phil Reilly wrote: > I attempting to allow for flexible rule matches on Syslogs from a web > front end (rather than entires into the rsyslog config files) > > I want to get regexp filters from a db to alert upon messages. Not sure > the best way to achieve this. I've so far though of. > > * Outputting to a pipe and runing it via an alerting script. > * Having file watch the messages. > * Recieving the messages then passing them to rsyslog (yuck) > > Can the rule engine allow for match rules outside of the config? is > there an elegant way of doing this? rsyslog doesn't give you this ability, but it's not really the best approach to use for alerting anyway. what are you trying to achieve by having the alert definitions in a database? there are several tools out there to do alerting (SEC, Simple Event Correlator) is one of the leading ones, but I'm not aware of any of them that use a database for their rulesets. I'm also scratching my head trying to figure out what the advantage of doing so would be. David Lang From philr at jaspers.co.nz Sun Nov 8 09:29:30 2009 From: philr at jaspers.co.nz (Phil Reilly) Date: Sun, 08 Nov 2009 21:29:30 +1300 Subject: [rsyslog] Alerting rules via a database In-Reply-To: References: <4AF5DAA7.5010001@jaspers.co.nz> Message-ID: <4AF6816A.6050901@jaspers.co.nz> david at lang.hm wrote: > On Sun, 8 Nov 2009, Phil Reilly wrote: > > >> I attempting to allow for flexible rule matches on Syslogs from a web >> front end (rather than entires into the rsyslog config files) >> >> I want to get regexp filters from a db to alert upon messages. Not sure >> the best way to achieve this. I've so far though of. >> >> * Outputting to a pipe and runing it via an alerting script. >> * Having file watch the messages. >> * Recieving the messages then passing them to rsyslog (yuck) >> >> Can the rule engine allow for match rules outside of the config? is >> there an elegant way of doing this? >> > > rsyslog doesn't give you this ability, but it's not really the best > approach to use for alerting anyway. > > what are you trying to achieve by having the alert definitions in a > database? there are several tools out there to do alerting (SEC, Simple > Event Correlator) is one of the leading ones, but I'm not aware of any of > them that use a database for their rulesets. > > I'm also scratching my head trying to figure out what the advantage of > doing so would be. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > Thanks David, We have a networked environment. We also have a web page that allows you to configure regexp to match certain syslog messages. These patterns are compiled and kept in a table. The current syslog process we use listens for udp. When it gets a syslog message, we examine the patterns (which are re-read upon addition or change) and pass them to an alertering process before writing the logs to disk. The existing system works well, but we now want to scale it over a few machines and I'm examining what syslog products out there cater for alerting. So a database will make configuring alerts far more dynamic than statically entering them into a config file. It will also allow for grouped views so different groups have the ability to have custom alerts based upon their own interpretation of syslog messages. From rgerhards at hq.adiscon.com Sun Nov 8 11:48:19 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 8 Nov 2009 11:48:19 +0100 Subject: [rsyslog] Alerting rules via a database References: <4AF5DAA7.5010001@jaspers.co.nz> <4AF6816A.6050901@jaspers.co.nz> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> So what you are actually looking for is a system that can work with dynamically changable alert definitions? As David said, there is no such thing currently, but the best road to approach is is to write a custom output plugin, that you pass each message to. That plugin can even decide if messages should be discarded and not further processed. I envisioned such a plugin, but had not yet time to write, for a similar use case. If you intend to write one AND contribute it to the project, I can help you get started with the interface, would even be willing to create you a custom skeleton that you can fill in your logic ;) HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phil Reilly > Sent: Sunday, November 08, 2009 9:30 AM > To: rsyslog-users > Subject: Re: [rsyslog] Alerting rules via a database > > david at lang.hm wrote: > > On Sun, 8 Nov 2009, Phil Reilly wrote: > > > > > >> I attempting to allow for flexible rule matches on Syslogs > from a web > >> front end (rather than entires into the rsyslog config files) > >> > >> I want to get regexp filters from a db to alert upon > messages. Not sure > >> the best way to achieve this. I've so far though of. > >> > >> * Outputting to a pipe and runing it via an alerting script. > >> * Having file watch the messages. > >> * Recieving the messages then passing them to rsyslog (yuck) > >> > >> Can the rule engine allow for match rules outside of the config? is > >> there an elegant way of doing this? > >> > > > > rsyslog doesn't give you this ability, but it's not really the best > > approach to use for alerting anyway. > > > > what are you trying to achieve by having the alert definitions in a > > database? there are several tools out there to do alerting > (SEC, Simple > > Event Correlator) is one of the leading ones, but I'm not > aware of any of > > them that use a database for their rulesets. > > > > I'm also scratching my head trying to figure out what the > advantage of > > doing so would be. > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > Thanks David, > > We have a networked environment. We also have a web page that > allows you > to configure regexp to match certain syslog messages. These > patterns are > compiled and kept in a table. The current syslog process we > use listens > for udp. When it gets a syslog message, we examine the > patterns (which > are re-read upon addition or change) and pass them to an alertering > process before writing the logs to disk. The existing system > works well, > but we now want to scale it over a few machines and I'm > examining what > syslog products out there cater for alerting. > > So a database will make configuring alerts far more dynamic than > statically entering them into a config file. It will also allow for > grouped views so different groups have the ability to have > custom alerts > based upon their own interpretation of syslog messages. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Sun Nov 8 16:40:50 2009 From: david at lang.hm (david at lang.hm) Date: Sun, 8 Nov 2009 07:40:50 -0800 (PST) Subject: [rsyslog] Alerting rules via a database In-Reply-To: <4AF6816A.6050901@jaspers.co.nz> References: <4AF5DAA7.5010001@jaspers.co.nz> <4AF6816A.6050901@jaspers.co.nz> Message-ID: On Sun, 8 Nov 2009, Phil Reilly wrote: > > david at lang.hm wrote: >> On Sun, 8 Nov 2009, Phil Reilly wrote: >> >> >>> I attempting to allow for flexible rule matches on Syslogs from a web >>> front end (rather than entires into the rsyslog config files) >>> >>> I want to get regexp filters from a db to alert upon messages. Not sure >>> the best way to achieve this. I've so far though of. >>> >>> * Outputting to a pipe and runing it via an alerting script. >>> * Having file watch the messages. >>> * Recieving the messages then passing them to rsyslog (yuck) >>> >>> Can the rule engine allow for match rules outside of the config? is >>> there an elegant way of doing this? >>> >> >> rsyslog doesn't give you this ability, but it's not really the best >> approach to use for alerting anyway. >> >> what are you trying to achieve by having the alert definitions in a >> database? there are several tools out there to do alerting (SEC, Simple >> Event Correlator) is one of the leading ones, but I'm not aware of any of >> them that use a database for their rulesets. >> >> I'm also scratching my head trying to figure out what the advantage of >> doing so would be. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > Thanks David, > > We have a networked environment. We also have a web page that allows you > to configure regexp to match certain syslog messages. These patterns are > compiled and kept in a table. The current syslog process we use listens > for udp. When it gets a syslog message, we examine the patterns (which > are re-read upon addition or change) and pass them to an alertering > process before writing the logs to disk. The existing system works well, > but we now want to scale it over a few machines and I'm examining what > syslog products out there cater for alerting. > > So a database will make configuring alerts far more dynamic than > statically entering them into a config file. It will also allow for > grouped views so different groups have the ability to have custom alerts > based upon their own interpretation of syslog messages. I don't know anything that will read a database like you are lookng for. I think you would be better off having your web gui create SEC rules or something like that (you can still store the basic info in a database) David Lang From jbondc at openmv.com Sun Nov 8 17:42:34 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Sun, 8 Nov 2009 11:42:34 -0500 Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing Message-ID: <003001ca6092$792833d0$6b789b70$@com> Hi all, Attached is a patch for 3 things: a) Check that TCP connection is still alive when using TLS b) Improve TAG parsing so that it ends when it sees a tab or escape control character. c) Added configuration directive: escapecontrolcharactertab In rsyslog.conf, you can set: $escapeControlCharactersOnReceive on $escapeControlCharacterTab off This avoids having a tab being escaped since it is after a viewable character J I'd prefer this being the default behavior but I've left $escapeControlCharacterTab on as default in case people expect tabs to be escaped. This is my first GIT patch, hope it works well ;) From mrdemeanour at jackpot.uk.net Sun Nov 8 18:23:31 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Sun, 08 Nov 2009 17:23:31 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> Message-ID: <4AF6FE93.6080805@jackpot.uk.net> Rainer Gerhards wrote: > Thanks, but I wasn't specific enough. For TLS, I also need to client > config, because I need two machines to reproduce any issues (these > two instances are also the challenge for the current testbench, what > requires hopefully fewer than I expect changes ;)). Perhaps someone could advise me on an alternative to resolving this problem by means of bug-hunting. While 4.4.2 is the version of rsyslog currently shipping in Debian squeeze, it's obviously a bit out-of-date from the point of view of the developers. Well, the reason I'm using that version is precisely that it ships with Debian. Or to be more precise, I upgraded to squeeze because it ships with a version of rsyslog that includes encryption support. So perhaps I should just cut free, and install 5.x stable? -- Jack. From jbondc at openmv.com Sun Nov 8 20:18:43 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Sun, 8 Nov 2009 14:18:43 -0500 Subject: [rsyslog] Postgresql module: standard_conforming_strings In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103377@GRFEXC.intern.adiscon.com> References: <002801ca5ef4$74e7cd20$5eb76760$@com> <9B6E2A8877C38245BFB15CC491A11DA7103377@GRFEXC.intern.adiscon.com> Message-ID: <000601ca60a8$49f44d40$dddce7c0$@com> On Fri Nov 6 10:45 AM, Rainer Gerhards wrote: > > > > > > My problem is I've set: standard_conforming_strings=On > > > > This means that the backslash espace ' gfgfdg \' ' is ignored and > > causes errors. There are many ways to fix this. > > > > This is controlled via the template. For postgre SQL, I think it needs > the STDSQL option. I will check if ompgsql requests that option. > > $template name,"insert ...",STDSQL > Thanks that fixed it, I was using $template name,"insert ...",SQL From philr at jaspers.co.nz Sun Nov 8 20:36:07 2009 From: philr at jaspers.co.nz (Phil Reilly) Date: Mon, 09 Nov 2009 08:36:07 +1300 Subject: [rsyslog] Alerting rules via a database In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> References: <4AF5DAA7.5010001@jaspers.co.nz> <4AF6816A.6050901@jaspers.co.nz> <9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> Message-ID: <4AF71DA7.2090107@jaspers.co.nz> More than happy to Rainer. A Skeleton will be welcome. Rainer Gerhards wrote: > So what you are actually looking for is a system that can work with > dynamically changable alert definitions? As David said, there is no such > thing currently, but the best road to approach is is to write a custom output > plugin, that you pass each message to. That plugin can even decide if > messages should be discarded and not further processed. I envisioned such a > plugin, but had not yet time to write, for a similar use case. > > If you intend to write one AND contribute it to the project, I can help you > get started with the interface, would even be willing to create you a custom > skeleton that you can fill in your logic ;) > > HTH > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phil Reilly >> Sent: Sunday, November 08, 2009 9:30 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Alerting rules via a database >> >> david at lang.hm wrote: >> >>> On Sun, 8 Nov 2009, Phil Reilly wrote: >>> >>> >>> >>>> I attempting to allow for flexible rule matches on Syslogs >>>> >> from a web >> >>>> front end (rather than entires into the rsyslog config files) >>>> >>>> I want to get regexp filters from a db to alert upon >>>> >> messages. Not sure >> >>>> the best way to achieve this. I've so far though of. >>>> >>>> * Outputting to a pipe and runing it via an alerting script. >>>> * Having file watch the messages. >>>> * Recieving the messages then passing them to rsyslog (yuck) >>>> >>>> Can the rule engine allow for match rules outside of the config? is >>>> there an elegant way of doing this? >>>> >>>> >>> rsyslog doesn't give you this ability, but it's not really the best >>> approach to use for alerting anyway. >>> >>> what are you trying to achieve by having the alert definitions in a >>> database? there are several tools out there to do alerting >>> >> (SEC, Simple >> >>> Event Correlator) is one of the leading ones, but I'm not >>> >> aware of any of >> >>> them that use a database for their rulesets. >>> >>> I'm also scratching my head trying to figure out what the >>> >> advantage of >> >>> doing so would be. >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> Thanks David, >> >> We have a networked environment. We also have a web page that >> allows you >> to configure regexp to match certain syslog messages. These >> patterns are >> compiled and kept in a table. The current syslog process we >> use listens >> for udp. When it gets a syslog message, we examine the >> patterns (which >> are re-read upon addition or change) and pass them to an alertering >> process before writing the logs to disk. The existing system >> works well, >> but we now want to scale it over a few machines and I'm >> examining what >> syslog products out there cater for alerting. >> >> So a database will make configuring alerts far more dynamic than >> statically entering them into a config file. It will also allow for >> grouped views so different groups have the ability to have >> custom alerts >> based upon their own interpretation of syslog messages. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Mon Nov 9 05:48:27 2009 From: david at lang.hm (david at lang.hm) Date: Sun, 8 Nov 2009 20:48:27 -0800 (PST) Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <4AF6FE93.6080805@jackpot.uk.net> References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net> Message-ID: On Sun, 8 Nov 2009, Mr. Demeanour wrote: > Rainer Gerhards wrote: >> Thanks, but I wasn't specific enough. For TLS, I also need to client >> config, because I need two machines to reproduce any issues (these >> two instances are also the challenge for the current testbench, what >> requires hopefully fewer than I expect changes ;)). > > Perhaps someone could advise me on an alternative to resolving this > problem by means of bug-hunting. While 4.4.2 is the version of rsyslog > currently shipping in Debian squeeze, it's obviously a bit out-of-date > from the point of view of the developers. the rate of change in rsyslog over the last year is dizzying. This makes rsyslog a much better progam, but it also means that it's very unlikly that the debian maintainers will be able to backport all the fixes. > Well, the reason I'm using that version is precisely that it ships with > Debian. Or to be more precise, I upgraded to squeeze because it ships > with a version of rsyslog that includes encryption support. So perhaps I > should just cut free, and install 5.x stable? unfortunantly the current 5.2 stable release has some known problems. the 5.x development is looking pretty good right now, but it is very much the bleeding edge of things. if you need a newer stable version _now_ go with the latest 4.x if you can affort to test the 5.x development branch, it will probably work (I've found it stable over the last couple of weeks, running at loads of ~10,000 logs/sec), but as always you may run into unexpected problems (like you did with 4.4.2). I don't use encryption so I can't vouch for that part of things. David Lang From rgerhards at hq.adiscon.com Mon Nov 9 06:38:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 9 Nov 2009 06:38:29 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls Message-ID: <000501ca60fe$fac9ae3a$100013ac@intern.adiscon.com> More info follows, but v5 has exactly the same tls subsystem as v4. So we need to track that bug down. ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 09.11.09 05:49 Betreff: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls On Sun, 8 Nov 2009, Mr. Demeanour wrote: > Rainer Gerhards wrote: >> Thanks, but I wasn't specific enough. For TLS, I also need to client >> config, because I need two machines to reproduce any issues (these >> two instances are also the challenge for the current testbench, what >> requires hopefully fewer than I expect changes ;)). > > Perhaps someone could advise me on an alternative to resolving this > problem by means of bug-hunting. While 4.4.2 is the version of rsyslog > currently shipping in Debian squeeze, it's obviously a bit out-of-date > from the point of view of the developers. the rate of change in rsyslog over the last year is dizzying. This makes rsyslog a much better progam, but it also means that it's very unlikly that the debian maintainers will be able to backport all the fixes. > Well, the reason I'm using that version is precisely that it ships with > Debian. Or to be more precise, I upgraded to squeeze because it ships > with a version of rsyslog that includes encryption support. So perhaps I > should just cut free, and install 5.x stable? unfortunantly the current 5.2 stable release has some known problems. the 5.x development is looking pretty good right now, but it is very much the bleeding edge of things. if you need a newer stable version _now_ go with the latest 4.x if you can affort to test the 5.x development branch, it will probably work (I've found it stable over the last couple of weeks, running at loads of ~10,000 logs/sec), but as always you may run into unexpected problems (like you did with 4.4.2). I don't use encryption so I can't vouch for that part of things. David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Nov 9 10:02:40 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 9 Nov 2009 10:02:40 +0100 Subject: [rsyslog] rsyslog priorities - was: RE: Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710338D@GRFEXC.intern.adiscon.com> >From my POV the real issue is that there is very much going on at the moment (I don't want to complain about that). But that also means I need to prioritize the things I do. While I would like to have bugfixing always on top of the list, it is plain truth that there are also some other factors (like how to obtain bread for breakfast ;)), so priorities are also dictated by commercial activities that need to run alongside. As a reminder, rsyslog development is still very largely depending on Adiscon funding and that also means things like EventReporter actually pay for it (some people may think well about that fact and its implications ;)). If I worked on rsyslog just as a byline hobby activity, we were nowhere close to where we are right now. But, as I said: the bottom line is that priorities come with a twist of "as time permits". Anyhow, I am quite concerned about this bug and hope to be able to look at it as soon as possible. As another side-node running development versions and providing good bug reports is also an excellent way to contribute to the project... If you followed the mailing list, my priority scheme also slightly involves that bug reports/feature whishes from those that contribute frequently have a somewhat higher priority than from those who do not. I find this is only fair. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Sunday, November 08, 2009 6:24 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Rainer Gerhards wrote: > > Thanks, but I wasn't specific enough. For TLS, I also need to client > > config, because I need two machines to reproduce any issues (these > > two instances are also the challenge for the current testbench, what > > requires hopefully fewer than I expect changes ;)). > > Perhaps someone could advise me on an alternative to resolving this > problem by means of bug-hunting. While 4.4.2 is the version of rsyslog > currently shipping in Debian squeeze, it's obviously a bit out-of-date > from the point of view of the developers. > > Well, the reason I'm using that version is precisely that it ships with > Debian. Or to be more precise, I upgraded to squeeze because it ships > with a version of rsyslog that includes encryption support. So perhaps > I > should just cut free, and install 5.x stable? > > -- > Jack. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mrdemeanour at jackpot.uk.net Mon Nov 9 10:13:56 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Mon, 09 Nov 2009 09:13:56 +0000 Subject: [rsyslog] rsyslog priorities - was: RE: Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710338D@GRFEXC.intern.adiscon.com> References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA710338D@GRFEXC.intern.adiscon.com> Message-ID: <4AF7DD54.6080001@jackpot.uk.net> Rainer Gerhards wrote: > While I would like to have bugfixing always on top of the list, it is > plain truth that there are also some other factors (like how to > obtain bread for breakfast ;)), so priorities are also dictated by > commercial activities that need to run alongside. I wouldn't dream of questioning your priorities. I don't know how you manage to do so much development, while at the same time attending to this list, participating in general syslog standards-related actvities and so on. Frankly, I'm rather surprised to learn that you actually have time for breakfast! I think I shall try 5.2 on the server; I'll let you know if the trouble recurs. -- Jack. From martinmie at PartyGaming.com Mon Nov 9 10:15:13 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Mon, 9 Nov 2009 10:15:13 +0100 Subject: [rsyslog] rsyslog 5.2.0 is a zombie Message-ID: <0B1B9163138571439EAF804646F3F6AA08A58AA7@GIBSVWIN004X.partygaming.local> Hi, Just for the records... uur rsyslog servers, both running on the stable 5.2.0, died over the weekend. Well, to be more precise, they became zombies: # ps -efl | grep rsyslog 5 Z root 2565 1 73 77 0 - 0 exit Nov05 ? 2-20:44:14 [rsyslogd] ...so we sent from an "uninterruptible sleep" on 4.2.0 to a more "eternal sleep" LOL. I think it's time to revert to the latest v4 branch... suggestions? Cheers, Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From rgerhards at hq.adiscon.com Mon Nov 9 11:35:26 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 9 Nov 2009 11:35:26 +0100 Subject: [rsyslog] computer hang-up and WorkDirectory References: <4AF47497.4070407@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103392@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Friday, November 06, 2009 8:10 PM > To: rsyslog-users > Subject: Re: [rsyslog] computer hang-up and WorkDirectory > > david at lang.hm wrote: > > On Fri, 6 Nov 2009, Miguel Angel Nieto wrote: > > > >>> in other cases you are willing to loose logs rather than freezing > >>> the machine and can configure rsyslog to accept messages, even > >>> when it can't do anything with them to avoid this sort of lockup. > >>> > >> How can I do to tell rsyslog to accept all messages and not use any > >> queue? > > > > you cannot tell rsyslog to not use a queue. > > Is that really true? > > "Direct queues are non-queuing queues. A queue in direct mode does > neither queue nor buffer any of the queue elements but rather passes > the > element directly (and immediately) from the producer to the consumer. > This sounds strange, but there is a good reason for this queue type." > > [...] > > "To create a direct queue, use the "$QueueType Direct" config > directive." > > e.g. (I suppose): > $MainQueueType Direct > > > http://www.rsyslog.com/doc-queues.html Sorry, I overlooked that message first. You are right. While rsyslog needs queues (more precisely: queue objects), one can configure queue objects not to queue. This is the default case for actions. However, you most often do not want to set the main queue to direct. But you can and there are use cases for it. The bottom line than is that sender and action processing are tightly couple, so with inputs like UDP you will most probably lose a lot of messages, especially when using slow backends like databases. One cure is to run those slow backends then on async action queues. HTH Rainer From rgerhards at hq.adiscon.com Mon Nov 9 12:24:04 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 9 Nov 2009 12:24:04 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103395@GRFEXC.intern.adiscon.com> Jack, Quick update: I was able to run a (relatively quick) test on a configuration that hopefully later becomes part of the testbench (I am working towards that goal and thought doing a manual check at that stage would not hurt). It is not based your config, and it is relatively simple and straightforward. Still, it uses TLS, and uses it in anon mode like you did. I used 4.4.2 on the server. I processed around 500,000 message (not kept track of the actual number). I did three such runs, all runing under the excellent valgrind[1] memory debugger. For none of the runs, valgrind reported any memory leaks. While this may not be an ultimate indication, valgrind is *very* effective in finding leaks and based on what you wrote I would have expected a small chunk of memory to be lost per message. So the bottom line is that I currently cannot reproduce the bug. This may change when I finally import your config. However, it would be useful for me if you could run valgrind on rsyslogd in your environment and let me know if valgrind reports any memory leaks. Doing so considerably slows down rsyslogd, but given your load, I'd expect that it would be acceptable. To run under valgrind is relatively simply. Valgrind is available as a package on almost all distros. All you need to do is run valgrind and specify your usual rsyslogd command line as the parameter. It is recommended to do this in the foreground (see rsyslog troubleshooting doc). So, for example, if you start rsyslogd usually by /sbin/rsyslogd -c4 -... you simply do valgrind /sbin/rsyslogd -c4 -... instead. By default, valgrind message go to stdout. At the end of the run (kill `rsyslog.pid`) valgrind prints out memory leaks it detects. During the run, it prints out any anomalies it sees. Would be great if you could give that a try. Rainer [1] http://www.valgrind.org > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Sunday, November 08, 2009 6:24 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Rainer Gerhards wrote: > > Thanks, but I wasn't specific enough. For TLS, I also need to client > > config, because I need two machines to reproduce any issues (these > > two instances are also the challenge for the current testbench, what > > requires hopefully fewer than I expect changes ;)). > > Perhaps someone could advise me on an alternative to resolving this > problem by means of bug-hunting. While 4.4.2 is the version of rsyslog > currently shipping in Debian squeeze, it's obviously a bit out-of-date > from the point of view of the developers. > > Well, the reason I'm using that version is precisely that it ships with > Debian. Or to be more precise, I upgraded to squeeze because it ships > with a version of rsyslog that includes encryption support. So perhaps > I > should just cut free, and install 5.x stable? > > -- > Jack. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mrdemeanour at jackpot.uk.net Mon Nov 9 17:24:45 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Mon, 09 Nov 2009 16:24:45 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103395@GRFEXC.intern.adiscon.com> References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA7103395@GRFEXC.intern.adiscon.com> Message-ID: <4AF8424D.30306@jackpot.uk.net> Rainer Gerhards wrote: > Jack, > > Quick update: I was able to run a (relatively quick) test on a > configuration that hopefully later becomes part of the testbench (I > am working towards that goal and thought doing a manual check at that > stage would not hurt). It is not based your config, and it is > relatively simple and straightforward. Still, it uses TLS, and uses > it in anon mode like you did. I used 4.4.2 on the server. I processed > around 500,000 message (not kept track of the actual number). I did > three such runs, all runing under the excellent valgrind[1] memory > debugger. > > For none of the runs, valgrind reported any memory leaks. While this > may not be an ultimate indication, valgrind is *very* effective in > finding leaks and based on what you wrote I would have expected a > small chunk of memory to be lost per message. OK (thanks). I have formed the impression that the problems are occurring after some period of running time; free memory decreases quite slowly, until some unknown event causes a rather rapid degeneration. That is: I suspect that any leak is not likely to be observable on a per-message basis, until this unknown event has occured. > > So the bottom line is that I currently cannot reproduce the bug. This > may change when I finally import your config. However, it would be > useful for me if you could run valgrind on rsyslogd in your > environment and let me know if valgrind reports any memory leaks. > Doing so considerably slows down rsyslogd, but given your load, I'd > expect that it would be acceptable. > > To run under valgrind is relatively simply. Valgrind is available as > a package on almost all distros. All you need to do is run valgrind > and specify your usual rsyslogd command line as the parameter. It is > recommended to do this in the foreground (see rsyslog troubleshooting > doc). > > So, for example, if you start rsyslogd usually by > > /sbin/rsyslogd -c4 -... So "valgrind /usr/sbin/rsyslogd -c4" results in rsyslogd backgrounding promptly, at which point valgrind prints its report (which shows no leaks - unsurprisingly). "valgrind /usr/sbin/rsyslogd -c4 -n" results in a hang. CTRL-C fails to kill the foreground task. "kill -9 " kills the task, but no valgrind report is produced. The same command without valgrind also results in a hung foreground task. If run under valgrind, memcheck-x86-li goes to 99% CPU on CTRL-C. I currently suspect problems with mySQL as the origin of these problems. I was this morning getting messages of the form "Lost connection to MySQL server at 'reading authorization packet'". I was also observing MySQL aborted clients and connects. I have increased the MySQL connect timeout, and can no longer reproduce these reports. For now, I assume that problem is fixed, but I can't yet say if the rsyslog hangs have stopped. I wonder if what was happening was that MySQL was "going away" in some sense, and that rsyslog was not reconnecting to it successfully, *and* not retrying? I noticed that although the ActionQueue for the mysql output module is not disk-assisted, the debug log records: action 4 queue: save on shutdown 1, max disk space allowed 0 So I've set $ActionQueueSaveOnShutdown off. However this hasn't changed the hang behaviour with -n. Since I can't get rsyslog to run in the foreground under valgrind, I am now running daemonized without valgrind (but with encryption); perhaps these changes have fixed the problem. I should know by late this evening - when the problem is observed, the server never lives for more than a few hours. -- Jack. From rgerhards at hq.adiscon.com Tue Nov 10 09:16:32 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 09:16:32 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls (valgrind log) References: <4AF8677B.40808@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033A3@GRFEXC.intern.adiscon.com> Thanks for the log file. Unfortunately, it is not a full run. I gues I needed to make some more things clear. The command line you need to use here $ valgrind /usr/sbin/rsyslogd -c4 -d >rsyslog.log 2>&1 Is the same that you usually use, except that -n must be specified. I unfortunately do no know what you (or better: your distro) usually uses, but you may see it in a process list. Before you start it that way, you must stop the regular syslogd and you must be sure to su to root. The -d switch, while generally useful, takes a lot of time and will generate an immense log. So it is best to not specify it for now. What I am after are valgrind violations (these are printed during the run) and the memory stats at the end of the run. The later will show any memory leaks. The log I received did not receive any message via TLS but two via UDP. It looks like a very short run. Thanks again for your help, I hope I have now explained better ;) Rainer > -----Original Message----- > From: Mr. Demeanour [mailto:mrdemeanour at jackpot.uk.net] > Sent: Monday, November 09, 2009 8:03 PM > To: Rainer Gerhards > Subject: Re: Rsyslog 4.4.2: server out-of-memory with gnutls > (valgrind log) > > Hi, > > I finally got a decent valgrind log using this command: > $ valgrind /usr/sbin/rsyslogd -c4 -d >rsyslog.log 2>&1 > > The log is attached. Free memory was declining steadily > throughout this run. > > Thanks, > -- > Jack. > From rgerhards at hq.adiscon.com Tue Nov 10 09:44:25 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 09:44:25 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103395@GRFEXC.intern.adiscon.com> <4AF8424D.30306@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033A8@GRFEXC.intern.adiscon.com> And finally a reply to "the meat" ;) > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Monday, November 09, 2009 5:25 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Rainer Gerhards wrote: > > Jack, > > > > Quick update: I was able to run a (relatively quick) test on a > > configuration that hopefully later becomes part of the testbench (I > > am working towards that goal and thought doing a manual > check at that > > stage would not hurt). It is not based your config, and it is > > relatively simple and straightforward. Still, it uses TLS, and uses > > it in anon mode like you did. I used 4.4.2 on the server. I > processed > > around 500,000 message (not kept track of the actual number). I did > > three such runs, all runing under the excellent valgrind[1] memory > > debugger. > > > > For none of the runs, valgrind reported any memory leaks. While this > > may not be an ultimate indication, valgrind is *very* effective in > > finding leaks and based on what you wrote I would have expected a > > small chunk of memory to be lost per message. > > OK (thanks). > > I have formed the impression that the problems are occurring > after some > period of running time; free memory decreases quite slowly, until some > unknown event causes a rather rapid degeneration. That is: I suspect > that any leak is not likely to be observable on a per-message basis, > until this unknown event has occured. This would explain what we currently see. It seems to be TLS-related, as you said. So it is unlikely to be a message at all, but rather a TLS event that causes the mem leak. With that said, I should probbably see if a connection abort can trigger one... > > So the bottom line is that I currently cannot reproduce the > bug. This > > may change when I finally import your config. However, it would be > > useful for me if you could run valgrind on rsyslogd in your > > environment and let me know if valgrind reports any memory leaks. > > Doing so considerably slows down rsyslogd, but given your load, I'd > > expect that it would be acceptable. > > > > To run under valgrind is relatively simply. Valgrind is available as > > a package on almost all distros. All you need to do is run valgrind > > and specify your usual rsyslogd command line as the parameter. It is > > recommended to do this in the foreground (see rsyslog > troubleshooting > > doc). > > > > So, for example, if you start rsyslogd usually by > > > > /sbin/rsyslogd -c4 -... > > So "valgrind /usr/sbin/rsyslogd -c4" results in rsyslogd backgrounding > promptly, at which point valgrind prints its report (which shows no > leaks - unsurprisingly). > > "valgrind /usr/sbin/rsyslogd -c4 -n" results in a hang. > CTRL-C fails to > kill the foreground task. That's intended behavior, ctrl-c is only enabled in debug mode (I took this over from sysklogd and never question it". > "kill -9 " kills the task, but no > valgrind report is produced. You must not use -9, which is untrappable. Just send the default SIGTERM: $ kill # no signal specified at all! >The same command without valgrind also > results in a hung foreground task. If run under valgrind, > memcheck-x86-li goes to 99% CPU on CTRL-C. > > I currently suspect problems with mySQL as the origin of > these problems. > I was this morning getting messages of the form > "Lost connection to MySQL server at 'reading authorization packet'". > I was also observing MySQL aborted clients and connects. I have > increased the MySQL connect timeout, and can no longer reproduce these > reports. For now, I assume that problem is fixed, but I can't > yet say if > the rsyslog hangs have stopped. > > I wonder if what was happening was that MySQL was "going away" in some > sense, and that rsyslog was not reconnecting to it successfully, *and* > not retrying? Hard to guess... > > I noticed that although the ActionQueue for the mysql output module is > not disk-assisted, the debug log records: > action 4 queue: save on shutdown 1, max disk space allowed 0 The debug output just spits out the values of the variables. Saveonshutdown is true by default, but if there is no disk queue, that won't help anything at all ;) In short: that's OK. > So I've set $ActionQueueSaveOnShutdown off. However this > hasn't changed > the hang behaviour with -n. > > Since I can't get rsyslog to run in the foreground under > valgrind, I am > now running daemonized without valgrind (but with encryption); perhaps > these changes have fixed the problem. I should know by late > this evening > - when the problem is observed, the server never lives for more than a > few hours. I hope this -and the other- mail will help straighten out the issue. Any logs you can send to my private mail address as you have already done ;) Rainer > -- > Jack. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 10 10:28:55 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 10:28:55 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA710335E@GRFEXC.intern.adiscon.com> <4AF3002B.9060302@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103360@GRFEXC.intern.adiscon.com> <4AF6FE93.6080805@jackpot.uk.net><9B6E2A8877C38245BFB15CC491A11DA7103395@GRFEXC.intern.adiscon.com><4AF8424D.30306@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033A8@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033AB@GRFEXC.intern.adiscon.com> > This would explain what we currently see. It seems to be TLS-related, > as you > said. So it is unlikely to be a message at all, but rather a TLS event > that > causes the mem leak. With that said, I should probbably see if a > connection > abort can trigger one... Just for the records, the connection aborts I triggered did NOT result in leaked memory :( Rainer From rgerhards at hq.adiscon.com Tue Nov 10 10:53:16 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 10:53:16 +0100 Subject: [rsyslog] 4.4.2 leaks: another log References: <4AF93722.8040101@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com> excellent! I also see some violations, what is a good thing. But I notice I unfortunately missed one thing: by default, the violations do not include the loadbale module addresses. Would it be possible that you create a custom build of rsyslog with the "--enable-valgrind" configure option? That does not cause any overhead, but permits to see the function names on exit (and thus is really useful). Thanks, Rainer > -----Original Message----- > From: Mr. Demeanour [mailto:mrdemeanour at jackpot.uk.net] > Sent: Tuesday, November 10, 2009 10:49 AM > To: Rainer Gerhards > Subject: 4.4.2 leaks: another log > > Hi. > > Thanks for all your help. I didn't realise that -n suppressed SIGKILL. > I > also didn't realise that it was taking up to five minutes with -n and > valgrind, between the successful opening of a UDP listener on 514 and a > TCP listener appearing on 10514! That (and my misunderstanding of > signal > handling) is why I thought it had hung. > > So perhaps something is strange about my certificate. I'll see if I can > test it somehow. > > Anyway, here is a log apparently showing leakage; it represents about > 400 TCP messages over 30 minutes. The command was: > valgrind --leak-check=full /usr/sbin/rsyslogd -c4 -n >rsyslog.log 2>&1 > > By the time I killed it, memory usage had climbed from about 70M to > 100M. With a non-encrypting rsyslog (and without valgrind), usage is > stable at around 70M. Here's a "free" while running a non-encrypting > service: > > # free > total used free shared buffers > cached > Mem: 774972 732564 42408 0 22544 > 641436 > -/+ buffers/cache: 68584 706388 > Swap: 1951856 5640 1946216 > > The picture remains like that more-or-less indefinitely. > > Just before I killed the valgrind run corresponding to that log, it > looked like this: > # free > total used free shared buffers > cached > Mem: 774972 766292 8680 0 22016 > 643940 > -/+ buffers/cache: 100336 674636 > Swap: 1951856 5640 1946216 > > > > Now I know how to do this, I should be able to do it on demand. Thanks > again. > > -- > Jack. From rgerhards at hq.adiscon.com Tue Nov 10 11:09:41 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 11:09:41 +0100 Subject: [rsyslog] 4.4.2 leaks: another log References: <4AF93722.8040101@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> It just occurred to me that you may be seeing an anomaly in clib's malloc subsystem. I fought with this in early summer, but somehow have not mentioned it in the change log. The change itself was done on June, 22nd 2009. I have just checked, 4.4.2 does not have this code. I noticed that while there were some reports in the valgrind log, they were not large enough to explain what you saw. The libc malloc issue was discussed on the mailing list, and this here is a link to a later wrap-up type of message: http://lists.adiscon.net/pipermail/rsyslog/2009-June/002372.html So my suggestion is to move to v4-stable and see if the problem persists there (obviously, contrary to what I said, v5 would have fixed it in that case as well - so never trust a dev... ;)). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, November 10, 2009 10:53 AM > To: rsyslog-users > Subject: Re: [rsyslog] 4.4.2 leaks: another log > > excellent! I also see some violations, what is a good thing. But I > notice I > unfortunately missed one thing: by default, the violations do not > include the > loadbale module addresses. Would it be possible that you create a > custom > build of rsyslog with the "--enable-valgrind" configure option? That > does not > cause any overhead, but permits to see the function names on exit (and > thus > is really useful). > > Thanks, > Rainer > > > -----Original Message----- > > From: Mr. Demeanour [mailto:mrdemeanour at jackpot.uk.net] > > Sent: Tuesday, November 10, 2009 10:49 AM > > To: Rainer Gerhards > > Subject: 4.4.2 leaks: another log > > > > Hi. > > > > Thanks for all your help. I didn't realise that -n suppressed > SIGKILL. > > I > > also didn't realise that it was taking up to five minutes with -n and > > valgrind, between the successful opening of a UDP listener on 514 and > a > > TCP listener appearing on 10514! That (and my misunderstanding of > > signal > > handling) is why I thought it had hung. > > > > So perhaps something is strange about my certificate. I'll see if I > can > > test it somehow. > > > > Anyway, here is a log apparently showing leakage; it represents about > > 400 TCP messages over 30 minutes. The command was: > > valgrind --leak-check=full /usr/sbin/rsyslogd -c4 -n >rsyslog.log > 2>&1 > > > > By the time I killed it, memory usage had climbed from about 70M to > > 100M. With a non-encrypting rsyslog (and without valgrind), usage is > > stable at around 70M. Here's a "free" while running a non-encrypting > > service: > > > > # free > > total used free shared buffers > > cached > > Mem: 774972 732564 42408 0 22544 > > 641436 > > -/+ buffers/cache: 68584 706388 > > Swap: 1951856 5640 1946216 > > > > The picture remains like that more-or-less indefinitely. > > > > Just before I killed the valgrind run corresponding to that log, it > > looked like this: > > # free > > total used free shared buffers > > cached > > Mem: 774972 766292 8680 0 22016 > > 643940 > > -/+ buffers/cache: 100336 674636 > > Swap: 1951856 5640 1946216 > > > > > > > > Now I know how to do this, I should be able to do it on demand. > Thanks > > again. > > > > -- > > Jack. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mrdemeanour at jackpot.uk.net Tue Nov 10 11:29:41 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Tue, 10 Nov 2009 10:29:41 +0000 Subject: [rsyslog] 4.4.2 leaks: another log In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> References: <4AF93722.8040101@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> Message-ID: <4AF94095.2040205@jackpot.uk.net> Rainer Gerhards wrote: > It just occurred to me that you may be seeing an anomaly in clib's > malloc subsystem. I fought with this in early summer, but somehow > have not mentioned it in the change log. The change itself was done > on June, 22nd 2009. I have just checked, 4.4.2 does not have this > code. I noticed that while there were some reports in the valgrind > log, they were not large enough to explain what you saw. > > The libc malloc issue was discussed on the mailing list, and this > here is a link to a later wrap-up type of message: > > http://lists.adiscon.net/pipermail/rsyslog/2009-June/002372.html > > So my suggestion is to move to v4-stable and see if the problem > persists there (obviously, contrary to what I said, v5 would have > fixed it in that case as well - so never trust a dev... ;)). OK. I have to be somewhere else in half an hour, so I'll return to this later. -- Jack. From rgerhards at hq.adiscon.com Tue Nov 10 11:30:59 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 11:30:59 +0100 Subject: [rsyslog] 4.4.2 leaks: another log References: <4AF93722.8040101@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> <4AF94095.2040205@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033AE@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Tuesday, November 10, 2009 11:30 AM > To: rsyslog-users > Subject: Re: [rsyslog] 4.4.2 leaks: another log > > Rainer Gerhards wrote: > > It just occurred to me that you may be seeing an anomaly in clib's > > malloc subsystem. I fought with this in early summer, but somehow > > have not mentioned it in the change log. The change itself was done > > on June, 22nd 2009. I have just checked, 4.4.2 does not have this > > code. I noticed that while there were some reports in the valgrind > > log, they were not large enough to explain what you saw. > > > > The libc malloc issue was discussed on the mailing list, and this > > here is a link to a later wrap-up type of message: > > > > http://lists.adiscon.net/pipermail/rsyslog/2009-June/002372.html > > > > So my suggestion is to move to v4-stable and see if the problem > > persists there (obviously, contrary to what I said, v5 would have > > fixed it in that case as well - so never trust a dev... ;)). args, important type: suggest to move to v4-beta NOT -stable... > > OK. > > I have to be somewhere else in half an hour, so I'll return to this > later. > > -- > Jack. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 10 14:03:00 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 10 Nov 2009 14:03:00 +0100 Subject: [rsyslog] On-Demand Debugging Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033B1@GRFEXC.intern.adiscon.com> Hi all, I worked today on the capability to obtain debug information from a running instance. Unfortunately, a small code change was needed to make it work well. If you cannot upgrade or apply the patch, I can create instructions to do it even without the patch, but that is a rather inconvenient way of doing thigs. The patch is here, it should apply to all recent v4 and v5 builds: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=2b5e15d549940c7ace9017ea f40d523e3741c9a2 I have also updated the documentation of the debug system and hope that it is much more usable this time. It specifically explains how to work with this new type of on-demand debugging. Details are online: http://www.rsyslog.com/doc-debug.html If that capability is useful, I could extend it in some ways. Most importantly, the current version does NOT permit to send debug logs to another remote instance. However, I tried to do as few changes as possible in an effort to help our current debugging efforts. Any new functionality will probably be v5-exclusive. I hope this is a useful new feature. So far, it is available via the git heads only, but I'll see that I release as soon as it makes sense (I don't like to release just for this mini-feature...). Rainer From szybalski at gmail.com Wed Nov 11 02:02:47 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Tue, 10 Nov 2009 19:02:47 -0600 Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> Message-ID: <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> Hello, I have a web application deployed in multiprocess and multi-thread scenario. ( 3 processes and 10 threads each). I want to log search query string from users to a file ?called todaysdate.log ...20091109.log I'm using a python app but I can't get rsyslog in debian system to catch and write my messages. Would you know how can I set this up? Here is a sample test case....in python. Save below to a file and run it. python testfile.py #----testfile.py----- from logging.handlers import SysLogHandler import logging # create logger logger = logging.getLogger("myapp") logger.setLevel(logging.DEBUG) ch = SysLogHandler('/dev/log') ch.setLevel(logging.DEBUG) # create formatter formatter = logging.Formatter("%(asctime)s - %(name)s-%(levelname)s - %(message)s") # add formatter to ch ch.setFormatter(formatter) logger.addHandler(ch) logger.info('This is a message') How can I setup rsyslog to filter "myapp" and save the messages to a file in /home/lucas/myapp/20091109.txt Thanks, Lucas From mrdemeanour at jackpot.uk.net Wed Nov 11 18:38:21 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Wed, 11 Nov 2009 17:38:21 +0000 Subject: [rsyslog] 4.4.2 leaks: another log In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71033AE@GRFEXC.intern.adiscon.com> References: <4AF93722.8040101@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> <4AF94095.2040205@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AE@GRFEXC.intern.adiscon.com> Message-ID: <4AFAF68D.9060203@jackpot.uk.net> Rainer Gerhards wrote: >>> >>> So my suggestion is to move to v4-stable and see if the problem >>> persists there (obviously, contrary to what I said, v5 would have >>> fixed it in that case as well - so never trust a dev... ;)). > > > args, important type: suggest to move to v4-beta NOT -stable... OK, so it looks like I've still got the problem, with v4 beta (4.5.6). I'll have another go with valgrind tomorrow. -- Jack. From szybalski at gmail.com Thu Nov 12 15:19:26 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Thu, 12 Nov 2009 08:19:26 -0600 Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> Message-ID: <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> Anybody know what the filter or a config line would be that would filter my "myapp" messages to a file in /home/lucas/20091112.txt? Ideas? Thanks, Lucas On Tue, Nov 10, 2009 at 7:02 PM, Lukasz Szybalski wrote: > Hello, > I have a web application deployed in multiprocess and multi-thread > scenario. ( 3 processes and 10 threads each). > > I want to log search query string from users to a file ?called > todaysdate.log ...20091109.log > > > I'm using a python app but I can't get rsyslog in debian system to > catch and write my messages. Would you know how can I set this up? > > > Here is a sample test case....in python. > Save below to a file and run it. > python testfile.py > > > #----testfile.py----- > from logging.handlers import SysLogHandler > > import logging > > > # create logger > logger = logging.getLogger("myapp") > logger.setLevel(logging.DEBUG) > ch = SysLogHandler('/dev/log') > ch.setLevel(logging.DEBUG) > # create formatter > formatter = logging.Formatter("%(asctime)s - %(name)s-%(levelname)s - > %(message)s") > # add formatter to ch > ch.setFormatter(formatter) > logger.addHandler(ch) > logger.info('This is a message') > > > How can I setup rsyslog to filter "myapp" and save the messages to a > file in /home/lucas/myapp/20091109.txt > > Thanks, > Lucas > -- Setup CalendarServer for your company. http://lucasmanual.com/mywiki/CalendarServer Automotive Recall Database - See if you vehicle has a recall http://lucasmanual.com/recall From rgerhards at hq.adiscon.com Thu Nov 12 15:23:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 12 Nov 2009 15:23:44 +0100 Subject: [rsyslog] multiprocess/multithread web app to rsyslog References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com><804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> If "myapp" is the tag, you can use this: :syslogtag, contains, "maypp" /home/lucas/20091112.txt There are better comparisons than "contains", but I don't know them out of my head. They are in the filter doc. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Lukasz Szybalski > Sent: Thursday, November 12, 2009 3:19 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] multiprocess/multithread web app to rsyslog > > Anybody know what the filter or a config line would be that would > filter my "myapp" messages to a file in /home/lucas/20091112.txt? > > Ideas? > Thanks, > Lucas > > > On Tue, Nov 10, 2009 at 7:02 PM, Lukasz Szybalski > wrote: > > Hello, > > I have a web application deployed in multiprocess and multi-thread > > scenario. ( 3 processes and 10 threads each). > > > > I want to log search query string from users to a file ?called > > todaysdate.log ...20091109.log > > > > > > I'm using a python app but I can't get rsyslog in debian system to > > catch and write my messages. Would you know how can I set this up? > > > > > > Here is a sample test case....in python. > > Save below to a file and run it. > > python testfile.py > > > > > > #----testfile.py----- > > from logging.handlers import SysLogHandler > > > > import logging > > > > > > # create logger > > logger = logging.getLogger("myapp") > > logger.setLevel(logging.DEBUG) > > ch = SysLogHandler('/dev/log') > > ch.setLevel(logging.DEBUG) > > # create formatter > > formatter = logging.Formatter("%(asctime)s - %(name)s-%(levelname)s - > > %(message)s") > > # add formatter to ch > > ch.setFormatter(formatter) > > logger.addHandler(ch) > > logger.info('This is a message') > > > > > > How can I setup rsyslog to filter "myapp" and save the messages to a > > file in /home/lucas/myapp/20091109.txt > > > > Thanks, > > Lucas > > > > > > -- > Setup CalendarServer for your company. > http://lucasmanual.com/mywiki/CalendarServer > Automotive Recall Database - See if you vehicle has a recall > http://lucasmanual.com/recall > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From szybalski at gmail.com Thu Nov 12 15:59:59 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Thu, 12 Nov 2009 08:59:59 -0600 Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> Message-ID: <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> On Thu, Nov 12, 2009 at 8:23 AM, Rainer Gerhards wrote: > If "myapp" is the tag, you can use this: > > :syslogtag, contains, "maypp" /home/lucas/20091112.txt > > There are better comparisons than "contains", but I don't know them out of my > head. They are in the filter doc. > Getting closer. I put a file in /etc/rsyslog.d/myapp.conf Inside I have: :syslogtag, contains, "myapp" /home/lucas/20091112.txt the rsyslog creates the file but no messages are logged in, they only appear in /var/log/messages Nov 12 08:45:32 localhost 2009-11-12 08:45:32,452 - myapp - INFO - This is a message Ideas why its not logging it in? Also, Can I make this config file like this? It doesn't seem to work...? :syslogtag, contains, "myapp" /home/lucas/$year$month$day.txt http://www.rsyslog.com/doc-property_replacer.html Thanks, Lucas > HTH > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Lukasz Szybalski >> Sent: Thursday, November 12, 2009 3:19 PM >> To: rsyslog at lists.adiscon.com >> Subject: Re: [rsyslog] multiprocess/multithread web app to rsyslog >> >> Anybody know what the filter or a config line would be that would >> filter my "myapp" messages to a file in /home/lucas/20091112.txt? >> >> Ideas? >> Thanks, >> Lucas >> >> >> On Tue, Nov 10, 2009 at 7:02 PM, Lukasz Szybalski >> wrote: >> > Hello, >> > I have a web application deployed in multiprocess and multi-thread >> > scenario. ( 3 processes and 10 threads each). >> > >> > I want to log search query string from users to a file ?called >> > todaysdate.log ...20091109.log >> > >> > >> > I'm using a python app but I can't get rsyslog in debian system to >> > catch and write my messages. Would you know how can I set this up? >> > >> > >> > Here is a sample test case....in python. >> > Save below to a file and run it. >> > python testfile.py >> > >> > >> > #----testfile.py----- >> > from logging.handlers import SysLogHandler >> > >> > import logging >> > >> > >> > # create logger >> > logger = logging.getLogger("myapp") >> > logger.setLevel(logging.DEBUG) >> > ch = SysLogHandler('/dev/log') >> > ch.setLevel(logging.DEBUG) >> > # create formatter >> > formatter = logging.Formatter("%(asctime)s - %(name)s-%(levelname)s - >> > %(message)s") >> > # add formatter to ch >> > ch.setFormatter(formatter) >> > logger.addHandler(ch) >> > logger.info('This is a message') >> > >> > >> > How can I setup rsyslog to filter "myapp" and save the messages to a >> > file in /home/lucas/myapp/20091109.txt >> > >> > Thanks, >> > Lucas >> > >> >> From rgerhards at hq.adiscon.com Thu Nov 12 16:33:34 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 12 Nov 2009 16:33:34 +0100 Subject: [rsyslog] multiprocess/multithread web app to rsyslog References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com><804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com><804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Lukasz Szybalski > Sent: Thursday, November 12, 2009 4:00 PM > To: rsyslog-users > Subject: Re: [rsyslog] multiprocess/multithread web app to rsyslog > > On Thu, Nov 12, 2009 at 8:23 AM, Rainer Gerhards > wrote: > > If "myapp" is the tag, you can use this: > > > > :syslogtag, contains, "maypp" /home/lucas/20091112.txt > > > > There are better comparisons than "contains", but I don't know them > out of my > > head. They are in the filter doc. > > > > Getting closer. > I put a file in /etc/rsyslog.d/myapp.conf > > Inside I have: > :syslogtag, contains, "myapp" /home/lucas/20091112.txt > > the rsyslog creates the file but no messages are logged in, they only > appear in /var/log/messages > > Nov 12 08:45:32 localhost 2009-11-12 08:45:32,452 - myapp - INFO - aha! this message is not really well-formed, please also see http://www.rsyslog.com/doc-syslog_parsing.html So the tag here is "2009-11-12". You can either use a custom parser, or if that would be sufficent, check msg instead of syslogtag. Rainer > This is a message > > > Ideas why its not logging it in? > > Also, > Can I make this config file like this? It doesn't seem to work...? > :syslogtag, contains, "myapp" /home/lucas/$year$month$day.txt > > > http://www.rsyslog.com/doc-property_replacer.html > > Thanks, > Lucas > > > > > > HTH > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Lukasz Szybalski > >> Sent: Thursday, November 12, 2009 3:19 PM > >> To: rsyslog at lists.adiscon.com > >> Subject: Re: [rsyslog] multiprocess/multithread web app to rsyslog > >> > >> Anybody know what the filter or a config line would be that would > >> filter my "myapp" messages to a file in /home/lucas/20091112.txt? > >> > >> Ideas? > >> Thanks, > >> Lucas > >> > >> > >> On Tue, Nov 10, 2009 at 7:02 PM, Lukasz Szybalski > > >> wrote: > >> > Hello, > >> > I have a web application deployed in multiprocess and multi-thread > >> > scenario. ( 3 processes and 10 threads each). > >> > > >> > I want to log search query string from users to a file ?called > >> > todaysdate.log ...20091109.log > >> > > >> > > >> > I'm using a python app but I can't get rsyslog in debian system to > >> > catch and write my messages. Would you know how can I set this up? > >> > > >> > > >> > Here is a sample test case....in python. > >> > Save below to a file and run it. > >> > python testfile.py > >> > > >> > > >> > #----testfile.py----- > >> > from logging.handlers import SysLogHandler > >> > > >> > import logging > >> > > >> > > >> > # create logger > >> > logger = logging.getLogger("myapp") > >> > logger.setLevel(logging.DEBUG) > >> > ch = SysLogHandler('/dev/log') > >> > ch.setLevel(logging.DEBUG) > >> > # create formatter > >> > formatter = logging.Formatter("%(asctime)s - %(name)s- > %(levelname)s - > >> > %(message)s") > >> > # add formatter to ch > >> > ch.setFormatter(formatter) > >> > logger.addHandler(ch) > >> > logger.info('This is a message') > >> > > >> > > >> > How can I setup rsyslog to filter "myapp" and save the messages to > a > >> > file in /home/lucas/myapp/20091109.txt > >> > > >> > Thanks, > >> > Lucas > >> > > >> > >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From szybalski at gmail.com Thu Nov 12 18:18:02 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Thu, 12 Nov 2009 11:18:02 -0600 Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> Message-ID: <804e5c70911120918s935f756j602d1c0ce7f0798b@mail.gmail.com> >> >> > >> >> > >> >> > Here is a sample test case....in python. >> >> > Save below to a file and run it. >> >> > python testfile.py >> >> > >> >> > >> >> > #----testfile.py----- >> >> > from logging.handlers import SysLogHandler >> >> > >> >> > import logging >> >> > >> >> > >> >> > # create logger >> >> > logger = logging.getLogger("myapp") >> >> > logger.setLevel(logging.DEBUG) >> >> > ch = SysLogHandler('/dev/log') >> >> > ch.setLevel(logging.DEBUG) >> >> > # create formatter >> >> > formatter = logging.Formatter("%(asctime)s - %(name)s- >> %(levelname)s - >> >> > %(message)s") >> >> > # add formatter to ch >> >> > ch.setFormatter(formatter) >> >> > logger.addHandler(ch) >> >> > logger.info('This is a message') >> >> > >> >> > >> >> > How can I setup rsyslog to filter "myapp" and save the messages to 1. So in my code I've defined a formatter to be: formatter = logging.Formatter("%(asctime)s - %(name)s- %(levelname)s - %(message)s") I don't care how it looks as long as it has a date, time, name of the app, level, and message. Is there a standard formatting I should be using? I don't want to create a custom solution if I can use standard one? 2. What about saving the file with todays date. I don't want to rotate the logs, but I want to have a file for each day saved in /home/lucas/$year$month$date.txt Above does not work, unless I'm missing some escape characters/parentheses? Thanks, Lucas From david at lang.hm Thu Nov 12 20:25:40 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 12 Nov 2009 11:25:40 -0800 (PST) Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <804e5c70911120918s935f756j602d1c0ce7f0798b@mail.gmail.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> <804e5c70911120918s935f756j602d1c0ce7f0798b@mail.gmail.com> Message-ID: On Thu, 12 Nov 2009, Lukasz Szybalski wrote: >>>>>> >>>>>> >>>>>> Here is a sample test case....in python. >>>>>> Save below to a file and run it. >>>>>> python testfile.py >>>>>> >>>>>> >>>>>> #----testfile.py----- >>>>>> from logging.handlers import SysLogHandler >>>>>> >>>>>> import logging >>>>>> >>>>>> >>>>>> # create logger >>>>>> logger = logging.getLogger("myapp") >>>>>> logger.setLevel(logging.DEBUG) >>>>>> ch = SysLogHandler('/dev/log') >>>>>> ch.setLevel(logging.DEBUG) >>>>>> # create formatter >>>>>> formatter = logging.Formatter("%(asctime)s - %(name)s- >>> %(levelname)s - >>>>>> %(message)s") >>>>>> # add formatter to ch >>>>>> ch.setFormatter(formatter) >>>>>> logger.addHandler(ch) >>>>>> logger.info('This is a message') >>>>>> >>>>>> >>>>>> How can I setup rsyslog to filter "myapp" and save the messages to > > 1. > So in my code I've defined a formatter to be: > formatter = logging.Formatter("%(asctime)s - %(name)s- %(levelname)s - > %(message)s") > > I don't care how it looks as long as it has a date, time, name of the > app, level, and message. Is there a standard formatting I should be > using? I don't want to create a custom solution if I can use standard > one? the format over the wire should be date server tag message if the message does not have this format, rsyslog will add a priority, date and server to the message in addition, the date you are setting is not formatted appropriately the date should be MMM DD HH:MM:SS I am not familiar enough with the logger package in python to tell you how to format it this way. however, based on the message that you sent earlier, if you set your format to be formatter = logging.Formatter("%(name)s %(message)s") it may be close enough to the correct thing to work. > 2. What about saving the file with todays date. I don't want to rotate > the logs, but I want to have a file for each day saved in > /home/lucas/$year$month$date.txt > Above does not work, unless I'm missing some escape characters/parentheses? look in the section on dynafiles. I have not used this feature, so I can't help directly. David Lang From szybalski at gmail.com Thu Nov 12 23:32:43 2009 From: szybalski at gmail.com (Lukasz Szybalski) Date: Thu, 12 Nov 2009 16:32:43 -0600 Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> <804e5c70911120918s935f756j602d1c0ce7f0798b@mail.gmail.com> Message-ID: <804e5c70911121432y404d0a00h2f080590140addf@mail.gmail.com> On Thu, Nov 12, 2009 at 1:25 PM, wrote: > On Thu, 12 Nov 2009, Lukasz Szybalski wrote: > >>>>>>> >>>>>>> >>>>>>> Here is a sample test case....in python. >>>>>>> Save below to a file and run it. >>>>>>> python testfile.py >>>>>>> >>>>>>> >>>>>>> #----testfile.py----- >>>>>>> from logging.handlers import SysLogHandler >>>>>>> >>>>>>> import logging >>>>>>> >>>>>>> >>>>>>> # create logger >>>>>>> logger = logging.getLogger("myapp") >>>>>>> logger.setLevel(logging.DEBUG) >>>>>>> ch = SysLogHandler('/dev/log') >>>>>>> ch.setLevel(logging.DEBUG) >>>>>>> # create formatter >>>>>>> formatter = logging.Formatter("%(asctime)s - %(name)s- >>>> %(levelname)s - >>>>>>> %(message)s") >>>>>>> # add formatter to ch >>>>>>> ch.setFormatter(formatter) >>>>>>> logger.addHandler(ch) >>>>>>> logger.info('This is a message') >>>>>>> >>>>>>> >>>>>>> How can I setup rsyslog to filter "myapp" and save the messages to >> >> 1. >> So in my code I've defined a formatter to be: >> formatter = logging.Formatter("%(asctime)s - %(name)s- %(levelname)s - >> %(message)s") >> >> I don't care how it looks as long as it has a date, time, name of the >> app, level, and message. Is there a standard formatting I should be >> using? I don't want to create a custom solution if I can use standard >> one? > > the format over the wire should be > > date server tag message > > if the message does not have this format, rsyslog will add a priority, > date and server to the message > > in addition, the date you are setting is not formatted appropriately > > the date should be > > MMM DD HH:MM:SS > > I am not familiar enough with the logger package in python to tell you how > to format it this way. > > however, based on the message that you sent earlier, if you set your > format to be > > formatter = logging.Formatter("%(name)s %(message)s") This works.... nice... > > it may be close enough to the correct thing to work. > >> 2. What about saving the file with todays date. I don't want to rotate >> the logs, but I want to have a file for each day saved in >> /home/lucas/$year$month$date.txt >> Above does not work, unless I'm missing some escape characters/parentheses? > > look in the section on dynafiles. I have not used this feature, so I can't > help directly. Now I just have to find the proper syntax to get the file to save to :syslogtag, contains, "myapp" /home/lucas/20091112.txt to /home/lucas/$year$month$date.txt Lucas From david at lang.hm Fri Nov 13 00:02:08 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 12 Nov 2009 15:02:08 -0800 (PST) Subject: [rsyslog] multiprocess/multithread web app to rsyslog In-Reply-To: <804e5c70911121432y404d0a00h2f080590140addf@mail.gmail.com> References: <804e5c70911091727s2d1e0f1fy289efa22b3f04d4c@mail.gmail.com> <804e5c70911101702l7ef7a796n849a346ad9152a5a@mail.gmail.com> <804e5c70911120619m42a4de0i2f260aa0a4fed336@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033D8@GRFEXC.intern.adiscon.com> <804e5c70911120659m1f61ba07v10cb72831cecdf34@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71033DA@GRFEXC.intern.adiscon.com> <804e5c70911120918s935f756j602d1c0ce7f0798b@mail.gmail.com> <804e5c70911121432y404d0a00h2f080590140addf@mail.gmail.com> Message-ID: On Thu, 12 Nov 2009, Lukasz Szybalski wrote: > On Thu, Nov 12, 2009 at 1:25 PM, wrote: >> On Thu, 12 Nov 2009, Lukasz Szybalski wrote: >> >> >> it may be close enough to the correct thing to work. >> >>> 2. What about saving the file with todays date. I don't want to rotate >>> the logs, but I want to have a file for each day saved in >>> /home/lucas/$year$month$date.txt >>> Above does not work, unless I'm missing some escape characters/parentheses? >> >> look in the section on dynafiles. I have not used this feature, so I can't >> help directly. > > Now I just have to find the proper syntax to get the file to save to > :syslogtag, contains, "myapp" /home/lucas/20091112.txt > to > /home/lucas/$year$month$date.txt look at DynFile in http://www.rsyslog.com/doc-rsyslog_conf_actions.html David Lang From tbergfeld at hq.adiscon.com Fri Nov 13 12:51:09 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 13 Nov 2009 12:51:09 +0100 Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com> Hi all, Today, we released a new v5-beta branch. It bases on the last v5-devel, plus has some bug fixes and some enhancements. The enhancements provide a slightly increased performance. This release is a very important milestone. Based on feedback from the v5-devel branches, it is in pretty good shape. Our aim is to have a quick beta phase this time (taking the problems with the current v5-stable) into account. To do so, we need your cooperation. Please try out this new version and report back any issues you may experience. It would also be good to know if you use it and do not run into any issues. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article427.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-187.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From david at lang.hm Sat Nov 14 03:48:06 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 13 Nov 2009 18:48:06 -0800 (PST) Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com> Message-ID: I've got this installed in production and it's working well so far. monday morning (pacific time) should give it a good stress test. David Lang On Fri, 13 Nov 2009, Tom Bergfeld wrote: > Date: Fri, 13 Nov 2009 12:51:09 +0100 > From: Tom Bergfeld > Reply-To: rsyslog-users > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released > > Hi all, > > Today, we released a new v5-beta branch. It bases on the last v5-devel, plus > has some bug fixes and some enhancements. The enhancements provide a slightly > increased performance. This release is a very important milestone. Based on > feedback from the v5-devel branches, it is in pretty good shape. Our aim is > to have a quick beta phase this time (taking the problems with the current > v5-stable) into account. To do so, we need your cooperation. Please try out > this new version and report back any issues you may experience. It would also > be good to know if you use it and do not run into any issues. > > See Changelog for more details. > > ChangeLog: > > http://www.rsyslog.com/Article427.phtml > > Download: > > http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-187.phtml > > > As always, feedback is appreciated. > > Best regards, > Tom Bergfeld > -- > Support > ======= > > Improving rsyslog is costly, but you can help! We are looking for > organizations that find rsyslog useful and wish to contribute back. You can > contribute by reporting bugs, improve the software, or donate money or > equipment. > > Commercial support contracts for rsyslog are available, and they help finance > continued maintenance. Adiscon GmbH, a privately held German company, is > currently funding rsyslog development. We are always looking for interesting > development projects. For details on how to help, please see > http://www.rsyslog.com/doc-how2help.html . > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mrdemeanour at jackpot.uk.net Sat Nov 14 14:59:43 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Sat, 14 Nov 2009 13:59:43 +0000 Subject: [rsyslog] 4.4.2 leaks: another log In-Reply-To: <4AFAF68D.9060203@jackpot.uk.net> References: <4AF93722.8040101@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AC@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71033AD@GRFEXC.intern.adiscon.com> <4AF94095.2040205@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71033AE@GRFEXC.intern.adiscon.com> <4AFAF68D.9060203@jackpot.uk.net> Message-ID: <4AFEB7CF.3060309@jackpot.uk.net> Mr. Demeanour wrote: > Rainer Gerhards wrote: >>>> So my suggestion is to move to v4-stable and see if the problem >>>> persists there (obviously, contrary to what I said, v5 would >>>> have fixed it in that case as well - so never trust a dev... >>>> ;)). >> >> args, important type: suggest to move to v4-beta NOT -stable... > > OK, so it looks like I've still got the problem, with v4 beta > (4.5.6). I'll have another go with valgrind tomorrow. I have been running a setup with 4.5.6 collecting messages by GnuTLS only, and logging to a file; and the problem has failed to materialise. Perhaps I didn't run it for long enough; but a few hours is normally sufficient for serious memory depletion to occur, and it hasn't so far. The other variables are that only one (LAN) client is logging to this instance - all previous runs have involved multiple clients, some logging via the internet; and that only one input and one output method is configured. Previous tests have had both imudp and imtcp configured, and have used both omfile and ommysql. I've now changed this from logging to a file to log via mySQL. If that remains stable after several hours, I'll reconfigure with dual input and output methods; then with multiple clients. Eventually I should be able to describe a method of replication. -- Jack. From rgerhards at hq.adiscon.com Mon Nov 16 15:00:20 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 16 Nov 2009 15:00:20 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> Hi all, I am working again on moving the DNS name resolution outside of the input thread of those sources where this is potentially time-consuming and affecting message acceptance rates. As it turned out, currently imudp seems to be the only case. While this is potentially easy to do, a problem is ACLs ($AllowedSender) which use system names rather than ip addresses. In order to check these ACLs, we need to do a DNS lookup. Especially in the case of UDP, such a lookup may actually case message loss and thus may be abused by an attacker to cause a certain degree of denial of service (what also points out that these types of ACLs are not really a good idea, even though requested by practice). In the light of this, I will now do something that sounds strange at first: I will always accept messages that require DNS lookups and enqueue these into the main queue and do the name resolution AND the final name-based ACL check only on the queue consumer part. Please note that it will be done BEFORE message content is parsed, so there is no chance that buffer overlow attacks can be carried out from non-authenticated hosts. The core idea is to move the lengthy, potentially message-loss causing code, away from the input thread. The only questionable effect I can currently see is that queue space is potentially taken up by messages which will immediately be discarded and should not be there in the first place. At the extreme end, that could lead to loss of valid messages. But on the other hand valid messages are more likely to be lost by the DNS name query overhead if I do the ACL check directly in the input thread. As such, I think my intended move is correct. Does anyone have an argument against the approach I am now taking? Thanks, Rainer From theinric at redhat.com Mon Nov 16 17:24:30 2009 From: theinric at redhat.com (Tomas Heinrich) Date: Mon, 16 Nov 2009 17:24:30 +0100 Subject: [rsyslog] support for arbitrary number of open files Message-ID: <4B017CBE.80904@redhat.com> Hi all, currently the total number of files (and tcp connections) that can be open simultaneously is limited by the select() system call. Ideally this would be changed to *poll(), but that can take some time. Until that happens, this patch[1] tries to remove the limitations of select() by enlarging the bit mask that is used for storing file descriptor information and redefining the macros that process it. This modification is inspired by Bind's use of select(). It is rather a workaround and may not be entirely portable. The actual changes to the code aren't big, but they are in many places so sufficient testing is needed. Allocating and freeing fd_sets in some frequently called functions may decrease performance. This can be dealt with but would require more pervasive changes. Thoughts? Thanks, Tomas [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited-select.patch From david at lang.hm Mon Nov 16 17:51:40 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 16 Nov 2009 08:51:40 -0800 (PST) Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 16 Nov 2009, Rainer Gerhards wrote: > Hi all, > > I am working again on moving the DNS name resolution outside of the input > thread of those sources where this is potentially time-consuming and > affecting message acceptance rates. As it turned out, currently imudp seems > to be the only case. > > While this is potentially easy to do, a problem is ACLs ($AllowedSender) > which use system names rather than ip addresses. In order to check these > ACLs, we need to do a DNS lookup. Especially in the case of UDP, such a > lookup may actually case message loss and thus may be abused by an attacker > to cause a certain degree of denial of service (what also points out that > these types of ACLs are not really a good idea, even though requested by > practice). > > In the light of this, I will now do something that sounds strange at first: I > will always accept messages that require DNS lookups and enqueue these into > the main queue and do the name resolution AND the final name-based ACL check > only on the queue consumer part. Please note that it will be done BEFORE > message content is parsed, so there is no chance that buffer overlow attacks > can be carried out from non-authenticated hosts. The core idea is to move the > lengthy, potentially message-loss causing code, away from the input thread. > The only questionable effect I can currently see is that queue space is > potentially taken up by messages which will immediately be discarded and > should not be there in the first place. At the extreme end, that could lead > to loss of valid messages. But on the other hand valid messages are more > likely to be lost by the DNS name query overhead if I do the ACL check > directly in the input thread. > > As such, I think my intended move is correct. Does anyone have an argument > against the approach I am now taking? personally I don't think that this sort of filtering belongs in rsyslog, it can be done at the OS level (with things like iptables), or rsyslog could use the tcpwrappers library. both cases would filter (by IP) prior to it hitting rsyslog in the first place. in addition, with UDP the source IP can be forged easily (rsyslog now contains this capability), so as a security measure it's questionable anyway. I agree that fewer messages will probably be lost by accepting them and checking later than by pausing to do the check initially. David Lang From rgerhards at hq.adiscon.com Mon Nov 16 17:54:28 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 16 Nov 2009 17:54:28 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> > personally I don't think that this sort of filtering belongs in > rsyslog, > it can be done at the OS level (with things like iptables), or rsyslog > could use the tcpwrappers library. both cases would filter (by IP) > prior > to it hitting rsyslog in the first place. > > in addition, with UDP the source IP can be forged easily (rsyslog now > contains this capability), so as a security measure it's questionable > anyway. I agree, but many people seem to use this functionality, and it was introduced some years ago by request. I do not feel comfortable with the idea of removing support for it. The newer protocols do not support ACLs in any case. Are there some more voices in regard to removing that functionality? Would make the implementation (and probably the throughput) a bit simpler/faster. > I agree that fewer messages will probably be lost by accepting them and > checking later than by pausing to do the check initially. Thanks for voicing this. Rainer From rgerhards at hq.adiscon.com Mon Nov 16 18:08:32 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 16 Nov 2009 18:08:32 +0100 Subject: [rsyslog] support for arbitrary number of open files References: <4B017CBE.80904@redhat.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103412@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > Sent: Monday, November 16, 2009 5:25 PM > To: rsyslog-users > Subject: [rsyslog] support for arbitrary number of open files > > Hi all, > > currently the total number of files (and tcp connections) that can be > open simultaneously is limited by the select() system call. Ideally > this > would be changed to *poll(), but that can take some time. For imudp, this change already happened (in the current devel). The others are on my radar. TCP is probably the most problematic, I need to redo the driver level. I wonder if I should split out gssapi while doing so, because that would simplify a lot of the calling conventions (at the price of some code duplication). For real-world scenarios, I think tcp is probably the only real issue, it is hard to believe that e.g. imuxsock will have an issue with that ;) > Until that happens, this patch[1] tries to remove the limitations of > select() by enlarging the bit mask that is used for storing file > descriptor information and redefining the macros that process it. > This modification is inspired by Bind's use of select(). > It is rather a workaround and may not be entirely portable. > > The actual changes to the code aren't big, but they are in many places > so sufficient testing is needed. Allocating and freeing fd_sets in > some > frequently called functions may decrease performance. This can be dealt > with but would require more pervasive changes. I need to have a close look at the patch. I consider it useful, but I agree that it must be very carefully checked. I've had a quick look and my understanding is that it is enabled by conditional compilation. That would ease many of my concerns, as we could disable it by default and only those that actually need it are at risk. > > Thoughts? I'd also appreciate feedback from other list members. Thanks, Rainer > > Thanks, > Tomas > > [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited- > select.patch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From aoz.syn at gmail.com Mon Nov 16 18:45:57 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 16 Nov 2009 10:45:57 -0700 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> Message-ID: <4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com> On Mon, Nov 16, 2009 at 09:54, Rainer Gerhards wrote: > I agree, but many people seem to use this functionality, and it was > introduced some years ago by request. I do not feel comfortable with the idea > of removing support for it. The newer protocols do not support ACLs in any > case. > > Are there some more voices in regard to removing that functionality? Would > make the implementation (and probably the throughput) a bit simpler/faster. I agree with david's assessment that "security" by this type of ACL is minimally effective. However, the functionality is occasionally useful for situations where management of the software is easier than management of the firewall (typically for business/operational constraints). I'd love to see it reimplemented as a modular part of the ephemeral "middle layer" along with other filtering and modification functionality. I know RanierScript is supposed to fill a lot of that void, but until it's implemented and proven performant my wishes still lie with a filter layer API. >> I agree that fewer messages will probably be lost by accepting them and >> checking later than by pausing to do the check initially. Ditto - although conceptually pure, front-end checking is probably too expensive for such a performance-critical component as the receiver. From rgerhards at hq.adiscon.com Mon Nov 16 20:56:38 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 16 Nov 2009 20:56:38 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> <4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of RB > Sent: Monday, November 16, 2009 6:46 PM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback request: ACLs, imudp and > accepting messages > > On Mon, Nov 16, 2009 at 09:54, Rainer Gerhards > wrote: > > I agree, but many people seem to use this functionality, and it was > > introduced some years ago by request. I do not feel > comfortable with the idea > > of removing support for it. The newer protocols do not > support ACLs in any > > case. > > > > Are there some more voices in regard to removing that > functionality? Would > > make the implementation (and probably the throughput) a bit > simpler/faster. > > I agree with david's assessment that "security" by this type of ACL is > minimally effective. >From a security POV, it may be more effective to remove that option, at least for UDP (people than know it is not secure, but neither is a similar firewall setting). But I fear all those broken configs, then without any protection at all. Maybe a thing for a new v5 default, but even then... > However, the functionality is occasionally > useful for situations where management of the software is easier than > management of the firewall (typically for business/operational > constraints). I'd love to see it reimplemented as a modular part of > the ephemeral "middle layer" along with other filtering and > modification functionality. I fail to see the interface you envision. Could you elaborate a bit? That would be most useful. I guess that a lot of this can now be done by the custom parsers, it may just need to be documented as a valid use case. > I know RanierScript is supposed to fill a > lot of that void, but until it's implemented and proven performant My thinking is moving a bit away from this "once size fits all" approach, primarily for performance reasons. It is quite expensive to start up a full scripting engine (with its VM), so native code loadable modules require more work to be done upfront, but are much faster. Rainer > my > wishes still lie with a filter layer API. > > >> I agree that fewer messages will probably be lost by > accepting them and > >> checking later than by pausing to do the check initially. > > Ditto - although conceptually pure, front-end checking is probably too > expensive for such a performance-critical component as the receiver. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From aoz.syn at gmail.com Mon Nov 16 21:28:54 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 16 Nov 2009 13:28:54 -0700 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> <4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com> Message-ID: <4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> On Mon, Nov 16, 2009 at 12:56, Rainer Gerhards wrote: > I fail to see the interface you envision. Could you elaborate a bit? ?That > would be most useful. Certainly. Some time ago, I expressed interest in writing a 'running checksum' module but had run into issues with a lack of state-maintenance in the output modules that helped my lack of time drop the issue. At the time, you'd hinted at a 'filter' module API that would (I presumed) fit somewhere between input and output and eventually address that use case. Such a layer could perform message modification, rate limiting, ACL enforcement, and so on. Perhaps this has already been addressed, since I've not kept as close of tabs on the development head as I'd like. Ideally, I think it would be a very interesting step to be able to 'stack' syslog modules to define the path a given set of messages take through the collector. Something akin to the Linux iptables concept, where a basic high-speed framework is laid out with a well-defined order and an API that enables administrators to elect how specific run-time modules interact within the stack. That's less of a request and more of a pipe dream, and is definitely influenced by the half network security hat I wear. From rgerhards at hq.adiscon.com Mon Nov 16 21:52:15 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 16 Nov 2009 21:52:15 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com><4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com> <4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103415@GRFEXC.intern.adiscon.com> Let me think about this, but it really sounds like custom parser modules could be moved into that direction with relative ease (at least for part of the functionality). Some info on them: http://www.rsyslog.com/doc-messageparser.html Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of RB > Sent: Monday, November 16, 2009 9:29 PM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback request: ACLs, imudp and > accepting messages > > On Mon, Nov 16, 2009 at 12:56, Rainer Gerhards > wrote: > > I fail to see the interface you envision. Could you > elaborate a bit? ?That > > would be most useful. > > Certainly. Some time ago, I expressed interest in writing a 'running > checksum' module but had run into issues with a lack of > state-maintenance in the output modules that helped my lack of time > drop the issue. At the time, you'd hinted at a 'filter' module API > that would (I presumed) fit somewhere between input and output and > eventually address that use case. Such a layer could perform message > modification, rate limiting, ACL enforcement, and so on. Perhaps this > has already been addressed, since I've not kept as close of tabs on > the development head as I'd like. > > Ideally, I think it would be a very interesting step to be able to > 'stack' syslog modules to define the path a given set of messages take > through the collector. Something akin to the Linux iptables concept, > where a basic high-speed framework is laid out with a well-defined > order and an API that enables administrators to elect how specific > run-time modules interact within the stack. That's less of a request > and more of a pipe dream, and is definitely influenced by the half > network security hat I wear. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Tue Nov 17 03:58:14 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 16 Nov 2009 18:58:14 -0800 (PST) Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages In-Reply-To: <4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com> <4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com> <4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> Message-ID: On Mon, 16 Nov 2009, RB wrote: > On Mon, Nov 16, 2009 at 12:56, Rainer Gerhards wrote: >> I fail to see the interface you envision. Could you elaborate a bit? ?That >> would be most useful. > > Certainly. Some time ago, I expressed interest in writing a 'running > checksum' module but had run into issues with a lack of > state-maintenance in the output modules that helped my lack of time > drop the issue. At the time, you'd hinted at a 'filter' module API > that would (I presumed) fit somewhere between input and output and > eventually address that use case. Such a layer could perform message > modification, rate limiting, ACL enforcement, and so on. Perhaps this > has already been addressed, since I've not kept as close of tabs on > the development head as I'd like. > > Ideally, I think it would be a very interesting step to be able to > 'stack' syslog modules to define the path a given set of messages take > through the collector. Something akin to the Linux iptables concept, > where a basic high-speed framework is laid out with a well-defined > order and an API that enables administrators to elect how specific > run-time modules interact within the stack. That's less of a request > and more of a pipe dream, and is definitely influenced by the half > network security hat I wear. hmm, the recent ability to have secondary queues would allow this to fit in very nicely as a filter for what to do when moving messages between the queues. David Lang From philr at jaspers.co.nz Tue Nov 17 04:10:13 2009 From: philr at jaspers.co.nz (Phil Reilly) Date: Tue, 17 Nov 2009 16:10:13 +1300 Subject: [rsyslog] Alerting rules via a database In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> References: <4AF5DAA7.5010001@jaspers.co.nz> <4AF6816A.6050901@jaspers.co.nz> <9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> Message-ID: <4B021415.7080906@jaspers.co.nz> Any luck with the template? Or should I just roll my own. Cheers, Phil Rainer Gerhards wrote: > So what you are actually looking for is a system that can work with > dynamically changable alert definitions? As David said, there is no such > thing currently, but the best road to approach is is to write a custom output > plugin, that you pass each message to. That plugin can even decide if > messages should be discarded and not further processed. I envisioned such a > plugin, but had not yet time to write, for a similar use case. > > If you intend to write one AND contribute it to the project, I can help you > get started with the interface, would even be willing to create you a custom > skeleton that you can fill in your logic ;) > > HTH > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phil Reilly >> Sent: Sunday, November 08, 2009 9:30 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Alerting rules via a database >> >> david at lang.hm wrote: >> >>> On Sun, 8 Nov 2009, Phil Reilly wrote: >>> >>> >>> >>>> I attempting to allow for flexible rule matches on Syslogs >>>> >> from a web >> >>>> front end (rather than entires into the rsyslog config files) >>>> >>>> I want to get regexp filters from a db to alert upon >>>> >> messages. Not sure >> >>>> the best way to achieve this. I've so far though of. >>>> >>>> * Outputting to a pipe and runing it via an alerting script. >>>> * Having file watch the messages. >>>> * Recieving the messages then passing them to rsyslog (yuck) >>>> >>>> Can the rule engine allow for match rules outside of the config? is >>>> there an elegant way of doing this? >>>> >>>> >>> rsyslog doesn't give you this ability, but it's not really the best >>> approach to use for alerting anyway. >>> >>> what are you trying to achieve by having the alert definitions in a >>> database? there are several tools out there to do alerting >>> >> (SEC, Simple >> >>> Event Correlator) is one of the leading ones, but I'm not >>> >> aware of any of >> >>> them that use a database for their rulesets. >>> >>> I'm also scratching my head trying to figure out what the >>> >> advantage of >> >>> doing so would be. >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> Thanks David, >> >> We have a networked environment. We also have a web page that >> allows you >> to configure regexp to match certain syslog messages. These >> patterns are >> compiled and kept in a table. The current syslog process we >> use listens >> for udp. When it gets a syslog message, we examine the >> patterns (which >> are re-read upon addition or change) and pass them to an alertering >> process before writing the logs to disk. The existing system >> works well, >> but we now want to scale it over a few machines and I'm >> examining what >> syslog products out there cater for alerting. >> >> So a database will make configuring alerts far more dynamic than >> statically entering them into a config file. It will also allow for >> grouped views so different groups have the ability to have >> custom alerts >> based upon their own interpretation of syslog messages. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Nov 17 07:28:25 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 07:28:25 +0100 Subject: [rsyslog] Alerting rules via a database Message-ID: <000901ca674f$38fd00a1$100013ac@intern.adiscon.com> Sorry, i have to admit it slipped my mind. Will create one this morning. rainer ----- Urspr?ngliche Nachricht ----- Von: "Phil Reilly" An: "rsyslog-users" Gesendet: 17.11.09 04:10 Betreff: Re: [rsyslog] Alerting rules via a database Any luck with the template? Or should I just roll my own. Cheers, Phil Rainer Gerhards wrote: > So what you are actually looking for is a system that can work with > dynamically changable alert definitions? As David said, there is no such > thing currently, but the best road to approach is is to write a custom output > plugin, that you pass each message to. That plugin can even decide if > messages should be discarded and not further processed. I envisioned such a > plugin, but had not yet time to write, for a similar use case. > > If you intend to write one AND contribute it to the project, I can help you > get started with the interface, would even be willing to create you a custom > skeleton that you can fill in your logic ;) > > HTH > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phil Reilly >> Sent: Sunday, November 08, 2009 9:30 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Alerting rules via a database >> >> david at lang.hm wrote: >> >>> On Sun, 8 Nov 2009, Phil Reilly wrote: >>> >>> >>> >>>> I attempting to allow for flexible rule matches on Syslogs >>>> >> from a web >> >>>> front end (rather than entires into the rsyslog config files) >>>> >>>> I want to get regexp filters from a db to alert upon >>>> >> messages. Not sure >> >>>> the best way to achieve this. I've so far though of. >>>> >>>> * Outputting to a pipe and runing it via an alerting script. >>>> * Having file watch the messages. >>>> * Recieving the messages then passing them to rsyslog (yuck) >>>> >>>> Can the rule engine allow for match rules outside of the config? is >>>> there an elegant way of doing this? >>>> >>>> >>> rsyslog doesn't give you this ability, but it's not really the best >>> approach to use for alerting anyway. >>> >>> what are you trying to achieve by having the alert definitions in a >>> database? there are several tools out there to do alerting >>> >> (SEC, Simple >> >>> Event Correlator) is one of the leading ones, but I'm not >>> >> aware of any of >> >>> them that use a database for their rulesets. >>> >>> I'm also scratching my head trying to figure out what the >>> >> advantage of >> >>> doing so would be. >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> Thanks David, >> >> We have a networked environment. We also have a web page that >> allows you >> to configure regexp to match certain syslog messages. These >> patterns are >> compiled and kept in a table. The current syslog process we >> use listens >> for udp. When it gets a syslog message, we examine the >> patterns (which >> are re-read upon addition or change) and pass them to an alertering >> process before writing the logs to disk. The existing system >> works well, >> but we now want to scale it over a few machines and I'm >> examining what >> syslog products out there cater for alerting. >> >> So a database will make configuring alerts far more dynamic than >> statically entering them into a config file. It will also allow for >> grouped views so different groups have the ability to have >> custom alerts >> based upon their own interpretation of syslog messages. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 17 08:15:22 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 08:15:22 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com><4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com><4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103417@GRFEXC.intern.adiscon.com> > > Ideally, I think it would be a very interesting step to be able to > > 'stack' syslog modules to define the path a given set of messages > take > > through the collector. Something akin to the Linux iptables concept, > > where a basic high-speed framework is laid out with a well-defined > > order and an API that enables administrators to elect how specific > > run-time modules interact within the stack. That's less of a request > > and more of a pipe dream, and is definitely influenced by the half > > network security hat I wear. > > hmm, the recent ability to have secondary queues would allow this to > fit > in very nicely as a filter for what to do when moving messages between > the > queues. I have thought more about this over the night. I think most of the capabilities are already present, just need to be a bit "re-labled". However, what is missing is functionality to use loadable modules as a filter condition. However, especially for complex filters, this sounds very valuable, both from a performance as well as from a capability POV. It also looks like it is not hard to add. I will definitely look into that direction. Thanks all for bringing up these thoughts, any further comments are of course appreciated as well ;) Rainer From rgerhards at hq.adiscon.com Tue Nov 17 08:49:28 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 08:49:28 +0100 Subject: [rsyslog] Alerting rules via a database References: <4AF5DAA7.5010001@jaspers.co.nz><4AF6816A.6050901@jaspers.co.nz><9B6E2A8877C38245BFB15CC491A11DA7103388@GRFEXC.intern.adiscon.com> <4B021415.7080906@jaspers.co.nz> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103418@GRFEXC.intern.adiscon.com> It is now present in the master branch. It resides in ./plugins/omdbalerting to enable it, use $ ./configure --enable-omdbalerting Obviously, the code now needs to be populated. I suggest that you create a simple sample with fixed constants, submit it to me, and then we can talk about how to obtain parameters from the config file. I hope this is useful. If you have any question (I guess you will have), please ask. But be sure to review module-template.h before beginning with any work. Looking at omtemplate.c is also a good idea. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Phil Reilly > Sent: Tuesday, November 17, 2009 4:10 AM > To: rsyslog-users > Subject: Re: [rsyslog] Alerting rules via a database > > Any luck with the template? > > Or should I just roll my own. > > Cheers, > > Phil > > Rainer Gerhards wrote: > > So what you are actually looking for is a system that can work with > > dynamically changable alert definitions? As David said, there is no > such > > thing currently, but the best road to approach is is to write a > custom output > > plugin, that you pass each message to. That plugin can even decide if > > messages should be discarded and not further processed. I envisioned > such a > > plugin, but had not yet time to write, for a similar use case. > > > > If you intend to write one AND contribute it to the project, I can > help you > > get started with the interface, would even be willing to create you a > custom > > skeleton that you can fill in your logic ;) > > > > HTH > > Rainer > > > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phil Reilly > >> Sent: Sunday, November 08, 2009 9:30 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Alerting rules via a database > >> > >> david at lang.hm wrote: > >> > >>> On Sun, 8 Nov 2009, Phil Reilly wrote: > >>> > >>> > >>> > >>>> I attempting to allow for flexible rule matches on Syslogs > >>>> > >> from a web > >> > >>>> front end (rather than entires into the rsyslog config files) > >>>> > >>>> I want to get regexp filters from a db to alert upon > >>>> > >> messages. Not sure > >> > >>>> the best way to achieve this. I've so far though of. > >>>> > >>>> * Outputting to a pipe and runing it via an alerting script. > >>>> * Having file watch the messages. > >>>> * Recieving the messages then passing them to rsyslog (yuck) > >>>> > >>>> Can the rule engine allow for match rules outside of the config? > is > >>>> there an elegant way of doing this? > >>>> > >>>> > >>> rsyslog doesn't give you this ability, but it's not really the best > >>> approach to use for alerting anyway. > >>> > >>> what are you trying to achieve by having the alert definitions in a > >>> database? there are several tools out there to do alerting > >>> > >> (SEC, Simple > >> > >>> Event Correlator) is one of the leading ones, but I'm not > >>> > >> aware of any of > >> > >>> them that use a database for their rulesets. > >>> > >>> I'm also scratching my head trying to figure out what the > >>> > >> advantage of > >> > >>> doing so would be. > >>> > >>> David Lang > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >>> > >> Thanks David, > >> > >> We have a networked environment. We also have a web page that > >> allows you > >> to configure regexp to match certain syslog messages. These > >> patterns are > >> compiled and kept in a table. The current syslog process we > >> use listens > >> for udp. When it gets a syslog message, we examine the > >> patterns (which > >> are re-read upon addition or change) and pass them to an alertering > >> process before writing the logs to disk. The existing system > >> works well, > >> but we now want to scale it over a few machines and I'm > >> examining what > >> syslog products out there cater for alerting. > >> > >> So a database will make configuring alerts far more dynamic than > >> statically entering them into a config file. It will also allow for > >> grouped views so different groups have the ability to have > >> custom alerts > >> based upon their own interpretation of syslog messages. > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 17 10:00:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 10:00:44 +0100 Subject: [rsyslog] support for arbitrary number of open files References: <4B017CBE.80904@redhat.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710341A@GRFEXC.intern.adiscon.com> Tomas, I am currently working on integration of this patch. What puzzles me is the call to "howmany()". I don't find any doc on it, nor an implementation inside the patch. Could you elaborate what it does and where it stems from? Also, I think there is a segfault in gss-misc, because the glbl interface is never aquired (the will result in a NULL-pointer dereference). I also need to change the glbl interface definitions, FdSetSize must always be present, else the interface is no longer well-defined. I will post the completely integrated patch when I am done. But feedback on howmany() would be most appreciated, because I currently do not know exactly how to handle it. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > Sent: Monday, November 16, 2009 5:25 PM > To: rsyslog-users > Subject: [rsyslog] support for arbitrary number of open files > > Hi all, > > currently the total number of files (and tcp connections) that can be > open simultaneously is limited by the select() system call. Ideally > this > would be changed to *poll(), but that can take some time. > > Until that happens, this patch[1] tries to remove the limitations of > select() by enlarging the bit mask that is used for storing file > descriptor information and redefining the macros that process it. > This modification is inspired by Bind's use of select(). > It is rather a workaround and may not be entirely portable. > > The actual changes to the code aren't big, but they are in many places > so sufficient testing is needed. Allocating and freeing fd_sets in > some > frequently called functions may decrease performance. This can be dealt > with but would require more pervasive changes. > > Thoughts? > > Thanks, > Tomas > > [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited- > select.patch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 17 10:49:16 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 10:49:16 +0100 Subject: [rsyslog] support for arbitrary number of open files References: <4B017CBE.80904@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA710341A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710341B@GRFEXC.intern.adiscon.com> OK, I have finally integrated the patch, I could work around my missing knowledge of howmany() [but still would appreciate some more info on it]. As of rsyslog policy, new features are never integrated into stable or beta builds. As such, it is available in the v4-devel and master branches. Note that I made a number of changes, you will probably like to re-check for your use case. I ran the testbench successfully in both modes and did some manual tests with the new functionality disabled. The v5-branch does NOT include the patch for imudp. As you said, it is a (good ;)) work-around and imudp already supports epoll(), so there is no need for this workaround. I plan to gradually remove that feature in v5 a I work towards supporting epoll in all inputs. The master branch will probably be released tomorrow, v4-devel as need arises (but it is available from git right now). Thanks again, I think this is a very useful addition. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, November 17, 2009 10:01 AM > To: rsyslog-users > Subject: Re: [rsyslog] support for arbitrary number of open files > > Tomas, > > I am currently working on integration of this patch. What puzzles me is > the > call to "howmany()". I don't find any doc on it, nor an implementation > inside > the patch. Could you elaborate what it does and where it stems from? > > Also, I think there is a segfault in gss-misc, because the glbl > interface is > never aquired (the will result in a NULL-pointer dereference). I also > need to > change the glbl interface definitions, FdSetSize must always be > present, else > the interface is no longer well-defined. I will post the completely > integrated patch when I am done. > > But feedback on howmany() would be most appreciated, because I > currently do > not know exactly how to handle it. > > Thanks, > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > > Sent: Monday, November 16, 2009 5:25 PM > > To: rsyslog-users > > Subject: [rsyslog] support for arbitrary number of open files > > > > Hi all, > > > > currently the total number of files (and tcp connections) that can be > > open simultaneously is limited by the select() system call. Ideally > > this > > would be changed to *poll(), but that can take some time. > > > > Until that happens, this patch[1] tries to remove the limitations of > > select() by enlarging the bit mask that is used for storing file > > descriptor information and redefining the macros that process it. > > This modification is inspired by Bind's use of select(). > > It is rather a workaround and may not be entirely portable. > > > > The actual changes to the code aren't big, but they are in many > places > > so sufficient testing is needed. Allocating and freeing fd_sets in > > some > > frequently called functions may decrease performance. This can be > dealt > > with but would require more pervasive changes. > > > > Thoughts? > > > > Thanks, > > Tomas > > > > [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited- > > select.patch > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Nov 17 11:29:57 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 17 Nov 2009 11:29:57 +0100 Subject: [rsyslog] Feedback request: ACLs, imudp and accepting messages References: <9B6E2A8877C38245BFB15CC491A11DA710340D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103411@GRFEXC.intern.adiscon.com><4255c2570911160945w7bba0fdctc7e31c16d77b91e1@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103413@GRFEXC.intern.adiscon.com><4255c2570911161228l72800241o5985ac34a98d5b36@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103417@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710341E@GRFEXC.intern.adiscon.com> I have added some doc about what modules can do today: http://www.rsyslog.com/doc-rsyslog_conf_modules.html Note that message modification is possible, it was just not spelled out. The actual code also currently creates the false impression it is not possible. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, November 17, 2009 8:15 AM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback request: ACLs, imudp and accepting > messages > > > > Ideally, I think it would be a very interesting step to be able to > > > 'stack' syslog modules to define the path a given set of messages > > take > > > through the collector. Something akin to the Linux iptables > concept, > > > where a basic high-speed framework is laid out with a well-defined > > > order and an API that enables administrators to elect how specific > > > run-time modules interact within the stack. That's less of a > request > > > and more of a pipe dream, and is definitely influenced by the half > > > network security hat I wear. > > > > hmm, the recent ability to have secondary queues would allow this to > > fit > > in very nicely as a filter for what to do when moving messages > between > > the > > queues. > > I have thought more about this over the night. I think most of the > capabilities are already present, just need to be a bit "re-labled". > However, > what is missing is functionality to use loadable modules as a filter > condition. However, especially for complex filters, this sounds very > valuable, both from a performance as well as from a capability POV. It > also > looks like it is not hard to add. > > I will definitely look into that direction. Thanks all for bringing up > these > thoughts, any further comments are of course appreciated as well ;) > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Nov 18 04:13:06 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 17 Nov 2009 19:13:06 -0800 (PST) Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 13 Nov 2009, david at lang.hm wrote: > I've got this installed in production and it's working well so far. monday > morning (pacific time) should give it a good stress test. it's survived since friday without a problem on a couple of dozen machines (all my perimiter relay boxes plus my core logging server) looking good. David Lang > David Lang > > On Fri, 13 Nov 2009, Tom Bergfeld wrote: > >> Date: Fri, 13 Nov 2009 12:51:09 +0100 >> From: Tom Bergfeld >> Reply-To: rsyslog-users >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released >> >> Hi all, >> >> Today, we released a new v5-beta branch. It bases on the last v5-devel, >> plus >> has some bug fixes and some enhancements. The enhancements provide a >> slightly >> increased performance. This release is a very important milestone. Based on >> feedback from the v5-devel branches, it is in pretty good shape. Our aim is >> to have a quick beta phase this time (taking the problems with the current >> v5-stable) into account. To do so, we need your cooperation. Please try out >> this new version and report back any issues you may experience. It would >> also >> be good to know if you use it and do not run into any issues. >> >> See Changelog for more details. >> >> ChangeLog: >> >> http://www.rsyslog.com/Article427.phtml >> >> Download: >> >> http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-187.phtml >> >> >> As always, feedback is appreciated. >> >> Best regards, >> Tom Bergfeld >> -- >> Support >> ======= >> >> Improving rsyslog is costly, but you can help! We are looking for >> organizations that find rsyslog useful and wish to contribute back. You >> can >> contribute by reporting bugs, improve the software, or donate money or >> equipment. >> >> Commercial support contracts for rsyslog are available, and they help >> finance >> continued maintenance. Adiscon GmbH, a privately held German company, is >> currently funding rsyslog development. We are always looking for >> interesting >> development projects. For details on how to help, please see >> http://www.rsyslog.com/doc-how2help.html . >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > From rgerhards at hq.adiscon.com Wed Nov 18 09:23:24 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 18 Nov 2009 09:23:24 +0100 Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released References: <9B6E2A8877C38245BFB15CC491A11DA71033F1@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710342D@GRFEXC.intern.adiscon.com> This is excellent news, thanks for sharing it. Did anybody else also try it? Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, November 18, 2009 4:13 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 5.3.5 (v5-beta) released > > On Fri, 13 Nov 2009, david at lang.hm wrote: > > > I've got this installed in production and it's working well so far. > monday > > morning (pacific time) should give it a good stress test. > > it's survived since friday without a problem on a couple of dozen > machines > (all my perimiter relay boxes plus my core logging server) > > looking good. > > David Lang > > > David Lang > > > > On Fri, 13 Nov 2009, Tom Bergfeld wrote: > > > >> Date: Fri, 13 Nov 2009 12:51:09 +0100 > >> From: Tom Bergfeld > >> Reply-To: rsyslog-users > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] rsyslog 5.3.5 (v5-beta) released > >> > >> Hi all, > >> > >> Today, we released a new v5-beta branch. It bases on the last v5- > devel, > >> plus > >> has some bug fixes and some enhancements. The enhancements provide a > >> slightly > >> increased performance. This release is a very important milestone. > Based on > >> feedback from the v5-devel branches, it is in pretty good shape. Our > aim is > >> to have a quick beta phase this time (taking the problems with the > current > >> v5-stable) into account. To do so, we need your cooperation. Please > try out > >> this new version and report back any issues you may experience. It > would > >> also > >> be good to know if you use it and do not run into any issues. > >> > >> See Changelog for more details. > >> > >> ChangeLog: > >> > >> http://www.rsyslog.com/Article427.phtml > >> > >> Download: > >> > >> http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid- > 187.phtml > >> > >> > >> As always, feedback is appreciated. > >> > >> Best regards, > >> Tom Bergfeld > >> -- > >> Support > >> ======= > >> > >> Improving rsyslog is costly, but you can help! We are looking for > >> organizations that find rsyslog useful and wish to contribute back. > You > >> can > >> contribute by reporting bugs, improve the software, or donate money > or > >> equipment. > >> > >> Commercial support contracts for rsyslog are available, and they > help > >> finance > >> continued maintenance. Adiscon GmbH, a privately held German > company, is > >> currently funding rsyslog development. We are always looking for > >> interesting > >> development projects. For details on how to help, please see > >> http://www.rsyslog.com/doc-how2help.html . > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From martinmie at PartyGaming.com Wed Nov 18 10:52:21 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Wed, 18 Nov 2009 10:52:21 +0100 Subject: [rsyslog] rsyslog in uninterruptable sleep In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032D3@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA0892EE19@GIBSVWIN004X.partygaming.local> <9B6E2A8877C38245BFB15CC491A11DA71032D3@GRFEXC.intern.adiscon.com> Message-ID: <0B1B9163138571439EAF804646F3F6AA08B398B6@GIBSVWIN004X.partygaming.local> Hi Rainer, Since some weeks ago we run rsyslog 4.4.2 and unfortunately yesterday I stumbled on rsyslog with "D"-state process again... so I guess that the fix is somewhere else. Can you please confirm which rsyslog version provides the solution for this problem? Thanks, Martin > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: 27 October 2009 16:02 > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog in uninterruptable sleep > > I think we just recently had this on the mailing list or bug tracker and I > made a fix for it. I guess the fix is included in 4.4.2. > > HTH > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Martin Mielke > > Sent: Tuesday, October 27, 2009 11:57 AM > > To: rsyslog-users > > Subject: [rsyslog] rsyslog in uninterruptable sleep > > > > > > Hi, > > > > I just did some small changes to the rsyslog.conf and wanted to restart > > the process: > > > > # /etc/init.d/rsyslog restart > > Shutting down system logger: [FAILED] > > Starting system logger: Already running. > > [FAILED] > > > > # ps -elf | grep rsyslog > > 5 D root 2037 1 1 76 0 - 12040 sync_b Oct18 ? > > 03:32:10 rsyslogd -c 4 -x -f /etc/rsyslog.conf -i /var/run/syslogd.pid > > > > > > This is rsyslog 4.2.0 running on RHEL5 and it's not the first time it > > happens. > > > > > > Any ideas on why this might happened and how to solve this w/o > > rebooting > > the system?? > > > > > > Thanks, > > Martin > > > > > > This email and any attachments are confidential, and may be legally > > privileged and protected by copyright. If you are not the intended > > recipient dissemination or copying of this email is prohibited. If you > > have received this in error, please notify the sender by replying by > > email and then delete the email completely from your system. > > > > Any views or opinions are solely those of the sender. This > > communication is not intended to form a binding contract unless > > expressly indicated to the contrary and properly authorised. Any > > actions taken on the basis of this email are at the recipient's own > > risk. > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > yslog > http://www.rsyslog.com From tbergfeld at hq.adiscon.com Wed Nov 18 11:05:28 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 18 Nov 2009 11:05:28 +0100 Subject: [rsyslog] rsyslog 4.5.7 (v4-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103435@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.5.7, a member of the v4-beta branch. Most importantly, a so-called "On Demand Debug" mode was added, in which debug output can be generated only after the process has started, but not right from the beginning. This is assumed to be useful for hard-to-find bugs. Also improved the doc on the debug system. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article429.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-188.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From tbergfeld at hq.adiscon.com Wed Nov 18 11:14:30 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 18 Nov 2009 11:14:30 +0100 Subject: [rsyslog] rsyslog 5.5.0 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103437@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.5.0, a member of the devel branch. The 5.5.0 version begins a new development branch. The primary new functionality in that release is moving away reverse DNS resolution from the udp receiver's input thread to backend processing, what should reduce UDP message loss. ChangeLog: http://www.rsyslog.com/Article431.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-189.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From theinric at redhat.com Wed Nov 18 14:18:48 2009 From: theinric at redhat.com (Tomas Heinrich) Date: Wed, 18 Nov 2009 14:18:48 +0100 Subject: [rsyslog] support for arbitrary number of open files In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710341B@GRFEXC.intern.adiscon.com> References: <4B017CBE.80904@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA710341A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710341B@GRFEXC.intern.adiscon.com> Message-ID: <4B03F438.5030303@redhat.com> Rainer, thanks for integrating it and thanks for fixing the things I've overlooked. I've skimmed through the commit logs and the changes look good, I'll run some more tests with the new sources. Regarding howmany(), it is defined here in my environment: /usr/include/sys/param.h: # define howmany(x, y) (((x) + ((y) - 1)) / (y)) It computes the number of 'y's needed to store 'x'. Tomas On 11/17/2009 10:49 AM, Rainer Gerhards wrote: > OK, I have finally integrated the patch, I could work around my missing > knowledge of howmany() [but still would appreciate some more info on it]. > > As of rsyslog policy, new features are never integrated into stable or beta > builds. As such, it is available in the v4-devel and master branches. Note > that I made a number of changes, you will probably like to re-check for your > use case. I ran the testbench successfully in both modes and did some manual > tests with the new functionality disabled. > > The v5-branch does NOT include the patch for imudp. As you said, it is a > (good ;)) work-around and imudp already supports epoll(), so there is no need > for this workaround. I plan to gradually remove that feature in v5 a I work > towards supporting epoll in all inputs. > > The master branch will probably be released tomorrow, v4-devel as need arises > (but it is available from git right now). > > Thanks again, I think this is a very useful addition. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Tuesday, November 17, 2009 10:01 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] support for arbitrary number of open files >> >> Tomas, >> >> I am currently working on integration of this patch. What puzzles me is >> the >> call to "howmany()". I don't find any doc on it, nor an implementation >> inside >> the patch. Could you elaborate what it does and where it stems from? >> >> Also, I think there is a segfault in gss-misc, because the glbl >> interface is >> never aquired (the will result in a NULL-pointer dereference). I also >> need to >> change the glbl interface definitions, FdSetSize must always be >> present, else >> the interface is no longer well-defined. I will post the completely >> integrated patch when I am done. >> >> But feedback on howmany() would be most appreciated, because I >> currently do >> not know exactly how to handle it. >> >> Thanks, >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich >>> Sent: Monday, November 16, 2009 5:25 PM >>> To: rsyslog-users >>> Subject: [rsyslog] support for arbitrary number of open files >>> >>> Hi all, >>> >>> currently the total number of files (and tcp connections) that can be >>> open simultaneously is limited by the select() system call. Ideally >>> this >>> would be changed to *poll(), but that can take some time. >>> >>> Until that happens, this patch[1] tries to remove the limitations of >>> select() by enlarging the bit mask that is used for storing file >>> descriptor information and redefining the macros that process it. >>> This modification is inspired by Bind's use of select(). >>> It is rather a workaround and may not be entirely portable. >>> >>> The actual changes to the code aren't big, but they are in many >> places >>> so sufficient testing is needed. Allocating and freeing fd_sets in >>> some >>> frequently called functions may decrease performance. This can be >> dealt >>> with but would require more pervasive changes. >>> >>> Thoughts? >>> >>> Thanks, >>> Tomas >>> >>> [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited- >>> select.patch >>> From rgerhards at hq.adiscon.com Wed Nov 18 14:35:33 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 18 Nov 2009 14:35:33 +0100 Subject: [rsyslog] support for arbitrary number of open files References: <4B017CBE.80904@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA710341A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710341B@GRFEXC.intern.adiscon.com> <4B03F438.5030303@redhat.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710343E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > Sent: Wednesday, November 18, 2009 2:19 PM > To: rsyslog-users > Subject: Re: [rsyslog] support for arbitrary number of open files > > Rainer, > > thanks for integrating it and thanks for fixing the things I've > overlooked. > > I've skimmed through the commit logs and the changes look good, I'll > run > some more tests with the new sources. excellent - pls let me know if you see some issues. As a side-note, I've also used this as a reminder to finally look at epoll() for the plain tcp driver, probably the most prominent case, also performance wise. I hope I am able continue work on it, so we should have it in the not so distant future (in v5). > > Regarding howmany(), it is defined here in my environment: > /usr/include/sys/param.h: > # define howmany(x, y) (((x) + ((y) - 1)) / (y)) > It computes the number of 'y's needed to store 'x'. > Ahhh! I should have done a grep ;) But somehow did not expect it there... Thanks, Rainer > Tomas > > On 11/17/2009 10:49 AM, Rainer Gerhards wrote: > > OK, I have finally integrated the patch, I could work around my > missing > > knowledge of howmany() [but still would appreciate some more info on > it]. > > > > As of rsyslog policy, new features are never integrated into stable > or beta > > builds. As such, it is available in the v4-devel and master branches. > Note > > that I made a number of changes, you will probably like to re-check > for your > > use case. I ran the testbench successfully in both modes and did some > manual > > tests with the new functionality disabled. > > > > The v5-branch does NOT include the patch for imudp. As you said, it > is a > > (good ;)) work-around and imudp already supports epoll(), so there is > no need > > for this workaround. I plan to gradually remove that feature in v5 a > I work > > towards supporting epoll in all inputs. > > > > The master branch will probably be released tomorrow, v4-devel as > need arises > > (but it is available from git right now). > > > > Thanks again, I think this is a very useful addition. > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >> Sent: Tuesday, November 17, 2009 10:01 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] support for arbitrary number of open files > >> > >> Tomas, > >> > >> I am currently working on integration of this patch. What puzzles me > is > >> the > >> call to "howmany()". I don't find any doc on it, nor an > implementation > >> inside > >> the patch. Could you elaborate what it does and where it stems from? > >> > >> Also, I think there is a segfault in gss-misc, because the glbl > >> interface is > >> never aquired (the will result in a NULL-pointer dereference). I > also > >> need to > >> change the glbl interface definitions, FdSetSize must always be > >> present, else > >> the interface is no longer well-defined. I will post the completely > >> integrated patch when I am done. > >> > >> But feedback on howmany() would be most appreciated, because I > >> currently do > >> not know exactly how to handle it. > >> > >> Thanks, > >> Rainer > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > >>> Sent: Monday, November 16, 2009 5:25 PM > >>> To: rsyslog-users > >>> Subject: [rsyslog] support for arbitrary number of open files > >>> > >>> Hi all, > >>> > >>> currently the total number of files (and tcp connections) that can > be > >>> open simultaneously is limited by the select() system call. Ideally > >>> this > >>> would be changed to *poll(), but that can take some time. > >>> > >>> Until that happens, this patch[1] tries to remove the limitations > of > >>> select() by enlarging the bit mask that is used for storing file > >>> descriptor information and redefining the macros that process it. > >>> This modification is inspired by Bind's use of select(). > >>> It is rather a workaround and may not be entirely portable. > >>> > >>> The actual changes to the code aren't big, but they are in many > >> places > >>> so sufficient testing is needed. Allocating and freeing fd_sets in > >>> some > >>> frequently called functions may decrease performance. This can be > >> dealt > >>> with but would require more pervasive changes. > >>> > >>> Thoughts? > >>> > >>> Thanks, > >>> Tomas > >>> > >>> [1] - http://people.redhat.com/pvrabec/rsyslog-4.4.2-unlimited- > >>> select.patch > >>> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Nov 19 10:58:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 19 Nov 2009 10:58:53 +0100 Subject: [rsyslog] clarification on priorities for rsyslog work Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710344B@GRFEXC.intern.adiscon.com> Hi all, as we currently have a lot of requests (both bugs and features), I would like to make *my* decision process a bit more transparent. So I have blogged about how I usually assign priorities: http://blog.gerhards.net/2009/11/priorities-for-rsyslog-work.html I hope this is useful. Rainer From mrdemeanour at jackpot.uk.net Thu Nov 19 19:19:02 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Thu, 19 Nov 2009 18:19:02 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <4AF2F9CD.9000402@jackpot.uk.net> References: <4AF2F9CD.9000402@jackpot.uk.net> Message-ID: <4B058C16.7010008@jackpot.uk.net> Mr. Demeanour wrote: > Hi, > > I'm running a central rsyslog server with a couple of remote WAN > (internet) clients and several remote LAN clients. Traffic is low - > of the order of 10,000 messages per day. Internet clients communicate > with the server using gnutls. LAN clients are currently using UDP. > The server writes client logs to mysql, and also writes messages of > local origin to disk. Further to this: I have been running 4.5.6 for about a week now, *without* gnutls enabled. No leaks. This evening I re-enabled gnutls, and almost immediately noted excessive memory usage, *and* 99% cpu. It seems that the high CPU usage occurs with hosts outside my local network; it may be that there is some misconfiguration of NAT that is behind that problem. I note that leaks are possible with the versions of gnutls shipping with Debian: http://permalink.gmane.org/gmane.network.gnutls.general/1465 That document describes a leak that would be expected to arise during connection setup, but not per message. I guess a dubious connection (e.g. resulting from misconfigured NAT) might result in repeated setup attempts, and so in leaks *and* cpu spiking. -- Jack. From mrdemeanour at jackpot.uk.net Thu Nov 19 19:50:16 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Thu, 19 Nov 2009 18:50:16 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <4B058C16.7010008@jackpot.uk.net> References: <4AF2F9CD.9000402@jackpot.uk.net> <4B058C16.7010008@jackpot.uk.net> Message-ID: <4B059368.2070701@jackpot.uk.net> Mr. Demeanour wrote: > > It seems that the high CPU usage occurs with hosts outside my local > network; it may be that there is some misconfiguration of NAT that is > behind that problem. Hmmm. What I meant is that the problem was only observed with gnutls *clients* outside my local network; the rsyslog server is inside. Sorry for being unclear. -- Jack. From rgerhards at hq.adiscon.com Thu Nov 19 21:30:22 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 19 Nov 2009 21:30:22 +0100 Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing References: <003001ca6092$792833d0$6b789b70$@com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> Hi, Sorry, I seem to have overlooked that message. But the mailing list processor has stripped the attachment. Could you please re-send to my personal mail address. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Jonathan Bond-Caron > Sent: Sunday, November 08, 2009 5:43 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing > > Hi all, > > > > Attached is a patch for 3 things: > > > > a) Check that TCP connection is still alive when using TLS > > > > b) Improve TAG parsing so that it ends when it sees a > tab or escape > control character. > > > > c) Added configuration directive: escapecontrolcharactertab > > In rsyslog.conf, you can set: > > $escapeControlCharactersOnReceive on > > $escapeControlCharacterTab off > > > > This avoids having a tab being escaped since it is after a viewable > character J > > I'd prefer this being the default behavior but I've left > $escapeControlCharacterTab on as default in case people > expect tabs to be > escaped. > > > > This is my first GIT patch, hope it works well ;) > > > > From david at lang.hm Thu Nov 19 21:37:32 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 19 Nov 2009 12:37:32 -0800 (PST) Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> References: <003001ca6092$792833d0$6b789b70$@com> <9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> Message-ID: for what it's worth, I think that b and c are very useful when dealing with strange data sources, and a is probably a bugfix. David Lang On Thu, 19 Nov 2009, Rainer Gerhards wrote: > > Sorry, I seem to have overlooked that message. But the mailing list processor > has stripped the attachment. Could you please re-send to my personal mail > address. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >> Jonathan Bond-Caron >> Sent: Sunday, November 08, 2009 5:43 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing >> >> Hi all, >> >> >> >> Attached is a patch for 3 things: >> >> >> >> a) Check that TCP connection is still alive when using TLS >> >> >> >> b) Improve TAG parsing so that it ends when it sees a >> tab or escape >> control character. >> >> >> >> c) Added configuration directive: escapecontrolcharactertab >> >> In rsyslog.conf, you can set: >> >> $escapeControlCharactersOnReceive on >> >> $escapeControlCharacterTab off >> >> >> >> This avoids having a tab being escaped since it is after a viewable >> character J >> >> I'd prefer this being the default behavior but I've left >> $escapeControlCharacterTab on as default in case people >> expect tabs to be >> escaped. >> >> >> >> This is my first GIT patch, hope it works well ;) >> >> >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Nov 20 11:03:59 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 20 Nov 2009 11:03:59 +0100 Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing References: <003001ca6092$792833d0$6b789b70$@com><9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710346E@GRFEXC.intern.adiscon.com> I, too, think this is useful. Will look at it as soon as I get the patch. The b) and c) parts may be a bit harder to integrate if they are not for the newest devel branch, as I rewrote the parser part of rsyslog. But in any case, it is worth trying to merge it in. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, November 19, 2009 9:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] [PATCH] TLS connection check + syslog TAG > parsing > > for what it's worth, I think that b and c are very useful when dealing > with strange data sources, and a is probably a bugfix. > > David Lang > > On Thu, 19 Nov 2009, Rainer Gerhards wrote: > > > > > Sorry, I seem to have overlooked that message. But the mailing list > processor > > has stripped the attachment. Could you please re-send to my personal > mail > > address. > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >> Jonathan Bond-Caron > >> Sent: Sunday, November 08, 2009 5:43 PM > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing > >> > >> Hi all, > >> > >> > >> > >> Attached is a patch for 3 things: > >> > >> > >> > >> a) Check that TCP connection is still alive when using TLS > >> > >> > >> > >> b) Improve TAG parsing so that it ends when it sees a > >> tab or escape > >> control character. > >> > >> > >> > >> c) Added configuration directive: escapecontrolcharactertab > >> > >> In rsyslog.conf, you can set: > >> > >> $escapeControlCharactersOnReceive on > >> > >> $escapeControlCharacterTab off > >> > >> > >> > >> This avoids having a tab being escaped since it is after a viewable > >> character J > >> > >> I'd prefer this being the default behavior but I've left > >> $escapeControlCharacterTab on as default in case people > >> expect tabs to be > >> escaped. > >> > >> > >> > >> This is my first GIT patch, hope it works well ;) > >> > >> > >> > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mrdemeanour at jackpot.uk.net Fri Nov 20 17:10:24 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 20 Nov 2009 16:10:24 +0000 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls In-Reply-To: <4B058C16.7010008@jackpot.uk.net> References: <4AF2F9CD.9000402@jackpot.uk.net> <4B058C16.7010008@jackpot.uk.net> Message-ID: <4B06BF70.6070900@jackpot.uk.net> Mr. Demeanour wrote: > Mr. Demeanour wrote: >> Hi, >> >> I'm running a central rsyslog server with a couple of remote WAN >> (internet) clients and several remote LAN clients. Traffic is low - >> of the order of 10,000 messages per day. Internet clients >> communicate with the server using gnutls. LAN clients are currently >> using UDP. The server writes client logs to mysql, and also writes >> messages of local origin to disk. > > Further to this: > > I have been running 4.5.6 for about a week now, *without* gnutls > enabled. No leaks. > > This evening I re-enabled gnutls, and almost immediately noted > excessive memory usage, *and* 99% cpu. > > It seems that the high CPU usage occurs with hosts outside my local > network; it may be that there is some misconfiguration of NAT that is > behind that problem. Not NAT. It seems that I had set up the server certificate with an incorrect CN. I guess the client was trying repeatedly to make a connection that was doomed to fail every time. That would explain the CPU spike. If there is also a memory leak in the gnutls server code concerning connection setup, that would explain the memory consumption also. Perhaps rsyslog should give up trying to connect to a remote server, or at least back off, if the error it encounters is of a kind that most likely requires human intervention? Such would generally be the case if a certificate is invalid. -- Jack. From mrdemeanour at jackpot.uk.net Fri Nov 20 17:11:37 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 20 Nov 2009 16:11:37 +0000 Subject: [rsyslog] Possible bug: REs in templates Message-ID: <4B06BFB9.8080100@jackpot.uk.net> The following template produces one character fewer than expected. %HOSTNAME:R,ERE,1,FIELD,0:([^\.]+)--end% If, for example, the HOSTNAME string is 192.168.1.1, then the output is "19". -- Jack. From rgerhards at hq.adiscon.com Sun Nov 22 12:42:20 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 22 Nov 2009 12:42:20 +0100 Subject: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls References: <4AF2F9CD.9000402@jackpot.uk.net> <4B058C16.7010008@jackpot.uk.net> <4B06BF70.6070900@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710348E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Friday, November 20, 2009 5:10 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Mr. Demeanour wrote: > > Mr. Demeanour wrote: > >> Hi, > >> > >> I'm running a central rsyslog server with a couple of remote WAN > >> (internet) clients and several remote LAN clients. Traffic is low - > >> of the order of 10,000 messages per day. Internet clients > >> communicate with the server using gnutls. LAN clients are currently > >> using UDP. The server writes client logs to mysql, and also writes > >> messages of local origin to disk. > > > > Further to this: > > > > I have been running 4.5.6 for about a week now, *without* gnutls > > enabled. No leaks. > > > > This evening I re-enabled gnutls, and almost immediately noted > > excessive memory usage, *and* 99% cpu. > > > > It seems that the high CPU usage occurs with hosts outside my local > > network; it may be that there is some misconfiguration of > NAT that is > > behind that problem. > > Not NAT. It seems that I had set up the server certificate with an > incorrect CN. > > I guess the client was trying repeatedly to make a connection that was > doomed to fail every time. That would explain the CPU spike. > If there is > also a memory leak in the gnutls server code concerning connection > setup, that would explain the memory consumption also. > > Perhaps rsyslog should give up trying to connect to a remote > server, or > at least back off, if the error it encounters is of a kind that most > likely requires human intervention? Such would generally be > the case if > a certificate is invalid. The situation is a bit complex in such cases. There, a shortcoming in RFC5425 "works together" with a shortcoming of GnuTLS. The end result is that I am unable to detect a problem occurs. This happens: RFC5425 does not specifiy app-level acknowledgements. Consequently, there is no way to convey an error condition back to the client. GnuTLS does not permit to check the certificate *BEFORE* accepting a connection. Consequently, the connection must be accepted, then the certificate checked and then terminated if the cert if invalid. However, to the client this looks like a successfully established connection (beause it was connected) that just quickly went down (but the client does not know an error state as there is no app-layer vehicle to provide it back). This also disables the usual rsyslog logic to pause failed connection attempts - because the connection succeeds in the first place. The only solution I currently see is either not to use RFC5425 or use something else than GnuTLS (for example, openssl permits to check the certificate BEFORE accepting the connection). An other approach is to modify GnuTLS so that it provides an ability to do the check before accepting the certificate. I looked into that option, but it requires far too much effort to me. So the right thing to do would probably write an additional TLS driver for e.g. openssl. While this is actually on the todo list, I have to admit that I do not have time to do so at the moment, nor do I see a spot in the near future where I could tackle that beast. I still have the hope that we will receive a contributed driver for NSS, which would most probably also solve this issue. Or wait for GnuTLS to evolve... I guess you get the picture ;) Rainer From xkubina at fi.muni.cz Mon Nov 23 16:56:14 2009 From: xkubina at fi.muni.cz (Tomas Kubina) Date: Mon, 23 Nov 2009 16:56:14 +0100 Subject: [rsyslog] imgssapi configuration Message-ID: <4B0AB09E.9030005@fi.muni.cz> Hi, I am trying to set up rsyslog server with GSSAPI auth. I have used the doc that is here http://www.rsyslog.com/doc-gssapi.html but without successful end. The issue is that when I want to start rsyslog server it hangs-up during starting procedure. Here are some details: I have version 5.2.0 and this is server's conf file: === $ModLoad immark # provides --MARK-- message capability $ModLoad imuxsock # provides support for local system logging (e.g. via logger) $ModLoad imklog # kernel logging (formerly provided by rklogd) #Global directives $ActionFileDefaultTemplate RSYSLOG_FileFormat <(rules for storing syslog messages)> $ModLoad imgssapi $InputGSSServerPermitPlainTCP off $InputGSSServerRun 10514 === client config is simple: === $ModLoad omgssapi *.info : omgssapi:server.example.com:10514 === Here is a part of debug output: ============================ 9766.638881000:2b51af4b1ac0: cfline: '$ModLoad imgssapi' 9766.638893000:2b51af4b1ac0: Requested to load module 'imgssapi' 9766.638905000:2b51af4b1ac0: loading module '/usr/local/lib/rsyslog/imgssapi.so' 9766.640185000:2b51af4b1ac0: caller requested object 'tcps_sess', not found (iRet -3003) 9766.640207000:2b51af4b1ac0: Requested to load module 'lmtcpsrv' 9766.640219000:2b51af4b1ac0: loading module '/usr/local/lib/rsyslog/lmtcpsrv.so' 9766.640292000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 80: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[1/2b51af4b1ac0] 9766.640309000:2b51af4b1ac0: caller requested object 'netstrm', not found (iRet -3003) 9766.640322000:2b51af4b1ac0: Requested to load module 'lmnetstrms' 9766.640333000:2b51af4b1ac0: loading module '/usr/local/lib/rsyslog/lmnetstrms.so' 9766.640394000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 80: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[1/2b51af4b1ac0] 9766.640409000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640424000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 80: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[1/2b51af4b1ac0] 9766.640436000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640450000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 80: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[1/2b51af4b1ac0] 9766.640461000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640477000:2b51af4b1ac0: module of type 2 being loaded. 9766.640488000:2b51af4b1ac0: entry point 'isCompatibleWithFeature' not present in module 9766.640500000:2b51af4b1ac0: source file tcps_sess.c requested reference for module 'lmnetstrms', reference count now 1 9766.640512000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640528000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640541000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640553000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640573000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640587000:2b51af4b1ac0: source file tcpsrv.c requested reference for module 'lmnet', reference count now 4 9766.640599000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640611000:2b51af4b1ac0: source file tcpsrv.c requested reference for module 'lmnetstrms', reference count now 2 9766.640624000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640637000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640651000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640665000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640678000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640691000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640705000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640717000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 82: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[2/2b51af4b1ac0] 9766.640737000:2b51af4b1ac0: module of type 2 being loaded. 9766.640748000:2b51af4b1ac0: entry point 'isCompatibleWithFeature' not present in module 9766.640759000:2b51af4b1ac0: source file imgssapi.c requested reference for module 'lmtcpsrv', reference count now 1 9766.640771000:2b51af4b1ac0: source file imgssapi.c requested reference for module 'lmtcpsrv', reference count now 2 9766.640784000:2b51af4b1ac0: caller requested object 'gssutil', not found (iRet -3003) 9766.640794000:2b51af4b1ac0: Requested to load module 'lmgssutil' 9766.640805000:2b51af4b1ac0: loading module '/usr/local/lib/rsyslog/lmgssutil.so' 9766.640880000:2b51af4b1ac0: obj.c:1126:UseObj:invocation 100: WARNING: mutex still owned by us as we exit function, mutex: 0x65cf00[1/2b51af4b1ac0] 9766.640897000:2b51af4b1ac0: module of type 2 being loaded. 9766.640908000:2b51af4b1ac0: entry point 'isCompatibleWithFeature' not present in module 9766.640919000:2b51af4b1ac0: source file imgssapi.c requested reference for module 'lmgssutil', reference count now 1 9766.640933000:2b51af4b1ac0: source file imgssapi.c requested reference for module 'lmnetstrms', reference count now 3 9766.640946000:2b51af4b1ac0: source file imgssapi.c requested reference for module 'lmnet', reference count now 5 9766.640969000:2b51af4b1ac0: module of type 0 being loaded. 9766.640985000:2b51af4b1ac0: cfline: '$InputGSSServerPermitPlainTCP off' 9766.641000000:2b51af4b1ac0: cfline: '$InputGSSServerRun 10514' 9766.641027000:2b51af4b1ac0: Signal 11 (SIGSEGV) occured, execution must be terminated. 9766.641147000:2b51af4b1ac0: 9766.641175000:2b51af4b1ac0: Recorded Call Order for Thread '2b51af4b1ac0': 9766.641186000:2b51af4b1ac0: 0: syslogd.c:3097:realMain: 9766.641196000:2b51af4b1ac0: 1: syslogd.c:2749:mainThread: 9766.641206000:2b51af4b1ac0: 2: syslogd.c:2185:init: 9766.641219000:2b51af4b1ac0: 3: conf.c:410:processConfFile: 9766.641229000:2b51af4b1ac0: 4: conf.c:1179:cfline: 9766.641238000:2b51af4b1ac0: 5: conf.c:359:cfsysline: 9766.641249000:2b51af4b1ac0: 6: cfsysline.c:902:processCfSysLineCommand: 9766.641259000:2b51af4b1ac0: 7: cfsysline.c:673:cslchCallHdlr: 9766.641269000:2b51af4b1ac0: 8: cfsysline.c:501:doGetWord: 9766.641281000:2b51af4b1ac0: 9: imgssapi.c:310:addGSSListener: 9766.641292000:2b51af4b1ac0: 10: tcpsrv.c:135:configureTCPListen: 9766.641301000:2b51af4b1ac0: 11: tcpsrv.c:102:addNewLstnPort: 9766.641311000:2b51af4b1ac0: maximum number of nested calls for this thread: 26. 9766.641324000:2b51af4b1ac0: NOTE: not all calls may have been recorded, code does not currently guarantee that! 9766.641333000:2b51af4b1ac0: Mutex log for all known mutex operations: 9766.641343000:2b51af4b1ac0: If the call trace is empty, you may want to ./configure --enable-rtinst 9766.641353000:2b51af4b1ac0: To submit bug reports, visit http://www.rsyslog.com/bugs 9766.641363000:2b51af4b1ac0: To submit bug reports, visit http://www.rsyslog.com/bugs Aborted === I am a newbie, any idea? Thanks, Tomas Kubina From josesan311 at yahoo.com Thu Nov 26 05:23:11 2009 From: josesan311 at yahoo.com (Jose Sanchez) Date: Wed, 25 Nov 2009 20:23:11 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog Message-ID: <948839.16588.qm@web35605.mail.mud.yahoo.com> Hello, I've been using classic syslog for centralizing apache access logs from one server to a remote syslog server, the thing is syslog adds some nasty tags before the lines in the access logs and I cant get them off, ie: "Nov 25 21:25:37 server1 logger:" I would like to know if rsyslog has the option to filter this kind of stuff, I just want to have the logs sent to the syslog server exactly like I was saving them in a local access.log file. Thanks in advance. From rgerhards at hq.adiscon.com Thu Nov 26 09:11:01 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 26 Nov 2009 09:11:01 +0100 Subject: [rsyslog] filter logger tags from syslog References: <948839.16588.qm@web35605.mail.mud.yahoo.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034AC@GRFEXC.intern.adiscon.com> I don't have the specifics at hand, but you can surely do that. I would even assume that if you just write the %msg% property to file, the infomation should look good. If not, you need to fiddle a bit with the property replacer. The basic idea is to use $template line,"%msg%\n" *.* /path/to/log;line HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jose Sanchez > Sent: Thursday, November 26, 2009 5:23 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] filter logger tags from syslog > > Hello, > > I've been using classic syslog for centralizing apache access > logs from one server to a remote syslog server, the thing is > syslog adds some nasty tags before the lines in the access > logs and I cant get them off, ie: > > "Nov 25 21:25:37 server1 logger:" > > I would like to know if rsyslog has the option to filter this > kind of stuff, I just want to have the logs sent to the > syslog server exactly like I was saving them in a local > access.log file. > > Thanks in advance. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu Nov 26 09:21:44 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 26 Nov 2009 00:21:44 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <948839.16588.qm@web35605.mail.mud.yahoo.com> References: <948839.16588.qm@web35605.mail.mud.yahoo.com> Message-ID: On Wed, 25 Nov 2009, Jose Sanchez wrote: > Hello, > > I've been using classic syslog for centralizing apache access logs from one server to a remote syslog server, the thing is syslog adds some nasty tags before the lines in the access logs and I cant get them off, ie: > > "Nov 25 21:25:37 server1 logger:" > > I would like to know if rsyslog has the option to filter this kind of stuff, I just want to have the logs sent to the syslog server exactly like I was saving them in a local access.log file. > > Thanks in advance. 'logger:' is added by the logger program that apache is using to send the logs to syslog. a properly formatted syslog message will include a timestamp and what server it came from (note that the apache logs do _not_ tell you what virtual server the log comes from, it usually uses a different file for each log, so when you mix them into syslog you won't be able to tell them apart) so you have three basic options 1. let logger do it's default thing and then use a formatting command to strip off the 'syslogie' parts to get back to the apache default in the file 2. leave the 'syslogie' parts in when you write it to a file and have your analysis tool strip them out 3. reformat the apache log message so that you put useful information in the 'syslogie' parts of the message. you can move the timestamp to the beginning (you can do this with or without the timezone, the format obviously differs slightly) you can put the name of the virtual host in the server field you can replace 'logger:' with something like apache[80]: or apache[443]: I am going to be setting up something along the lines of #3 in the next few weeks. I figure I will also want to tinker with other things in the log message. there are items that apache can log, but does not log by default (I believe that how long it took to process the request is one of these), and also since syslog defaults to limiting log messages to 1-2K (depending on your impementation), there are some fields that I would want to move late in the message so that if they get very long they don't cause other fields to be lost due to truncation (URL and referrer fields can be several K long by themselves) David Lang From vanderleeden at logicunited.com Thu Nov 26 13:57:54 2009 From: vanderleeden at logicunited.com (Rudolf van der Leeden) Date: Thu, 26 Nov 2009 13:57:54 +0100 Subject: [rsyslog] Indefinite number of retries with ompgsql Message-ID: Hi, I'm currently testing ompgsql and running into a problem with the number of pgsql retries if the INSERT statement is wrong. My rsyslog.conf setting: $ModLoad ompgsql # load the output driver for PostgreSQL $ModLoad imudp # network reception $MaxMessageSize 8192 # default 2048; use 32768 to support large message sizes for IHE $UDPServerRun 514 # start a udp server at port 514 $WorkDirectory /var/spool/rsyslog/work # default location for work (spool) files $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName dbq # set file name, also enables disk mode $ActionResumeRetryCount 0 $template applogFormat,"INSERT INTO applog (message, host, priority, priotext, tag, logtime) VALUES \ (E'%msg%', '%hostname%', '%syslogpriority%', '%syslogpriority-text %', '%programname%', '%timegenerated:::date-pgsql%')",stdsql if $programname contains 'production' then :ompgsql:loghost,syslog,syslog,syslog;applogFormat I changed $ActionResumeRetryCount from -1 to 1 and then to 0, but always the same result: indefinite retries of rsyslogd to get the INSERT done. This setup is basically running OK. Only if a binary 0 (\000) is contained in the message Postgres returns: ERROR: invalid byte sequence for encoding "UTF8": 0x00 What am I doing wrong? Any hints are welcome. Thanks for your help, Rudolf. Rudolf VanderLeeden Scoreloop GmbH vanderleeden at scoreloop.com From rgerhards at hq.adiscon.com Thu Nov 26 17:47:26 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 26 Nov 2009 17:47:26 +0100 Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710346E@GRFEXC.intern.adiscon.com> References: <003001ca6092$792833d0$6b789b70$@com> <9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710346E@GRFEXC.intern.adiscon.com> Message-ID: <1259254046.16407.1.camel@rgf11> Hi all, I have now integrated the TLS part of the patch into all relevant versions. It is available in public git now. I will try to look into the parsing part tomorrow. Thanks again to Jonathan for sharing it! Rainer On Fri, 2009-11-20 at 11:03 +0100, Rainer Gerhards wrote: > I, too, think this is useful. Will look at it as soon as I get the patch. The > b) and c) parts may be a bit harder to integrate if they are not for the > newest devel branch, as I rewrote the parser part of rsyslog. But in any > case, it is worth trying to merge it in. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Thursday, November 19, 2009 9:38 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] [PATCH] TLS connection check + syslog TAG > > parsing > > > > for what it's worth, I think that b and c are very useful when dealing > > with strange data sources, and a is probably a bugfix. > > > > David Lang > > > > On Thu, 19 Nov 2009, Rainer Gerhards wrote: > > > > > > > > Sorry, I seem to have overlooked that message. But the mailing list > > processor > > > has stripped the attachment. Could you please re-send to my personal > > mail > > > address. > > > > > > Rainer > > > > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com > > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > >> Jonathan Bond-Caron > > >> Sent: Sunday, November 08, 2009 5:43 PM > > >> To: rsyslog at lists.adiscon.com > > >> Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing > > >> > > >> Hi all, > > >> > > >> > > >> > > >> Attached is a patch for 3 things: > > >> > > >> > > >> > > >> a) Check that TCP connection is still alive when using TLS > > >> > > >> > > >> > > >> b) Improve TAG parsing so that it ends when it sees a > > >> tab or escape > > >> control character. > > >> > > >> > > >> > > >> c) Added configuration directive: escapecontrolcharactertab > > >> > > >> In rsyslog.conf, you can set: > > >> > > >> $escapeControlCharactersOnReceive on > > >> > > >> $escapeControlCharacterTab off > > >> > > >> > > >> > > >> This avoids having a tab being escaped since it is after a viewable > > >> character J > > >> > > >> I'd prefer this being the default behavior but I've left > > >> $escapeControlCharacterTab on as default in case people > > >> expect tabs to be > > >> escaped. > > >> > > >> > > >> > > >> This is my first GIT patch, hope it works well ;) > > >> > > >> > > >> > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From josesan311 at yahoo.com Fri Nov 27 01:25:22 2009 From: josesan311 at yahoo.com (Jose Sanchez) Date: Thu, 26 Nov 2009 16:25:22 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: Message-ID: <886471.94508.qm@web35606.mail.mud.yahoo.com> Hello, I appreciate all the responses. Im not sure how can I can acconplish options 1) and 2) automatically. For option 3) the thing is I need "combined" log type so I cannot reform this. Im trying to centralize an access_log file from one server to the rsyslog server and I need to completely remove the tags I mentioned on my previous post. I have also tried using a perl script mentioned at the botton of this email, but it salso arriving with a tag, "apache_syslog:" as showed below, "apache_syslog: XXX.XXX.XXX.XXX - - [26/Nov/2009:18:23:02 -0600] \"GET /.." Basically, this log will be parsed by awstats which is pretty much stricted with the log format so that's why I need a clean log sent from the apache server to the rsyslog server. Thank you very much for all the help. Below is the Perl script: #!/usr/local/bin/perl # script: apache-access-logger use Sys::Syslog; $SERVER_NAME = shift || ''; $PRIORITY = 'info'; $FACILITY = 'local1'; Sys::Syslog::setlogsock('unix'); openlog ($SERVER_NAME,'ndelay', $FACILITY); while (<>) { chomp; syslog($PRIORITY,$_); } closelog; --- On Thu, 11/26/09, david at lang.hm wrote: > From: david at lang.hm > Subject: Re: [rsyslog] filter logger tags from syslog > To: "rsyslog-users" > Date: Thursday, November 26, 2009, 2:21 AM > On Wed, 25 Nov 2009, Jose Sanchez > wrote: > > > Hello, > > > > I've been using classic syslog for centralizing apache > access logs from one server to a remote syslog server, the > thing is syslog adds some nasty tags before the lines in the > access logs and I cant get them off, ie: > > > > "Nov 25 21:25:37 server1 logger:" > > > > I would like to know if rsyslog has the option to > filter this kind of stuff, I just want to have the logs sent > to the syslog server exactly like I was saving them in a > local access.log file. > > > > Thanks in advance. > > 'logger:' is added by the logger program that apache is > using to send the > logs to syslog. > > a properly formatted syslog message will include a > timestamp and what > server it came from (note that the apache logs do _not_ > tell you what > virtual server the log comes from, it usually uses a > different file for > each log, so when you mix them into syslog you won't be > able to tell them > apart) > > so you have three basic options > > 1. let logger do it's default thing and then use a > formatting command to > strip off the 'syslogie' parts to get back to the apache > default in the > file > > 2. leave the 'syslogie' parts in when you write it to a > file and have your > analysis tool strip them out > > 3. reformat the apache log message so that you put useful > information in > the 'syslogie' parts of the message. > > you can move the timestamp to the beginning (you can do > this with or > without the timezone, the format obviously differs > slightly) > > you can put the name of the virtual host in the server > field > > you can replace 'logger:' with something like apache[80]: > or apache[443]: > > I am going to be setting up something along the lines of #3 > in the next > few weeks. I figure I will also want to tinker with other > things in the > log message. there are items that apache can log, but does > not log by > default (I believe that how long it took to process the > request is one of > these), and also since syslog defaults to limiting log > messages to 1-2K > (depending on your impementation), there are some fields > that I would want > to move late in the message so that if they get very long > they don't cause > other fields to be lost due to truncation (URL and referrer > fields can be > several K long by themselves) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From josesan311 at yahoo.com Fri Nov 27 01:26:24 2009 From: josesan311 at yahoo.com (Jose Sanchez) Date: Thu, 26 Nov 2009 16:26:24 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71034AC@GRFEXC.intern.adiscon.com> Message-ID: <840944.77123.qm@web35608.mail.mud.yahoo.com> Hello, Will this work if Im using rsyslog just at the server side? (client is using syslog) Many thanks. --- On Thu, 11/26/09, Rainer Gerhards wrote: > From: Rainer Gerhards > Subject: Re: [rsyslog] filter logger tags from syslog > To: "rsyslog-users" > Date: Thursday, November 26, 2009, 2:11 AM > I don't have the specifics at hand, > but you can surely do that. I would even > assume that if you just write the %msg% property to file, > the infomation > should look good. If not, you need to fiddle a bit with the > property > replacer. > > The basic idea is to use > > $template line,"%msg%\n" > *.* /path/to/log;line > > HTH > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com > > > [mailto:rsyslog-bounces at lists.adiscon.com] > On Behalf Of Jose Sanchez > > Sent: Thursday, November 26, 2009 5:23 AM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] filter logger tags from syslog > > > > Hello, > > > > I've been using classic syslog for centralizing apache > access > > logs from one server to a remote syslog server, the > thing is > > syslog adds some nasty tags before the lines in the > access > > logs and I cant get them off, ie: > > > > "Nov 25 21:25:37 server1 logger:" > > > > I would like to know if rsyslog has the option to > filter this > > kind of stuff, I just want to have the logs sent to > the > > syslog server exactly like I was saving them in a > local > > access.log file. > > > > Thanks in advance. > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Nov 27 01:30:52 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 26 Nov 2009 16:30:52 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <840944.77123.qm@web35608.mail.mud.yahoo.com> References: <840944.77123.qm@web35608.mail.mud.yahoo.com> Message-ID: On Thu, 26 Nov 2009, Jose Sanchez wrote: > Will this work if Im using rsyslog just at the server side? (client is using syslog) yes, although depending on exactly which syslog you may need to tinker with the template. try what Rainer suggested below and see what you get out. David Lang > Many thanks. > > --- On Thu, 11/26/09, Rainer Gerhards wrote: > >> From: Rainer Gerhards >> Subject: Re: [rsyslog] filter logger tags from syslog >> To: "rsyslog-users" >> Date: Thursday, November 26, 2009, 2:11 AM >> I don't have the specifics at hand, >> but you can surely do that. I would even >> assume that if you just write the %msg% property to file, >> the infomation >> should look good. If not, you need to fiddle a bit with the >> property >> replacer. >> >> The basic idea is to use >> >> $template line,"%msg%\n" >> *.* /path/to/log;line >> >> HTH >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com >> >>> [mailto:rsyslog-bounces at lists.adiscon.com] >> On Behalf Of Jose Sanchez >>> Sent: Thursday, November 26, 2009 5:23 AM >>> To: rsyslog at lists.adiscon.com >>> Subject: [rsyslog] filter logger tags from syslog >>> >>> Hello, >>> >>> I've been using classic syslog for centralizing apache >> access >>> logs from one server to a remote syslog server, the >> thing is >>> syslog adds some nasty tags before the lines in the >> access >>> logs and I cant get them off, ie: >>> >>> "Nov 25 21:25:37 server1 logger:" >>> >>> I would like to know if rsyslog has the option to >> filter this >>> kind of stuff, I just want to have the logs sent to >> the >>> syslog server exactly like I was saving them in a >> local >>> access.log file. >>> >>> Thanks in advance. >>> >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Nov 27 01:38:03 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 26 Nov 2009 16:38:03 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <886471.94508.qm@web35606.mail.mud.yahoo.com> References: <886471.94508.qm@web35606.mail.mud.yahoo.com> Message-ID: On Thu, 26 Nov 2009, Jose Sanchez wrote: > Hello, > > I appreciate all the responses. > Im not sure how can I can acconplish options 1) and 2) automatically. > For option 3) the thing is I need "combined" log type so I cannot reform this. > > Im trying to centralize an access_log file from one server to the rsyslog server and I need to completely remove the tags I mentioned on my previous post. > I have also tried using a perl script mentioned at the botton of this email, but it salso arriving with a tag, "apache_syslog:" as showed below, > > "apache_syslog: XXX.XXX.XXX.XXX - - [26/Nov/2009:18:23:02 -0600] \"GET /.." > > Basically, this log will be parsed by awstats which is pretty much stricted with the log format so that's why I need a clean log sent from the apache server to the rsyslog server. don't forget that you need to filter these messages into a seperate file, otherwise you will have your apache combined log messages mixed with other syslog messages (which will really confuse awstats) option 1 is what Rainer suggested option 2 is to run the log through another step before awstats runs, something along the lines of cut -c 16- file |cut -f 3- -d ' ' |awstats the first cut removes the timestamp (always 15 characters, but with a variable number of spaces in it), the second cut removes the servername and the syslog tag ('logger:' in your first example) David Lang > Thank you very much for all the help. > > Below is the Perl script: > > #!/usr/local/bin/perl > # script: apache-access-logger > > use Sys::Syslog; > $SERVER_NAME = shift || ''; > > $PRIORITY = 'info'; > $FACILITY = 'local1'; > > Sys::Syslog::setlogsock('unix'); > openlog ($SERVER_NAME,'ndelay', $FACILITY); > > while (<>) { > chomp; > syslog($PRIORITY,$_); > } > closelog; > > --- On Thu, 11/26/09, david at lang.hm wrote: > >> From: david at lang.hm >> Subject: Re: [rsyslog] filter logger tags from syslog >> To: "rsyslog-users" >> Date: Thursday, November 26, 2009, 2:21 AM >> On Wed, 25 Nov 2009, Jose Sanchez >> wrote: >> >>> Hello, >>> >>> I've been using classic syslog for centralizing apache >> access logs from one server to a remote syslog server, the >> thing is syslog adds some nasty tags before the lines in the >> access logs and I cant get them off, ie: >>> >>> "Nov 25 21:25:37 server1 logger:" >>> >>> I would like to know if rsyslog has the option to >> filter this kind of stuff, I just want to have the logs sent >> to the syslog server exactly like I was saving them in a >> local access.log file. >>> >>> Thanks in advance. >> >> 'logger:' is added by the logger program that apache is >> using to send the >> logs to syslog. >> >> a properly formatted syslog message will include a >> timestamp and what >> server it came from (note that the apache logs do _not_ >> tell you what >> virtual server the log comes from, it usually uses a >> different file for >> each log, so when you mix them into syslog you won't be >> able to tell them >> apart) >> >> so you have three basic options >> >> 1. let logger do it's default thing and then use a >> formatting command to >> strip off the 'syslogie' parts to get back to the apache >> default in the >> file >> >> 2. leave the 'syslogie' parts in when you write it to a >> file and have your >> analysis tool strip them out >> >> 3. reformat the apache log message so that you put useful >> information in >> the 'syslogie' parts of the message. >> >> you can move the timestamp to the beginning (you can do >> this with or >> without the timezone, the format obviously differs >> slightly) >> >> you can put the name of the virtual host in the server >> field >> >> you can replace 'logger:' with something like apache[80]: >> or apache[443]: >> >> I am going to be setting up something along the lines of #3 >> in the next >> few weeks. I figure I will also want to tinker with other >> things in the >> log message. there are items that apache can log, but does >> not log by >> default (I believe that how long it took to process the >> request is one of >> these), and also since syslog defaults to limiting log >> messages to 1-2K >> (depending on your impementation), there are some fields >> that I would want >> to move late in the message so that if they get very long >> they don't cause >> other fields to be lost due to truncation (URL and referrer >> fields can be >> several K long by themselves) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From josesan311 at yahoo.com Fri Nov 27 05:13:00 2009 From: josesan311 at yahoo.com (Jose Sanchez) Date: Thu, 26 Nov 2009 20:13:00 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: Message-ID: <925071.2214.qm@web35604.mail.mud.yahoo.com> Hello, Thanks again for the great response. It's actually working! rsyslog is removing the "logger:" thing and all the nasty stuff from it automatically, how come? Is it because we are not adding any tag in the template? Im still not understanding how rsyslog removes the logger thing. Ok, Im now getting the proper output and like David said Im now getting issues with filtering the apache logs with all the rsyslog messages. I've tried to use the following filter but for some reason is not working and Im not 100% if this is the best solution to use, This is how I had set it up, $template line,"%msg%\n" if $msg contains 'GET' then /var/log/apache.test.log;line *.* /var/log/test.log;line Not sure if Im on the right path, any help will be appreciated. I have also tried the "if" sentence without specifying the template name. Many Thanks. --- On Thu, 11/26/09, david at lang.hm wrote: > From: david at lang.hm > Subject: Re: [rsyslog] filter logger tags from syslog > To: "rsyslog-users" > Date: Thursday, November 26, 2009, 6:38 PM > On Thu, 26 Nov 2009, Jose Sanchez > wrote: > > > Hello, > > > > I appreciate all the responses. > > Im not sure how can I can acconplish options 1) and 2) > automatically. > > For option 3) the thing is I need "combined" log type > so I cannot reform this. > > > > Im trying to centralize an access_log file from one > server to the rsyslog server and I need to completely remove > the tags I mentioned on my previous post. > > I have also tried using a perl script mentioned at the > botton of this email, but it salso arriving with a tag, > "apache_syslog:" as showed below, > > > > "apache_syslog: XXX.XXX.XXX.XXX - - > [26/Nov/2009:18:23:02 -0600] \"GET /.." > > > > Basically, this log will be parsed by awstats which is > pretty much stricted with the log format so that's why I > need a clean log sent from the apache server to the rsyslog > server. > > don't forget that you need to filter these messages into a > seperate file, > otherwise you will have your apache combined log messages > mixed with other > syslog messages (which will really confuse awstats) > > option 1 is what Rainer suggested > > option 2 is to run the log through another step before > awstats runs, > something along the lines of > > cut -c 16- file |cut -f 3- -d ' ' |awstats > > the first cut removes the timestamp (always 15 characters, > but with a > variable number of spaces in it), the second cut removes > the servername > and the syslog tag ('logger:' in your first example) > > David Lang > > > Thank you very much for all the help. > > > > Below is the Perl script: > > > > #!/usr/local/bin/perl > > # script: apache-access-logger > > > > use Sys::Syslog; > > $SERVER_NAME = shift || ''; > > > > $PRIORITY = 'info'; > > $FACILITY = 'local1'; > > > > Sys::Syslog::setlogsock('unix'); > > openlog ($SERVER_NAME,'ndelay', $FACILITY); > > > > while (<>) { > >? chomp; > >? syslog($PRIORITY,$_); > > } > > closelog; > > > > --- On Thu, 11/26/09, david at lang.hm > wrote: > > > >> From: david at lang.hm > >> Subject: Re: [rsyslog] filter logger tags from > syslog > >> To: "rsyslog-users" > >> Date: Thursday, November 26, 2009, 2:21 AM > >> On Wed, 25 Nov 2009, Jose Sanchez > >> wrote: > >> > >>> Hello, > >>> > >>> I've been using classic syslog for > centralizing apache > >> access logs from one server to a remote syslog > server, the > >> thing is syslog adds some nasty tags before the > lines in the > >> access logs and I cant get them off, ie: > >>> > >>> "Nov 25 21:25:37 server1 logger:" > >>> > >>> I would like to know if rsyslog has the option > to > >> filter this kind of stuff, I just want to have the > logs sent > >> to the syslog server exactly like I was saving > them in a > >> local access.log file. > >>> > >>> Thanks in advance. > >> > >> 'logger:' is added by the logger program that > apache is > >> using to send the > >> logs to syslog. > >> > >> a properly formatted syslog message will include > a > >> timestamp and what > >> server it came from (note that the apache logs do > _not_ > >> tell you what > >> virtual server the log comes from, it usually uses > a > >> different file for > >> each log, so when you mix them into syslog you > won't be > >> able to tell them > >> apart) > >> > >> so you have three basic options > >> > >> 1. let logger do it's default thing and then use > a > >> formatting command to > >> strip off the 'syslogie' parts to get back to the > apache > >> default in the > >> file > >> > >> 2. leave the 'syslogie' parts in when you write it > to a > >> file and have your > >> analysis tool strip them out > >> > >> 3. reformat the apache log message so that you put > useful > >> information in > >> the 'syslogie' parts of the message. > >> > >> you can move the timestamp to the beginning (you > can do > >> this with or > >> without the timezone, the format obviously > differs > >> slightly) > >> > >> you can put the name of the virtual host in the > server > >> field > >> > >> you can replace 'logger:' with something like > apache[80]: > >> or apache[443]: > >> > >> I am going to be setting up something along the > lines of #3 > >> in the next > >> few weeks. I figure I will also want to tinker > with other > >> things in the > >> log message. there are items that apache can log, > but does > >> not log by > >> default (I believe that how long it took to > process the > >> request is one of > >> these), and also since syslog defaults to limiting > log > >> messages to 1-2K > >> (depending on your impementation), there are some > fields > >> that I would want > >> to move late in the message so that if they get > very long > >> they don't cause > >> other fields to be lost due to truncation (URL and > referrer > >> fields can be > >> several K long by themselves) > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Nov 27 09:41:13 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 27 Nov 2009 09:41:13 +0100 Subject: [rsyslog] filter logger tags from syslog References: <925071.2214.qm@web35604.mail.mud.yahoo.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034C3@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Jose Sanchez > Sent: Friday, November 27, 2009 5:13 AM > To: rsyslog-users > Subject: Re: [rsyslog] filter logger tags from syslog > > Hello, > > Thanks again for the great response. > It's actually working! rsyslog is removing the "logger:" thing and all > the nasty stuff from it automatically, how come? you really need to read through the property doc. I know the doc is not great, but with some persistence you'll find anything you need (and doc contributions are always very welcome!). Understanding properties is key to getting complex configurations right. > Is it because we are > not adding any tag in the template? Yes, because the template specifies which properties are used (and how). > Im still not understanding how > rsyslog removes the logger thing. > > Ok, Im now getting the proper output and like David said Im now getting > issues with filtering the apache logs with all the rsyslog messages. > I've tried to use the following filter but for some reason is not > working and Im not 100% if this is the best solution to use, > > This is how I had set it up, > > $template line,"%msg%\n" > if $msg contains 'GET' then /var/log/apache.test.log;line > *.* /var/log/test.log;line If you want to split the messages, you need to discard the one that you just wrote: if $msg contains 'GET' then /var/log/apache.test.log;line & ~ Helpful read is http://www.rsyslog.com/doc-queues_analogy.html HTH Rainer > > Not sure if Im on the right path, any help will be appreciated. > I have also tried the "if" sentence without specifying the template > name. > > Many Thanks. > > --- On Thu, 11/26/09, david at lang.hm wrote: > > > From: david at lang.hm > > Subject: Re: [rsyslog] filter logger tags from syslog > > To: "rsyslog-users" > > Date: Thursday, November 26, 2009, 6:38 PM > > On Thu, 26 Nov 2009, Jose Sanchez > > wrote: > > > > > Hello, > > > > > > I appreciate all the responses. > > > Im not sure how can I can acconplish options 1) and 2) > > automatically. > > > For option 3) the thing is I need "combined" log type > > so I cannot reform this. > > > > > > Im trying to centralize an access_log file from one > > server to the rsyslog server and I need to completely remove > > the tags I mentioned on my previous post. > > > I have also tried using a perl script mentioned at the > > botton of this email, but it salso arriving with a tag, > > "apache_syslog:" as showed below, > > > > > > "apache_syslog: XXX.XXX.XXX.XXX - - > > [26/Nov/2009:18:23:02 -0600] \"GET /.." > > > > > > Basically, this log will be parsed by awstats which is > > pretty much stricted with the log format so that's why I > > need a clean log sent from the apache server to the rsyslog > > server. > > > > don't forget that you need to filter these messages into a > > seperate file, > > otherwise you will have your apache combined log messages > > mixed with other > > syslog messages (which will really confuse awstats) > > > > option 1 is what Rainer suggested > > > > option 2 is to run the log through another step before > > awstats runs, > > something along the lines of > > > > cut -c 16- file |cut -f 3- -d ' ' |awstats > > > > the first cut removes the timestamp (always 15 characters, > > but with a > > variable number of spaces in it), the second cut removes > > the servername > > and the syslog tag ('logger:' in your first example) > > > > David Lang > > > > > Thank you very much for all the help. > > > > > > Below is the Perl script: > > > > > > #!/usr/local/bin/perl > > > # script: apache-access-logger > > > > > > use Sys::Syslog; > > > $SERVER_NAME = shift || ''; > > > > > > $PRIORITY = 'info'; > > > $FACILITY = 'local1'; > > > > > > Sys::Syslog::setlogsock('unix'); > > > openlog ($SERVER_NAME,'ndelay', $FACILITY); > > > > > > while (<>) { > > >? chomp; > > >? syslog($PRIORITY,$_); > > > } > > > closelog; > > > > > > --- On Thu, 11/26/09, david at lang.hm > > wrote: > > > > > >> From: david at lang.hm > > >> Subject: Re: [rsyslog] filter logger tags from > > syslog > > >> To: "rsyslog-users" > > >> Date: Thursday, November 26, 2009, 2:21 AM > > >> On Wed, 25 Nov 2009, Jose Sanchez > > >> wrote: > > >> > > >>> Hello, > > >>> > > >>> I've been using classic syslog for > > centralizing apache > > >> access logs from one server to a remote syslog > > server, the > > >> thing is syslog adds some nasty tags before the > > lines in the > > >> access logs and I cant get them off, ie: > > >>> > > >>> "Nov 25 21:25:37 server1 logger:" > > >>> > > >>> I would like to know if rsyslog has the option > > to > > >> filter this kind of stuff, I just want to have the > > logs sent > > >> to the syslog server exactly like I was saving > > them in a > > >> local access.log file. > > >>> > > >>> Thanks in advance. > > >> > > >> 'logger:' is added by the logger program that > > apache is > > >> using to send the > > >> logs to syslog. > > >> > > >> a properly formatted syslog message will include > > a > > >> timestamp and what > > >> server it came from (note that the apache logs do > > _not_ > > >> tell you what > > >> virtual server the log comes from, it usually uses > > a > > >> different file for > > >> each log, so when you mix them into syslog you > > won't be > > >> able to tell them > > >> apart) > > >> > > >> so you have three basic options > > >> > > >> 1. let logger do it's default thing and then use > > a > > >> formatting command to > > >> strip off the 'syslogie' parts to get back to the > > apache > > >> default in the > > >> file > > >> > > >> 2. leave the 'syslogie' parts in when you write it > > to a > > >> file and have your > > >> analysis tool strip them out > > >> > > >> 3. reformat the apache log message so that you put > > useful > > >> information in > > >> the 'syslogie' parts of the message. > > >> > > >> you can move the timestamp to the beginning (you > > can do > > >> this with or > > >> without the timezone, the format obviously > > differs > > >> slightly) > > >> > > >> you can put the name of the virtual host in the > > server > > >> field > > >> > > >> you can replace 'logger:' with something like > > apache[80]: > > >> or apache[443]: > > >> > > >> I am going to be setting up something along the > > lines of #3 > > >> in the next > > >> few weeks. I figure I will also want to tinker > > with other > > >> things in the > > >> log message. there are items that apache can log, > > but does > > >> not log by > > >> default (I believe that how long it took to > > process the > > >> request is one of > > >> these), and also since syslog defaults to limiting > > log > > >> messages to 1-2K > > >> (depending on your impementation), there are some > > fields > > >> that I would want > > >> to move late in the message so that if they get > > very long > > >> they don't cause > > >> other fields to be lost due to truncation (URL and > > referrer > > >> fields can be > > >> several K long by themselves) > > >> > > >> David Lang > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Nov 27 09:49:35 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 27 Nov 2009 00:49:35 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <925071.2214.qm@web35604.mail.mud.yahoo.com> References: <925071.2214.qm@web35604.mail.mud.yahoo.com> Message-ID: On Thu, 26 Nov 2009, Jose Sanchez wrote: > Hello, > > Thanks again for the great response. > It's actually working! rsyslog is removing the "logger:" thing and all > the nasty stuff from it automatically, how come? Is it because we are > not adding any tag in the template? Im still not understanding how > rsyslog removes the logger thing. > > Ok, Im now getting the proper output and like David said Im now getting > issues with filtering the apache logs with all the rsyslog messages. > I've tried to use the following filter but for some reason is not > working and Im not 100% if this is the best solution to use, > > This is how I had set it up, > > $template line,"%msg%\n" > if $msg contains 'GET' then /var/log/apache.test.log;line > *.* /var/log/test.log;line > > Not sure if Im on the right path, any help will be appreciated. > I have also tried the "if" sentence without specifying the template name. when rsyslog receives a message it parses it. the message over the wire hasn't changed (still has the timestamp, servername, logger: etc), but rsyslog puts those parts into the seperate variables and puts what is left of the message into the %msg% variable. so when you change the template from the default of %timestamp% %hostname% %syslogtag%%msg% to just %msg% the log file has just the part you care about. now for the filtering. you could do :%programname, isequal, "logger" /var/log/apache.test.log;line (as I understand it, this format is a bit more efficiant for rsyslog than the equivalent of if $programname eq "logger" then /var/log/apache.test.log;line ) I would actually suggest that you use the perl script that you posted, and filter for programname equal to "apache_syslog", filtering on just 'logger' means that you can't use logger for anything else. you don't want to just filter for 'GET' as there are a bunch of log files that won't have GET in them David Lang > Many Thanks. > > --- On Thu, 11/26/09, david at lang.hm wrote: > >> From: david at lang.hm >> Subject: Re: [rsyslog] filter logger tags from syslog >> To: "rsyslog-users" >> Date: Thursday, November 26, 2009, 6:38 PM >> On Thu, 26 Nov 2009, Jose Sanchez >> wrote: >> >>> Hello, >>> >>> I appreciate all the responses. >>> Im not sure how can I can acconplish options 1) and 2) >> automatically. >>> For option 3) the thing is I need "combined" log type >> so I cannot reform this. >>> >>> Im trying to centralize an access_log file from one >> server to the rsyslog server and I need to completely remove >> the tags I mentioned on my previous post. >>> I have also tried using a perl script mentioned at the >> botton of this email, but it salso arriving with a tag, >> "apache_syslog:" as showed below, >>> >>> "apache_syslog: XXX.XXX.XXX.XXX - - >> [26/Nov/2009:18:23:02 -0600] \"GET /.." >>> >>> Basically, this log will be parsed by awstats which is >> pretty much stricted with the log format so that's why I >> need a clean log sent from the apache server to the rsyslog >> server. >> >> don't forget that you need to filter these messages into a >> seperate file, >> otherwise you will have your apache combined log messages >> mixed with other >> syslog messages (which will really confuse awstats) >> >> option 1 is what Rainer suggested >> >> option 2 is to run the log through another step before >> awstats runs, >> something along the lines of >> >> cut -c 16- file |cut -f 3- -d ' ' |awstats >> >> the first cut removes the timestamp (always 15 characters, >> but with a >> variable number of spaces in it), the second cut removes >> the servername >> and the syslog tag ('logger:' in your first example) >> >> David Lang >> >>> Thank you very much for all the help. >>> >>> Below is the Perl script: >>> >>> #!/usr/local/bin/perl >>> # script: apache-access-logger >>> >>> use Sys::Syslog; >>> $SERVER_NAME = shift || ''; >>> >>> $PRIORITY = 'info'; >>> $FACILITY = 'local1'; >>> >>> Sys::Syslog::setlogsock('unix'); >>> openlog ($SERVER_NAME,'ndelay', $FACILITY); >>> >>> while (<>) { >>> ? chomp; >>> ? syslog($PRIORITY,$_); >>> } >>> closelog; >>> >>> --- On Thu, 11/26/09, david at lang.hm >> wrote: >>> >>>> From: david at lang.hm >>>> Subject: Re: [rsyslog] filter logger tags from >> syslog >>>> To: "rsyslog-users" >>>> Date: Thursday, November 26, 2009, 2:21 AM >>>> On Wed, 25 Nov 2009, Jose Sanchez >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> I've been using classic syslog for >> centralizing apache >>>> access logs from one server to a remote syslog >> server, the >>>> thing is syslog adds some nasty tags before the >> lines in the >>>> access logs and I cant get them off, ie: >>>>> >>>>> "Nov 25 21:25:37 server1 logger:" >>>>> >>>>> I would like to know if rsyslog has the option >> to >>>> filter this kind of stuff, I just want to have the >> logs sent >>>> to the syslog server exactly like I was saving >> them in a >>>> local access.log file. >>>>> >>>>> Thanks in advance. >>>> >>>> 'logger:' is added by the logger program that >> apache is >>>> using to send the >>>> logs to syslog. >>>> >>>> a properly formatted syslog message will include >> a >>>> timestamp and what >>>> server it came from (note that the apache logs do >> _not_ >>>> tell you what >>>> virtual server the log comes from, it usually uses >> a >>>> different file for >>>> each log, so when you mix them into syslog you >> won't be >>>> able to tell them >>>> apart) >>>> >>>> so you have three basic options >>>> >>>> 1. let logger do it's default thing and then use >> a >>>> formatting command to >>>> strip off the 'syslogie' parts to get back to the >> apache >>>> default in the >>>> file >>>> >>>> 2. leave the 'syslogie' parts in when you write it >> to a >>>> file and have your >>>> analysis tool strip them out >>>> >>>> 3. reformat the apache log message so that you put >> useful >>>> information in >>>> the 'syslogie' parts of the message. >>>> >>>> you can move the timestamp to the beginning (you >> can do >>>> this with or >>>> without the timezone, the format obviously >> differs >>>> slightly) >>>> >>>> you can put the name of the virtual host in the >> server >>>> field >>>> >>>> you can replace 'logger:' with something like >> apache[80]: >>>> or apache[443]: >>>> >>>> I am going to be setting up something along the >> lines of #3 >>>> in the next >>>> few weeks. I figure I will also want to tinker >> with other >>>> things in the >>>> log message. there are items that apache can log, >> but does >>>> not log by >>>> default (I believe that how long it took to >> process the >>>> request is one of >>>> these), and also since syslog defaults to limiting >> log >>>> messages to 1-2K >>>> (depending on your impementation), there are some >> fields >>>> that I would want >>>> to move late in the message so that if they get >> very long >>>> they don't cause >>>> other fields to be lost due to truncation (URL and >> referrer >>>> fields can be >>>> several K long by themselves) >>>> >>>> David Lang >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Nov 27 11:46:03 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 27 Nov 2009 11:46:03 +0100 Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710346E@GRFEXC.intern.adiscon.com> References: <003001ca6092$792833d0$6b789b70$@com> <9B6E2A8877C38245BFB15CC491A11DA710345B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710346E@GRFEXC.intern.adiscon.com> Message-ID: <1259318763.15106.1.camel@rgf11> I have now implemented the complete patch. For v5, I needed to rewrite it, but that wasn't much of an issue once I knew what was required ;) v5 now even has an automatted test for this functionality. Thanks again, Rainer On Fri, 2009-11-20 at 11:03 +0100, Rainer Gerhards wrote: > I, too, think this is useful. Will look at it as soon as I get the patch. The > b) and c) parts may be a bit harder to integrate if they are not for the > newest devel branch, as I rewrote the parser part of rsyslog. But in any > case, it is worth trying to merge it in. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Thursday, November 19, 2009 9:38 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] [PATCH] TLS connection check + syslog TAG > > parsing > > > > for what it's worth, I think that b and c are very useful when dealing > > with strange data sources, and a is probably a bugfix. > > > > David Lang > > > > On Thu, 19 Nov 2009, Rainer Gerhards wrote: > > > > > > > > Sorry, I seem to have overlooked that message. But the mailing list > > processor > > > has stripped the attachment. Could you please re-send to my personal > > mail > > > address. > > > > > > Rainer > > > > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com > > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > >> Jonathan Bond-Caron > > >> Sent: Sunday, November 08, 2009 5:43 PM > > >> To: rsyslog at lists.adiscon.com > > >> Subject: [rsyslog] [PATCH] TLS connection check + syslog TAG parsing > > >> > > >> Hi all, > > >> > > >> > > >> > > >> Attached is a patch for 3 things: > > >> > > >> > > >> > > >> a) Check that TCP connection is still alive when using TLS > > >> > > >> > > >> > > >> b) Improve TAG parsing so that it ends when it sees a > > >> tab or escape > > >> control character. > > >> > > >> > > >> > > >> c) Added configuration directive: escapecontrolcharactertab > > >> > > >> In rsyslog.conf, you can set: > > >> > > >> $escapeControlCharactersOnReceive on > > >> > > >> $escapeControlCharacterTab off > > >> > > >> > > >> > > >> This avoids having a tab being escaped since it is after a viewable > > >> character J > > >> > > >> I'd prefer this being the default behavior but I've left > > >> $escapeControlCharacterTab on as default in case people > > >> expect tabs to be > > >> escaped. > > >> > > >> > > >> > > >> This is my first GIT patch, hope it works well ;) > > >> > > >> > > >> > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From tbergfeld at hq.adiscon.com Fri Nov 27 12:08:17 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 27 Nov 2009 12:08:17 +0100 Subject: [rsyslog] rsyslog 5.5.1 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034CC@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.5.1, a member of the v5-devel branch. The primary new feature is that the epoll() interface is now supported for plain tcp syslog. This results in better performance and also removes the upper bound of connections in select() in a portable way. The select() interface is still supported for platforms that do not provide epoll(). There are also some important fixes. It is a recommended update for all users of the v5-beta. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article433.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-190.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From josesan311 at yahoo.com Sat Nov 28 01:42:02 2009 From: josesan311 at yahoo.com (Jose Sanchez) Date: Fri, 27 Nov 2009 16:42:02 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: Message-ID: <472628.6567.qm@web35606.mail.mud.yahoo.com> Hello David and Reiner, First I would like to thank you for all the help offered, I was able to setup almost everything because of you guys. I had some issues today, though. I found that rsyslog was removing the "logger" properly but it was adding an extra empty space not sure why so I had to cut if off (by watching how to do it on video tutorial first!) by modifying the template that David gave me, I currently have it like this, $template line,"%msg:2:1000%\n" The thing here is Im not sure if this is a reliable solution, I couldnt find if there is any setting that will tell rsyslog to simply remove the empty space or to get everything until the last letter so I configured a very long (1000) number in case rsyslog cuts some part of the text. Not sure if there is any negative impact on doing it this way, if there is any other better way, please let me know. Regarding David's reply, > I would actually suggest that you use the perl script that > you posted, and > filter for programname equal to "apache_syslog", filtering > on just > 'logger' means that you can't use logger for anything > else. I ended using logger because with logger I can specify tags when sending the logs and since I have multiple vhosts and two webserver nodes I could identify from which webserver is coming the logs from, ie: :programname, isequal, "node1.www.domain.com" /var/log/httpd/node1/www.domain.com.log;line :programname, isequal, "node1.blog.domain.com" /var/log/httpd/node1/blog.domain.com.log;line :programname, isequal, "node2.www.domain.com" /var/log/httpd/node2/www.domain.com.log;line :programname, isequal, "node2.blog.domain.com" /var/log/httpd/node2/blog.domain.com.log;line I spent like 3 hours adding those lines due the amount of vhosts I have anyway this created every log and started delivering every vhost for each node on their specific log file just fine. Same thing as above Im still not really sure if this is the most realiable way of doing this with rsyslog. I hope this is the right path for doing such a big thing. Thank you again for everything. --- On Fri, 11/27/09, david at lang.hm wrote: > From: david at lang.hm > Subject: Re: [rsyslog] filter logger tags from syslog > To: "rsyslog-users" > Date: Friday, November 27, 2009, 2:49 AM > On Thu, 26 Nov 2009, Jose Sanchez > wrote: > > > Hello, > > > > Thanks again for the great response. > > It's actually working! rsyslog is removing the > "logger:" thing and all > > the nasty stuff from it automatically, how come? Is it > because we are > > not adding any tag in the template? Im still not > understanding how > > rsyslog removes the logger thing. > > > > Ok, Im now getting the proper output and like David > said Im now getting > > issues with filtering the apache logs with all the > rsyslog messages. > > > I've tried to use the following filter but for some > reason is not > > working and Im not 100% if this is the best solution > to use, > > > > This is how I had set it up, > > > > $template line,"%msg%\n" > > if $msg contains 'GET' then > /var/log/apache.test.log;line > > *.* /var/log/test.log;line > > > > Not sure if Im on the right path, any help will be > appreciated. > > I have also tried the "if" sentence without specifying > the template name. > > > when rsyslog receives a message it parses it. the message > over the wire > hasn't changed (still has the timestamp, servername, > logger: etc), but > rsyslog puts those parts into the seperate variables and > puts what is left > of the message into the %msg% variable. > > so when you change the template from the default of > %timestamp% %hostname% %syslogtag%%msg% > to just > %msg% > > the log file has just the part you care about. > > now for the filtering. > > you could do > :%programname, isequal, "logger" > /var/log/apache.test.log;line > > (as I understand it, this format is a bit more efficiant > for rsyslog than > the equivalent of > if $programname eq "logger" then > /var/log/apache.test.log;line > ) > > I would actually suggest that you use the perl script that > you posted, and > filter for programname equal to "apache_syslog", filtering > on just > 'logger' means that you can't use logger for anything > else. > > you don't want to just filter for 'GET' as there are a > bunch of log files > that won't have GET in them > > David Lang > > > > Many Thanks. > > > > --- On Thu, 11/26/09, david at lang.hm > wrote: > > > >> From: david at lang.hm > >> Subject: Re: [rsyslog] filter logger tags from > syslog > >> To: "rsyslog-users" > >> Date: Thursday, November 26, 2009, 6:38 PM > >> On Thu, 26 Nov 2009, Jose Sanchez > >> wrote: > >> > >>> Hello, > >>> > >>> I appreciate all the responses. > >>> Im not sure how can I can acconplish options > 1) and 2) > >> automatically. > >>> For option 3) the thing is I need "combined" > log type > >> so I cannot reform this. > >>> > >>> Im trying to centralize an access_log file > from one > >> server to the rsyslog server and I need to > completely remove > >> the tags I mentioned on my previous post. > >>> I have also tried using a perl script > mentioned at the > >> botton of this email, but it salso arriving with a > tag, > >> "apache_syslog:" as showed below, > >>> > >>> "apache_syslog: XXX.XXX.XXX.XXX - - > >> [26/Nov/2009:18:23:02 -0600] \"GET /.." > >>> > >>> Basically, this log will be parsed by awstats > which is > >> pretty much stricted with the log format so that's > why I > >> need a clean log sent from the apache server to > the rsyslog > >> server. > >> > >> don't forget that you need to filter these > messages into a > >> seperate file, > >> otherwise you will have your apache combined log > messages > >> mixed with other > >> syslog messages (which will really confuse > awstats) > >> > >> option 1 is what Rainer suggested > >> > >> option 2 is to run the log through another step > before > >> awstats runs, > >> something along the lines of > >> > >> cut -c 16- file |cut -f 3- -d ' ' |awstats > >> > >> the first cut removes the timestamp (always 15 > characters, > >> but with a > >> variable number of spaces in it), the second cut > removes > >> the servername > >> and the syslog tag ('logger:' in your first > example) > >> > >> David Lang > >> > >>> Thank you very much for all the help. > >>> > >>> Below is the Perl script: > >>> > >>> #!/usr/local/bin/perl > >>> # script: apache-access-logger > >>> > >>> use Sys::Syslog; > >>> $SERVER_NAME = shift || ''; > >>> > >>> $PRIORITY = 'info'; > >>> $FACILITY = 'local1'; > >>> > >>> Sys::Syslog::setlogsock('unix'); > >>> openlog ($SERVER_NAME,'ndelay', $FACILITY); > >>> > >>> while (<>) { > >>> ? chomp; > >>> ? syslog($PRIORITY,$_); > >>> } > >>> closelog; > >>> > >>> --- On Thu, 11/26/09, david at lang.hm > >> wrote: > >>> > >>>> From: david at lang.hm > >>>> Subject: Re: [rsyslog] filter logger tags > from > >> syslog > >>>> To: "rsyslog-users" > >>>> Date: Thursday, November 26, 2009, 2:21 > AM > >>>> On Wed, 25 Nov 2009, Jose Sanchez > >>>> wrote: > >>>> > >>>>> Hello, > >>>>> > >>>>> I've been using classic syslog for > >> centralizing apache > >>>> access logs from one server to a remote > syslog > >> server, the > >>>> thing is syslog adds some nasty tags > before the > >> lines in the > >>>> access logs and I cant get them off, ie: > >>>>> > >>>>> "Nov 25 21:25:37 server1 logger:" > >>>>> > >>>>> I would like to know if rsyslog has > the option > >> to > >>>> filter this kind of stuff, I just want to > have the > >> logs sent > >>>> to the syslog server exactly like I was > saving > >> them in a > >>>> local access.log file. > >>>>> > >>>>> Thanks in advance. > >>>> > >>>> 'logger:' is added by the logger program > that > >> apache is > >>>> using to send the > >>>> logs to syslog. > >>>> > >>>> a properly formatted syslog message will > include > >> a > >>>> timestamp and what > >>>> server it came from (note that the apache > logs do > >> _not_ > >>>> tell you what > >>>> virtual server the log comes from, it > usually uses > >> a > >>>> different file for > >>>> each log, so when you mix them into syslog > you > >> won't be > >>>> able to tell them > >>>> apart) > >>>> > >>>> so you have three basic options > >>>> > >>>> 1. let logger do it's default thing and > then use > >> a > >>>> formatting command to > >>>> strip off the 'syslogie' parts to get back > to the > >> apache > >>>> default in the > >>>> file > >>>> > >>>> 2. leave the 'syslogie' parts in when you > write it > >> to a > >>>> file and have your > >>>> analysis tool strip them out > >>>> > >>>> 3. reformat the apache log message so that > you put > >> useful > >>>> information in > >>>> the 'syslogie' parts of the message. > >>>> > >>>> you can move the timestamp to the > beginning (you > >> can do > >>>> this with or > >>>> without the timezone, the format > obviously > >> differs > >>>> slightly) > >>>> > >>>> you can put the name of the virtual host > in the > >> server > >>>> field > >>>> > >>>> you can replace 'logger:' with something > like > >> apache[80]: > >>>> or apache[443]: > >>>> > >>>> I am going to be setting up something > along the > >> lines of #3 > >>>> in the next > >>>> few weeks. I figure I will also want to > tinker > >> with other > >>>> things in the > >>>> log message. there are items that apache > can log, > >> but does > >>>> not log by > >>>> default (I believe that how long it took > to > >> process the > >>>> request is one of > >>>> these), and also since syslog defaults to > limiting > >> log > >>>> messages to 1-2K > >>>> (depending on your impementation), there > are some > >> fields > >>>> that I would want > >>>> to move late in the message so that if > they get > >> very long > >>>> they don't cause > >>>> other fields to be lost due to truncation > (URL and > >> referrer > >>>> fields can be > >>>> several K long by themselves) > >>>> > >>>> David Lang > >>>> > _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> > _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > -----Inline Attachment Follows----- > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From ryan.b.lynch at gmail.com Sat Nov 28 05:45:42 2009 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Fri, 27 Nov 2009 23:45:42 -0500 Subject: [rsyslog] Auto-creating directories, when using templates and the property replacer. Message-ID: <115906d10911272045v237eada0n96b204fcaf1fa6d4@mail.gmail.com> I recently migrated a few Fedora and CentOS hosts from Rsyslog 4.x to 5.x. Several of my systems use output file path templates like this, with multiple dynamically-named directory components: `template DefaultOutputPath,"/var/log/rsyslog/%hostname:::secpath-replace%/%syslogfacility-text:::secpath-replace%/%programname:::secpath-replace%/%$year%/%$month%/%$day%/%$hour%/%$minute%/_.log"` Under 4.x, Rsyslog will automatically create any directories that don't already exist, as needed. But under 5.x, it just fails silently if any of the directories in the path don't exist. Is there a configuration setting that governs this in 5.x, or some way to get back the old automatic directory creation behavior? -Ryan Ryan B. Lynch ryan.b.lynch at gmail.com From rgerhards at hq.adiscon.com Sat Nov 28 09:52:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 28 Nov 2009 09:52:12 +0100 Subject: [rsyslog] Auto-creating directories, when using templates and the property replacer. Message-ID: <000101ca7008$230eb68e$100013ac@intern.adiscon.com> That sounds like a bug, will look into it hopefully early next week. Thanks for reporting! Oh: which version do you use? 5.2.0? Then you should try the recent beta first... rainer ----- Urspr?ngliche Nachricht ----- Von: "Ryan Lynch" An: "rsyslog at lists.adiscon.com" Gesendet: 28.11.09 05:46 Betreff: [rsyslog] Auto-creating directories,when using templates and the property replacer. I recently migrated a few Fedora and CentOS hosts from Rsyslog 4.x to 5.x. Several of my systems use output file path templates like this, with multiple dynamically-named directory components: `template DefaultOutputPath,"/var/log/rsyslog/%hostname:::secpath-replace%/%syslogfacility-text:::secpath-replace%/%programname:::secpath-replace%/%$year%/%$month%/%$day%/%$hour%/%$minute%/_.log"` Under 4.x, Rsyslog will automatically create any directories that don't already exist, as needed. But under 5.x, it just fails silently if any of the directories in the path don't exist. Is there a configuration setting that governs this in 5.x, or some way to get back the old automatic directory creation behavior? -Ryan Ryan B. Lynch ryan.b.lynch at gmail.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Sat Nov 28 10:43:03 2009 From: david at lang.hm (david at lang.hm) Date: Sat, 28 Nov 2009 01:43:03 -0800 (PST) Subject: [rsyslog] filter logger tags from syslog In-Reply-To: <472628.6567.qm@web35606.mail.mud.yahoo.com> References: <472628.6567.qm@web35606.mail.mud.yahoo.com> Message-ID: On Fri, 27 Nov 2009, Jose Sanchez wrote: > Hello David and Reiner, > > First I would like to thank you for all the help offered, I was able to setup almost everything because of you guys. > > I had some issues today, though. I found that rsyslog was removing the "logger" properly but it was adding an extra empty space not sure why so I had to cut if off (by watching how to do it on video tutorial first!) by modifying the template that David gave me, I currently have it like this, > > $template line,"%msg:2:1000%\n" > > The thing here is Im not sure if this is a reliable solution, I couldnt find if there is any setting that will tell rsyslog to simply remove the empty space or to get everything until the last letter so I configured a very long (1000) number in case rsyslog cuts some part of the text. Not sure if there is any negative impact on doing it this way, if there is any other better way, please let me know. if you use '$' instead of '1000' it will go to the end of the message, no matter how long it is (1000 is not long enough for some messages) I think that what you are doing is probably the best way to deal with this space. David Lang From ryan.b.lynch at gmail.com Sat Nov 28 11:08:17 2009 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Sat, 28 Nov 2009 05:08:17 -0500 Subject: [rsyslog] Auto-creating directories, when using templates and the property replacer. In-Reply-To: <000101ca7008$230eb68e$100013ac@intern.adiscon.com> References: <000101ca7008$230eb68e$100013ac@intern.adiscon.com> Message-ID: <115906d10911280208m3ee48754i1c755ebaee67a1d2@mail.gmail.com> On Sat, Nov 28, 2009 at 03:52, Rainer Gerhards wrote: > Oh: which version do you use? 5.2.0? Then you should try the recent beta first... I'm using Rsyslog v5.5.1, and Librelp v.0.1.3. -Ryan From rgerhards at hq.adiscon.com Mon Nov 30 11:38:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 30 Nov 2009 11:38:29 +0100 Subject: [rsyslog] Auto-creating directories, when using templates and the property replacer. References: <000101ca7008$230eb68e$100013ac@intern.adiscon.com> <115906d10911280208m3ee48754i1c755ebaee67a1d2@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034DB@GRFEXC.intern.adiscon.com> Thanks, I was able to reproduced the problem, will now look into it. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > Sent: Saturday, November 28, 2009 11:08 AM > To: rsyslog-users > Subject: Re: [rsyslog] Auto-creating directories,when using templates > and the property replacer. > > On Sat, Nov 28, 2009 at 03:52, Rainer Gerhards > wrote: > > Oh: which version do you use? 5.2.0? Then you should try the recent > beta first... > > I'm using Rsyslog v5.5.1, and Librelp v.0.1.3. > > -Ryan > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Nov 30 12:08:09 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 30 Nov 2009 12:08:09 +0100 Subject: [rsyslog] Auto-creating directories, when using templates and the property replacer. References: <000101ca7008$230eb68e$100013ac@intern.adiscon.com><115906d10911280208m3ee48754i1c755ebaee67a1d2@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71034DB@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034DC@GRFEXC.intern.adiscon.com> Actually, it looks like it was just an improperly initialized variable, so that the default for creating directories was most often "on" (as documented), but could randomly be "off" as well. This bug was hidden all the way back to v3 (I didn't bother to look at v2...). Simple solution, now merged into all branches. The patch for is http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=7b40604e9ae8a0948f17eafd 4299eeb7fb3356c2 Or probably better just pull the newest git version. I would appreciate if you let me know if it works for you. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Monday, November 30, 2009 11:38 AM > To: rsyslog-users > Subject: Re: [rsyslog] Auto-creating directories,when using templates > and the property replacer. > > Thanks, I was able to reproduced the problem, will now look into it. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > > Sent: Saturday, November 28, 2009 11:08 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] Auto-creating directories,when using templates > > and the property replacer. > > > > On Sat, Nov 28, 2009 at 03:52, Rainer Gerhards > > wrote: > > > Oh: which version do you use? 5.2.0? Then you should try the recent > > beta first... > > > > I'm using Rsyslog v5.5.1, and Librelp v.0.1.3. > > > > -Ryan > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From xkubina at fi.muni.cz Mon Nov 30 14:55:55 2009 From: xkubina at fi.muni.cz (Tomas Kubina) Date: Mon, 30 Nov 2009 14:55:55 +0100 Subject: [rsyslog] TLS/GSSAPI client message lost Message-ID: <4B13CEEB.4040504@fi.muni.cz> Hi, I want to use TLS or GSS for message delivering to central rsyslog server. The problem is that the first message logged after server's shutdown is lost, but when I use plain TCP this issue doesn't happen. Is it a feature or mistake in my config? This is config for client: ################# #### MODULES #### ################# $ModLoad imuxsock # provides support for local system logging $ModLoad imklog # provides kernel logging support (previously done by rklogd) $ModLoad immark # provides --MARK-- message capability ########################### #### GLOBAL DIRECTIVES #### ########################### # # Use traditional timestamp format. # To enable high precision timestamps, comment out the following line. # $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat ############### #### RULES #### ############### # # First some standard log files. Log by facility. # auth,authpriv.* /var/log/auth.log *.*;auth,authpriv.none -/var/log/syslog #cron.* /var/log/cron.log daemon.* -/var/log/daemon.log kern.* -/var/log/kern.log lpr.* -/var/log/lpr.log mail.* -/var/log/mail.log user.* -/var/log/user.log # # Logging for the mail system. Split it up so that # it is easy to write scripts to parse these files. # mail.info -/var/log/mail.info mail.warn -/var/log/mail.warn mail.err /var/log/mail.err # # Logging for INN news system. # news.crit /var/log/news/news.crit news.err /var/log/news/news.err news.notice -/var/log/news/news.notice # # Some "catch-all" log files. # *.=debug;\ auth,authpriv.none;\ news.none;mail.none -/var/log/debug *.=info;*.=notice;*.=warn;\ auth,authpriv.none;\ cron,daemon.none;\ mail,news.none -/var/log/messages # # Emergencies are sent to everybody logged in. # *.emerg * # # I like to have messages displayed on the console, but only on a virtual # console I usually leave idle. # #daemon,mail.*;\ # news.=crit;news.=err;news.=notice;\ # *.=debug;*.=info;\ # *.=notice;*.=warn /dev/tty8 # The named pipe /dev/xconsole is for the `xconsole' utility. To use it, # you must invoke `xconsole' with the `-file' option: # # $ xconsole -file /dev/xconsole [...] # # NOTE: adjust the list below, or you'll go crazy if you have a reasonably # busy site.. # daemon.*;mail.*;\ news.err;\ *.=debug;*.=info;\ *.=notice;*.=warn |/dev/xconsole # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. $WorkDirectory /var/tmp/rsyslog/spool # where to place spool files $ActionQueueFileName uniqName # unique name prefix for spool files $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) $ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously $ActionResumeRetryCount -1 # infinite retries if host is down #$DefaultNetstreamDriver gtls # use gtls netstream driver # certificate files - just CA for a client #$DefaultNetstreamDriverCAFile /root/ils/cacert.pem #$DefaultNetstreamDriverCertFile /root/ils/bobatko_cert.pem #$DefaultNetstreamDriverKeyFile /root/ils/bobatko_cert.pem # set up the action #$ActionSendStreamDriverMode 1 # require TLS for the connection #$ActionSendStreamDriverAuthMode anon # server is NOT authenticated #local7.info @@example.com:10514 $ModLoad omgssapi local7.info : omgssapi:example.com:10514 and the server side: $ModLoad immark # provides --MARK-- message capability $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imklog # kernel logging (formerly provided by rklogd) # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none -/var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* -/var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit -/var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) #$ModLoad imtcp # TCP listener # make gtls driver the default #$DefaultNetstreamDriver gtls # certificate files #$DefaultNetstreamDriverCAFile /root/rsyslog/cacert.pem #$DefaultNetstreamDriverCertFile /root/rsyslog/rsyslog_cert.pem #$DefaultNetstreamDriverKeyFile /root/rsyslog/rsyslog_key.pem #$InputTCPServerStreamDriverAuthMode x509/name #$InputTCPServerStreamDriverPermittedPeer bobatko #$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode #$InputTCPServerRun 10514 # start up listener at port 10514 $ModLoad imgssapi $InputGSSServerRun 10514 Thank you for the answer, Regards, Tomas Kubina From rgerhards at hq.adiscon.com Mon Nov 30 15:06:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 30 Nov 2009 15:06:53 +0100 Subject: [rsyslog] TLS/GSSAPI client message lost References: <4B13CEEB.4040504@fi.muni.cz> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034DF@GRFEXC.intern.adiscon.com> I unfortunately do not have enough insight into GSSAPI to provide a real answer. My guess is that this happens for the same reason it can happen with any ack-less transport. In the plain tcp driver, I have done quite some work to try to limit loss potential. I have also some ideas of how to further try to prevent message loss, but you cannot totally aovid it. some background: http://blog.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tomas Kubina > Sent: Monday, November 30, 2009 2:56 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] TLS/GSSAPI client message lost > > Hi, > > I want to use TLS or GSS for message delivering to central rsyslog > server. > The problem is that the first message logged after server's shutdown is > lost, > but when I use plain TCP this issue doesn't happen. Is it a feature or > mistake > in my config? > > This is config for client: > > ################# > #### MODULES #### > ################# > > $ModLoad imuxsock # provides support for local system logging > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > $ModLoad immark # provides --MARK-- message capability > > > ########################### > #### GLOBAL DIRECTIVES #### > ########################### > > # > # Use traditional timestamp format. > # To enable high precision timestamps, comment out the following line. > # > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > ############### > #### RULES #### > ############### > > # > # First some standard log files. Log by facility. > # > auth,authpriv.* /var/log/auth.log > *.*;auth,authpriv.none -/var/log/syslog > #cron.* /var/log/cron.log > daemon.* -/var/log/daemon.log > kern.* -/var/log/kern.log > lpr.* -/var/log/lpr.log > mail.* -/var/log/mail.log > user.* -/var/log/user.log > > # > # Logging for the mail system. Split it up so that > # it is easy to write scripts to parse these files. > # > mail.info -/var/log/mail.info > mail.warn -/var/log/mail.warn > mail.err /var/log/mail.err > > # > # Logging for INN news system. > # > news.crit /var/log/news/news.crit > news.err /var/log/news/news.err > news.notice -/var/log/news/news.notice > > # > # Some "catch-all" log files. > # > *.=debug;\ > auth,authpriv.none;\ > news.none;mail.none -/var/log/debug > *.=info;*.=notice;*.=warn;\ > auth,authpriv.none;\ > cron,daemon.none;\ > mail,news.none -/var/log/messages > > # > # Emergencies are sent to everybody logged in. > # > *.emerg * > > # > # I like to have messages displayed on the console, but only on a > virtual > # console I usually leave idle. > # > #daemon,mail.*;\ > # news.=crit;news.=err;news.=notice;\ > # *.=debug;*.=info;\ > # *.=notice;*.=warn /dev/tty8 > > # The named pipe /dev/xconsole is for the `xconsole' utility. To use > it, > # you must invoke `xconsole' with the `-file' option: > # > # $ xconsole -file /dev/xconsole [...] > # > # NOTE: adjust the list below, or you'll go crazy if you have a > reasonably > # busy site.. > # > daemon.*;mail.*;\ > news.err;\ > *.=debug;*.=info;\ > *.=notice;*.=warn |/dev/xconsole > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/tmp/rsyslog/spool # where to place spool files > $ActionQueueFileName uniqName # unique name prefix for spool files > $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) > $ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > $ActionResumeRetryCount -1 # infinite retries if host is down > > > #$DefaultNetstreamDriver gtls # use gtls netstream driver > > # certificate files - just CA for a client > #$DefaultNetstreamDriverCAFile /root/ils/cacert.pem > #$DefaultNetstreamDriverCertFile /root/ils/bobatko_cert.pem > #$DefaultNetstreamDriverKeyFile /root/ils/bobatko_cert.pem > > # set up the action > #$ActionSendStreamDriverMode 1 # require TLS for the connection > #$ActionSendStreamDriverAuthMode anon # server is NOT authenticated > > #local7.info @@example.com:10514 > > > $ModLoad omgssapi > local7.info : omgssapi:example.com:10514 > > > and the server side: > > $ModLoad immark # provides --MARK-- message capability > $ModLoad imuxsock # provides support for local system logging (e.g. via > logger command) > $ModLoad imklog # kernel logging (formerly provided by rklogd) > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none -/var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* -/var/log/maillog > > > # Log cron stuff > cron.* -/var/log/cron > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit -/var/log/spooler > > # Save boot messages also to boot.log > local7.* /var/log/boot.log > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > > #$ModLoad imtcp # TCP listener > > # make gtls driver the default > #$DefaultNetstreamDriver gtls > > # certificate files > #$DefaultNetstreamDriverCAFile /root/rsyslog/cacert.pem > #$DefaultNetstreamDriverCertFile /root/rsyslog/rsyslog_cert.pem > #$DefaultNetstreamDriverKeyFile /root/rsyslog/rsyslog_key.pem > > #$InputTCPServerStreamDriverAuthMode x509/name > #$InputTCPServerStreamDriverPermittedPeer bobatko > #$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode > > #$InputTCPServerRun 10514 # start up listener at port 10514 > > $ModLoad imgssapi > $InputGSSServerRun 10514 > > Thank you for the answer, > > Regards, > Tomas Kubina > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From iamsayan at gmail.com Mon Nov 30 17:20:16 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Mon, 30 Nov 2009 11:20:16 -0500 Subject: [rsyslog] using arbitrary facility id Message-ID: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com> Hello All, Is it possible to use a facility id other than local0-local7? I was using a facility id of 50 in some of the messages , and I had written a selector line in my rsyslog file as well to log messages with facility id of 50 into a seperate file. However, I see that sometimes the messages are being written into all the files(/var/log/messages, boot.log,/var/log/secure etc) instead of the one I specified in the rsyslog.conf. If I restart rsyslog the problem goes away. I am using rsyslog version 4.2.0. Here is the selector line in my config if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and $syslogseverity <= '6' then $log_rotation_50 where $log_rotation_50 is an outchannel which is configured to rotate the file when it reaches the size of 2MB. Regards, Sayan From rgerhards at hq.adiscon.com Mon Nov 30 17:40:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 30 Nov 2009 17:40:17 +0100 Subject: [rsyslog] using arbitrary facility id References: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com> You can use facilities other than local0..local7, but you cannot use a facility with a numerical value greater than 23, because the relevant standards do not permit this (see RFC5424, Table 1). It may be that rsyslog does not properly prevent this, maybe it uses modulo 24 in this case. Will check that. But it is invalid in any case (not only with rsyslog but with any syslogd). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Monday, November 30, 2009 5:20 PM > To: rsyslog-users > Subject: [rsyslog] using arbitrary facility id > > Hello All, > Is it possible to use a facility id other than local0-local7? > I was using a facility id of 50 in some of the messages , and I had > written > a selector line in my rsyslog file as well to log messages with > facility id > of 50 into a seperate file. > However, I see that sometimes the messages are being written into all > the > files(/var/log/messages, boot.log,/var/log/secure etc) instead of the > one I > specified in the rsyslog.conf. If I restart rsyslog the problem goes > away. > I am using rsyslog version 4.2.0. > > Here is the selector line in my config > > > if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and > $syslogseverity <= '6' then $log_rotation_50 > > where $log_rotation_50 is an outchannel which is configured to rotate > the > file when it reaches the size of 2MB. > Regards, > Sayan > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From iamsayan at gmail.com Mon Nov 30 17:51:40 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Mon, 30 Nov 2009 11:51:40 -0500 Subject: [rsyslog] using arbitrary facility id In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com> References: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540911300851r1f0bab68ldf7f995ed0a094d8@mail.gmail.com> Hello Rainer, Thanks... Yes I agree what I am doing is invalid. I was just trying it out, my idea was if I can use numbers greater than 23, then I can maybe use it as my own customized logging system, and I would use numbers greater than 23 for my application and use different ids (>23) for different kind of application level log messages. I would try to look into the code as well, is this something you thing may be useful in general? Regards, Sayan On Mon, Nov 30, 2009 at 11:40 AM, Rainer Gerhards wrote: > You can use facilities other than local0..local7, but you cannot use a > facility with a numerical value greater than 23, because the relevant > standards do not permit this (see RFC5424, Table 1). > > It may be that rsyslog does not properly prevent this, maybe it uses modulo > 24 in this case. Will check that. But it is invalid in any case (not only > with rsyslog but with any syslogd). > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > Sent: Monday, November 30, 2009 5:20 PM > > To: rsyslog-users > > Subject: [rsyslog] using arbitrary facility id > > > > Hello All, > > Is it possible to use a facility id other than local0-local7? > > I was using a facility id of 50 in some of the messages , and I had > > written > > a selector line in my rsyslog file as well to log messages with > > facility id > > of 50 into a seperate file. > > However, I see that sometimes the messages are being written into all > > the > > files(/var/log/messages, boot.log,/var/log/secure etc) instead of the > > one I > > specified in the rsyslog.conf. If I restart rsyslog the problem goes > > away. > > I am using rsyslog version 4.2.0. > > > > Here is the selector line in my config > > > > > > if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and > > $syslogseverity <= '6' then $log_rotation_50 > > > > where $log_rotation_50 is an outchannel which is configured to rotate > > the > > file when it reaches the size of 2MB. > > Regards, > > Sayan > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iamsayan at gmail.com Mon Nov 30 17:58:46 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Mon, 30 Nov 2009 11:58:46 -0500 Subject: [rsyslog] using arbitrary facility id In-Reply-To: <87a1ae540911300851r1f0bab68ldf7f995ed0a094d8@mail.gmail.com> References: <87a1ae540911300820s3ce60cafq5c258b355ed3e887@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71034E7@GRFEXC.intern.adiscon.com> <87a1ae540911300851r1f0bab68ldf7f995ed0a094d8@mail.gmail.com> Message-ID: <87a1ae540911300858v1d4cda71k787a53158e162b7d@mail.gmail.com> BTW, just out of interest, why is this number restricted to 23 in the rfc? also each facility other than local0-7 seems to be rigidly defined. Just wanted to know why is it so ..shouldn't they have left scope for extension on this? On Mon, Nov 30, 2009 at 11:51 AM, Sayan Chowdhury wrote: > Hello Rainer, > Thanks... Yes I agree what I am doing is invalid. > I was just trying it out, my idea was if I can use numbers greater than > 23, then I can maybe use it as my own customized logging system, and I would > use numbers greater than 23 for my application and use different ids (>23) > for different kind of application level log messages. I would try to look > into the code as well, is this something you thing may be useful in general? > Regards, > Sayan > > > > On Mon, Nov 30, 2009 at 11:40 AM, Rainer Gerhards < > rgerhards at hq.adiscon.com> wrote: > >> You can use facilities other than local0..local7, but you cannot use a >> facility with a numerical value greater than 23, because the relevant >> standards do not permit this (see RFC5424, Table 1). >> >> It may be that rsyslog does not properly prevent this, maybe it uses >> modulo >> 24 in this case. Will check that. But it is invalid in any case (not only >> with rsyslog but with any syslogd). >> >> Rainer >> >> > -----Original Message----- >> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury >> > Sent: Monday, November 30, 2009 5:20 PM >> > To: rsyslog-users >> > Subject: [rsyslog] using arbitrary facility id >> > >> > Hello All, >> > Is it possible to use a facility id other than local0-local7? >> > I was using a facility id of 50 in some of the messages , and I had >> > written >> > a selector line in my rsyslog file as well to log messages with >> > facility id >> > of 50 into a seperate file. >> > However, I see that sometimes the messages are being written into all >> > the >> > files(/var/log/messages, boot.log,/var/log/secure etc) instead of the >> > one I >> > specified in the rsyslog.conf. If I restart rsyslog the problem goes >> > away. >> > I am using rsyslog version 4.2.0. >> > >> > Here is the selector line in my config >> > >> > >> > if $fromhost-ip == '127.0.0.1' and $syslogfacility == '50' and >> > $syslogseverity <= '6' then $log_rotation_50 >> > >> > where $log_rotation_50 is an outchannel which is configured to rotate >> > the >> > file when it reaches the size of 2MB. >> > Regards, >> > Sayan >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > From jbondc at openmv.com Mon Nov 30 21:49:24 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Mon, 30 Nov 2009 15:49:24 -0500 Subject: [rsyslog] TLS/GSSAPI client message lost In-Reply-To: <4B13CEEB.4040504@fi.muni.cz> References: <4B13CEEB.4040504@fi.muni.cz> Message-ID: <006d01ca71fe$9a625d50$cf2717f0$@com> On Mon Nov 30 08:55 AM, Tomas Kubina wrote: > Hi, > > I want to use TLS or GSS for message delivering to central rsyslog > server. > The problem is that the first message logged after server's shutdown > is lost, but when I use plain TCP this issue doesn't happen. Is it a > feature or mistake in my config? > I think your problem will be fixed in the latest rsyslog: http://www.rsyslog.com/Article433.phtml