From igalvarez at gmail.com Thu Oct 1 05:42:56 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Wed, 30 Sep 2009 22:42:56 -0500 Subject: [rsyslog] sending logs outside /etc/rsyslog.conf to remote server Message-ID: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> I've some daemons with their logs not define in /etc/rsyslog.conf and files outside /var/log folder. I have a central rsyslog receiving logs from severral servers. My question is: How can setup rsyslog on clients to send these logs outside /etc/rsyslog.conf to the central rsyslog server? Is this possible? -- Regards; Israel Garcia From david at lang.hm Thu Oct 1 06:17:19 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 21:17:19 -0700 (PDT) Subject: [rsyslog] sending logs outside /etc/rsyslog.conf to remote server In-Reply-To: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> References: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> Message-ID: On Wed, 30 Sep 2009, Israel Garcia wrote: > I've some daemons with their logs not define in /etc/rsyslog.conf and > files outside /var/log folder. I have a central rsyslog receiving logs > from severral servers. > > My question is: > > How can setup rsyslog on clients to send these logs outside > /etc/rsyslog.conf to the central rsyslog server? Is this possible? the first problem is to get the messages to rsyslog. if the applications don't have an option to log to syslog you can use the imfile input module in rsyslog to read lines from the file and process them. David Lang From rgerhards at hq.adiscon.com Thu Oct 1 11:01:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 1 Oct 2009 11:01:17 +0200 Subject: [rsyslog] DNS cache and expiration References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103110@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710311A@GRFEXC.intern.adiscon.com> I have now reviewed the discussion in depth. Many thanks for all the good thoughts. I will now first begin to move the name resolution calls to the "backend" threads (away from the inputs). However, I will create an option that the name resolution may be done before enqueing the message, which may be used by some folks for whatever reason (plus, it is trivially to do). I also think I will leave in the capability that inputs do the name resolution themselfs, at least if they prefer. For example, imuxsock always has the local host as input, so it makes limited sense to do name resolution calls for messages received from that input. Similarly, imtcp uses the name that was (once) obtained when the session was created (btw, this is also some expiry-less "name resolution", but I think here it is even harder to argue against it ;)). When I am done with this, I have a nice central place where the name resolution is done. In that place, I will then build a central name cache, at first with only two options - cache disabled - cache without expiration (requires shutdown) After that, I'll probably look into HUP plus probably provide a maximum TTL to rsyslog that says e.g. "check entry after n seconds". Even if n is low (maybe 10 seconds), the extra effort for doing these lookups is minimal: If the system has low throughput, the lookup cost is irrelevant in any case. If it is a high performance system, with more than e.g. 10,000 msgs/s, then n=10 means we do one lookup for at most 1/100,000th message, which doesn't matter either. Most importantly, I do not like the idea to use a third-party resolver library, as this creates another set of dependencies, and this for a core module. Booting rsyslog would get even more complicated. So I think the "expire ever n seconds" approach is a very practical one. You will see move these changes gradually into the master branch. One word on the performance: an avl-tree based internal cache can resolve names *very* fast. It has a depth of almost log n where n being the number of names cached. Building and maintaining the tree is almost for free, because the structure is rather static. Expiration, as I intend to implement it, is also a very simple operation, the node does not need to be removed and re-inserted but rather I need to update the resolution info by means of a new lookup (what costs the majority of time). So for a typical search, I would assume that the name can be resolved with less than 500 machine instructions (what I consider a conservative choice). The way I intend to implement it probably does not include any mutex calls, so there is no context switch involved. I consider the cost of a single context switch to be much higher than such an avl-tree lookup. Now consider any external solution. You need to do at least four context switches (rsyslog -> OS -> resolver -> OS -> rsyslog), probably more. Also, at least the OS scheduler is involved to activate the threads that are synchronized, plus probably a bit more of OS logic. Then, the external resolver can at best perform at the same rate rsyslog itself can (using a similar algorithm), but probably slower because it must be more generic. Looking at this picture, I think, clarifies why an (proper) internal implementation is so much more efficient than any external solution. Also note that I assume that in almost all cases, due to the structure of syslog traffic, no real lookup needs to be done, everything sits already in the resolver caches. I hope this is useful information. If someone has any additional comments, I would love to hear them. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Wednesday, September 30, 2009 9:57 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Wednesday, September 30, 2009 9:53 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] DNS cache and expiration > > > > On Wed, 30 Sep 2009, Rainer Gerhards wrote: > > > > > Thanks for the good discussion. I am lacking somewhat > > behind, but will review > > > it in depth tomorrow morning. > > > > > > I just wanted to stress the point that an external cache > > does not really > > > help, much for the reason David mentioned: if you process > > messages at very > > > high data rates, the context switch overhead involved with > > any external > > > solution is extremely costly. Also, in the usual cases, I > > may do several > > > million queries within a few seconds for just a handful of > > hosts. With an > > > internal cache, the overhead in doing so is very minimal. > > With an external > > > solution, the overhead in calling the external cache causes a lot > of > > > performance degredation, what in the case of UDP also > > implies (heavy!) > > > message loss. > > > > the message loss problem with UDP will not be solved completely by an > > internal cache. when the source is not in the cache and you > > have to go out > > to find it the lookup can take several seconds. > > > > moving the lookup out of the input module and into the output > > module would > > address this, anything else would leave you with losses as > > the cache gets > > populated. > > That's right and that's one reason why I intend to move this > (optionally) > over to the "backend" processing. However, even that does not > completely > solve the message loss problem, as we, in extreme cases, may loose > messages > when the queue is full - and for a myriad of other reasons, like > routers > discarding frames and such. Of course, you know that, but I'd like to > mention > if for those folks that at some time find our conversation via Google > ;) > > Rainer > > > > David Lang > > > > > Rainer > > > > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com > > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > david at lang.hm > > >> Sent: Wednesday, September 30, 2009 8:54 PM > > >> To: rsyslog-users > > >> Subject: Re: [rsyslog] DNS cache and expiration > > >> > > >> On Wed, 30 Sep 2009, Kenneth Marshall wrote: > > >> > > >>>>> mechanism to map IP addresses to names for the purposes > > of logging > > >>>>> error messages. As such, pretty much the only piece > > that needs to > > >>>>> be tracked within rsyslog is the TTL for the entry and the ip - > > > > >>>>> name mapping. A thread would be responsible for expiring > entries > > >>>>> from the cache (or refreshing the timeout) after validating the > > >>>>> correctness of the mapping. I think the DNS lookups should be > > >>>>> handled by a good resolver like pdns-recursor, djbdns,... The > > >>>>> goal here is to allow names in the log entries and not just IP > > >>>>> addresses and in a very high performance logging environment. > > >>>> > > >>>> the trouble is that doing _proper_ TTL expiration isn't as > > >> simple as it > > >>>> sounds. > > >>>> > > >>>> and if you are willing to back away from 'proper' > > >> expiration to something > > >>>> that will work in practice, why not go much further (as I > > >> have detailed in > > >>>> the other messages) > > >>>> > > >>>> David Lang > > >>> > > >>> I agree. I only mention TTL values as a reasonable > > upperbound of the > > >>> refresh check. The advantage is to remove the large pause > > in logging > > >>> due to a DNS refresh after a HUP to rsyslog, if that were the > only > > >>> method to flush/refresh. Like you mention, the IPs/names > > of systems > > >>> being logged change rarely so we should tune this for speed and > > >>> not worry about expiration correctness. I do not agree so other > > >>> statements that an external cache will perform as well as > > >> an internal > > >>> cache. Too many software products that I work with have > > >> needed exactly > > >>> that functionality to support very high levels of performance. > > >> > > >> actually, you could have the cache be configurable in three modes > > >> > > >> 1. no caching > > >> > > >> 2. blank the cache on HUP > > >> > > >> 3. never blank the cache (i.e. require a full restart to clear it) > > >> > > >> David Lang > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From igalvarez at gmail.com Thu Oct 1 17:09:04 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Thu, 1 Oct 2009 10:09:04 -0500 Subject: [rsyslog] sending logs outside /etc/rsyslog.conf to remote server In-Reply-To: References: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> Message-ID: <194a2c240910010809k99cd7e7o46d40e428923912f@mail.gmail.com> On 9/30/09, david at lang.hm wrote: > On Wed, 30 Sep 2009, Israel Garcia wrote: > >> I've some daemons with their logs not define in /etc/rsyslog.conf and >> files outside /var/log folder. I have a central rsyslog receiving logs >> from severral servers. >> >> My question is: >> >> How can setup rsyslog on clients to send these logs outside >> /etc/rsyslog.conf to the central rsyslog server? Is this possible? > > the first problem is to get the messages to rsyslog. > > if the applications don't have an option to log to syslog you can use the > imfile input module in rsyslog to read lines from the file and process > them. Hi David, Has anybody on the list setup apache, proftpd, momgrel or any other daemon logs to forward to a centralized syslog server? I'd appreciate your experiences. regards, Israel. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From rgerhards at hq.adiscon.com Thu Oct 1 17:25:24 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 1 Oct 2009 17:25:24 +0200 Subject: [rsyslog] sending logs outside /etc/rsyslog.conf to remoteserver References: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> <194a2c240910010809k99cd7e7o46d40e428923912f@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710312D@GRFEXC.intern.adiscon.com> no experience on my side, but I wanted to make you aware of the wiki, which (I think) contains a number of samples: http://wiki.rsyslog.com Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Israel Garcia > Sent: Thursday, October 01, 2009 5:09 PM > To: rsyslog-users > Subject: Re: [rsyslog] sending logs outside /etc/rsyslog.conf to > remoteserver > > On 9/30/09, david at lang.hm wrote: > > On Wed, 30 Sep 2009, Israel Garcia wrote: > > > >> I've some daemons with their logs not define in /etc/rsyslog.conf > and > >> files outside /var/log folder. I have a central rsyslog receiving > logs > >> from severral servers. > >> > >> My question is: > >> > >> How can setup rsyslog on clients to send these logs outside > >> /etc/rsyslog.conf to the central rsyslog server? Is this possible? > > > > the first problem is to get the messages to rsyslog. > > > > if the applications don't have an option to log to syslog you can use > the > > imfile input module in rsyslog to read lines from the file and > process > > them. > > Hi David, > > Has anybody on the list setup apache, proftpd, momgrel or any other > daemon logs to forward to a centralized syslog server? I'd appreciate > your experiences. > > regards, > Israel. > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > -- > Regards; > Israel Garcia > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Thu Oct 1 19:06:56 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 1 Oct 2009 10:06:56 -0700 (PDT) Subject: [rsyslog] sending logs outside /etc/rsyslog.conf to remote server In-Reply-To: <194a2c240910010809k99cd7e7o46d40e428923912f@mail.gmail.com> References: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> <194a2c240910010809k99cd7e7o46d40e428923912f@mail.gmail.com> Message-ID: On Thu, 1 Oct 2009, Israel Garcia wrote: > On 9/30/09, david at lang.hm wrote: >> On Wed, 30 Sep 2009, Israel Garcia wrote: >> >>> I've some daemons with their logs not define in /etc/rsyslog.conf and >>> files outside /var/log folder. I have a central rsyslog receiving logs >>> from severral servers. >>> >>> My question is: >>> >>> How can setup rsyslog on clients to send these logs outside >>> /etc/rsyslog.conf to the central rsyslog server? Is this possible? >> >> the first problem is to get the messages to rsyslog. >> >> if the applications don't have an option to log to syslog you can use the >> imfile input module in rsyslog to read lines from the file and process >> them. > > Hi David, > > Has anybody on the list setup apache, proftpd, momgrel or any other > daemon logs to forward to a centralized syslog server? I'd appreciate > your experiences. I haven't done it yet, but this page makes it seem pretty easy to setup apache http://www.oreillynet.com/pub/a/sysadmin/2006/10/12/httpd-syslog.html I've had proftpd send things to syslog in the past (IIRC it was a trivial config option) no expriance with mongrel David Lang From david at lang.hm Thu Oct 1 21:26:23 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 1 Oct 2009 12:26:23 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710311A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103110@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710311A@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 1 Oct 2009, Rainer Gerhards wrote: > I have now reviewed the discussion in depth. Many thanks for all the good > thoughts. > > I will now first begin to move the name resolution calls to the "backend" > threads (away from the inputs). However, I will create an option that the > name resolution may be done before enqueing the message, which may be used by > some folks for whatever reason (plus, it is trivially to do). I also think I > will leave in the capability that inputs do the name resolution themselfs, at > least if they prefer. For example, imuxsock always has the local host as > input, so it makes limited sense to do name resolution calls for messages > received from that input. Similarly, imtcp uses the name that was (once) > obtained when the session was created (btw, this is also some expiry-less > "name resolution", but I think here it is even harder to argue against it > ;)). on note on this, if you can delay the resolution to the point where something actually accesses the property that uses the name you may be able to skip doing name resolution entirely in many cases. David Lang > When I am done with this, I have a nice central place where the name > resolution is done. In that place, I will then build a central name cache, at > first with only two options > > - cache disabled > - cache without expiration (requires shutdown) > > After that, I'll probably look into HUP plus probably provide a maximum TTL > to rsyslog that says e.g. "check entry after n seconds". Even if n is low > (maybe 10 seconds), the extra effort for doing these lookups is minimal: > > If the system has low throughput, the lookup cost is irrelevant in any case. > If it is a high performance system, with more than e.g. 10,000 msgs/s, then > n=10 means we do one lookup for at most 1/100,000th message, which doesn't > matter either. > > Most importantly, I do not like the idea to use a third-party resolver > library, as this creates another set of dependencies, and this for a core > module. Booting rsyslog would get even more complicated. So I think the > "expire ever n seconds" approach is a very practical one. > > You will see move these changes gradually into the master branch. > > One word on the performance: an avl-tree based internal cache can resolve > names *very* fast. It has a depth of almost log n where n being the number of > names cached. Building and maintaining the tree is almost for free, because > the structure is rather static. Expiration, as I intend to implement it, is > also a very simple operation, the node does not need to be removed and > re-inserted but rather I need to update the resolution info by means of a new > lookup (what costs the majority of time). So for a typical search, I would > assume that the name can be resolved with less than 500 machine instructions > (what I consider a conservative choice). The way I intend to implement it > probably does not include any mutex calls, so there is no context switch > involved. > > I consider the cost of a single context switch to be much higher than such an > avl-tree lookup. Now consider any external solution. You need to do at least > four context switches (rsyslog -> OS -> resolver -> OS -> rsyslog), probably > more. Also, at least the OS scheduler is involved to activate the threads > that are synchronized, plus probably a bit more of OS logic. Then, the > external resolver can at best perform at the same rate rsyslog itself can > (using a similar algorithm), but probably slower because it must be more > generic. Looking at this picture, I think, clarifies why an (proper) internal > implementation is so much more efficient than any external solution. Also > note that I assume that in almost all cases, due to the structure of syslog > traffic, no real lookup needs to be done, everything sits already in the > resolver caches. > > I hope this is useful information. If someone has any additional comments, I > would love to hear them. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Wednesday, September 30, 2009 9:57 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] DNS cache and expiration >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com >>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Wednesday, September 30, 2009 9:53 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] DNS cache and expiration >>> >>> On Wed, 30 Sep 2009, Rainer Gerhards wrote: >>> >>>> Thanks for the good discussion. I am lacking somewhat >>> behind, but will review >>>> it in depth tomorrow morning. >>>> >>>> I just wanted to stress the point that an external cache >>> does not really >>>> help, much for the reason David mentioned: if you process >>> messages at very >>>> high data rates, the context switch overhead involved with >>> any external >>>> solution is extremely costly. Also, in the usual cases, I >>> may do several >>>> million queries within a few seconds for just a handful of >>> hosts. With an >>>> internal cache, the overhead in doing so is very minimal. >>> With an external >>>> solution, the overhead in calling the external cache causes a lot >> of >>>> performance degredation, what in the case of UDP also >>> implies (heavy!) >>>> message loss. >>> >>> the message loss problem with UDP will not be solved completely by an >>> internal cache. when the source is not in the cache and you >>> have to go out >>> to find it the lookup can take several seconds. >>> >>> moving the lookup out of the input module and into the output >>> module would >>> address this, anything else would leave you with losses as >>> the cache gets >>> populated. >> >> That's right and that's one reason why I intend to move this >> (optionally) >> over to the "backend" processing. However, even that does not >> completely >> solve the message loss problem, as we, in extreme cases, may loose >> messages >> when the queue is full - and for a myriad of other reasons, like >> routers >> discarding frames and such. Of course, you know that, but I'd like to >> mention >> if for those folks that at some time find our conversation via Google >> ;) >> >> Rainer >>> >>> David Lang >>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com >>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >>> david at lang.hm >>>>> Sent: Wednesday, September 30, 2009 8:54 PM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] DNS cache and expiration >>>>> >>>>> On Wed, 30 Sep 2009, Kenneth Marshall wrote: >>>>> >>>>>>>> mechanism to map IP addresses to names for the purposes >>> of logging >>>>>>>> error messages. As such, pretty much the only piece >>> that needs to >>>>>>>> be tracked within rsyslog is the TTL for the entry and the ip - >>> >>>>>>>> name mapping. A thread would be responsible for expiring >> entries >>>>>>>> from the cache (or refreshing the timeout) after validating the >>>>>>>> correctness of the mapping. I think the DNS lookups should be >>>>>>>> handled by a good resolver like pdns-recursor, djbdns,... The >>>>>>>> goal here is to allow names in the log entries and not just IP >>>>>>>> addresses and in a very high performance logging environment. >>>>>>> >>>>>>> the trouble is that doing _proper_ TTL expiration isn't as >>>>> simple as it >>>>>>> sounds. >>>>>>> >>>>>>> and if you are willing to back away from 'proper' >>>>> expiration to something >>>>>>> that will work in practice, why not go much further (as I >>>>> have detailed in >>>>>>> the other messages) >>>>>>> >>>>>>> David Lang >>>>>> >>>>>> I agree. I only mention TTL values as a reasonable >>> upperbound of the >>>>>> refresh check. The advantage is to remove the large pause >>> in logging >>>>>> due to a DNS refresh after a HUP to rsyslog, if that were the >> only >>>>>> method to flush/refresh. Like you mention, the IPs/names >>> of systems >>>>>> being logged change rarely so we should tune this for speed and >>>>>> not worry about expiration correctness. I do not agree so other >>>>>> statements that an external cache will perform as well as >>>>> an internal >>>>>> cache. Too many software products that I work with have >>>>> needed exactly >>>>>> that functionality to support very high levels of performance. >>>>> >>>>> actually, you could have the cache be configurable in three modes >>>>> >>>>> 1. no caching >>>>> >>>>> 2. blank the cache on HUP >>>>> >>>>> 3. never blank the cache (i.e. require a full restart to clear it) >>>>> >>>>> David Lang >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu Oct 1 21:27:40 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 1 Oct 2009 12:27:40 -0700 (PDT) Subject: [rsyslog] fromhost-ip not working with -x Message-ID: I am running all my servers with -x to disable name resolution. I have found that fromhost exists, but fromhost-ip does not this is either a bug in the documentation, or fromhost-ip doesn't get created if no name resolution is done. david Lang From rgerhards at hq.adiscon.com Thu Oct 1 21:45:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 1 Oct 2009 21:45:56 +0200 Subject: [rsyslog] DNS cache and expiration References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103110@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710311A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103133@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, October 01, 2009 9:26 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > On Thu, 1 Oct 2009, Rainer Gerhards wrote: > > > I have now reviewed the discussion in depth. Many thanks > for all the good > > thoughts. > > > > I will now first begin to move the name resolution calls to > the "backend" > > threads (away from the inputs). However, I will create an > option that the > > name resolution may be done before enqueing the message, > which may be used by > > some folks for whatever reason (plus, it is trivially to > do). I also think I > > will leave in the capability that inputs do the name > resolution themselfs, at > > least if they prefer. For example, imuxsock always has the > local host as > > input, so it makes limited sense to do name resolution > calls for messages > > received from that input. Similarly, imtcp uses the name > that was (once) > > obtained when the session was created (btw, this is also > some expiry-less > > "name resolution", but I think here it is even harder to > argue against it > > ;)). > > on note on this, if you can delay the resolution to the point where > something actually accesses the property that uses the name > you may be > able to skip doing name resolution entirely in many cases. This I plan to do, but in the TCP case I think it probably is better to do a name resolution once per *session* even if the name may never be used. Typically, there are many messages in a session and the overhead if the one name lookup does not really matter. At least this is my current thinking. Rainer From david at lang.hm Thu Oct 1 22:07:31 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 1 Oct 2009 13:07:31 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103133@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103110@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710311A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103133@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 1 Oct 2009, Rainer Gerhards wrote: >> -----Original Message----- >> On Thu, 1 Oct 2009, Rainer Gerhards wrote: >> >>> I have now reviewed the discussion in depth. Many thanks >> for all the good >>> thoughts. >>> >>> I will now first begin to move the name resolution calls to >> the "backend" >>> threads (away from the inputs). However, I will create an >> option that the >>> name resolution may be done before enqueing the message, >> which may be used by >>> some folks for whatever reason (plus, it is trivially to >> do). I also think I >>> will leave in the capability that inputs do the name >> resolution themselfs, at >>> least if they prefer. For example, imuxsock always has the >> local host as >>> input, so it makes limited sense to do name resolution >> calls for messages >>> received from that input. Similarly, imtcp uses the name >> that was (once) >>> obtained when the session was created (btw, this is also >> some expiry-less >>> "name resolution", but I think here it is even harder to >> argue against it >>> ;)). >> >> on note on this, if you can delay the resolution to the point where >> something actually accesses the property that uses the name >> you may be >> able to skip doing name resolution entirely in many cases. > > This I plan to do, but in the TCP case I think it probably is better to do a > name resolution once per *session* even if the name may never be used. > Typically, there are many messages in a session and the overhead if the one > name lookup does not really matter. At least this is my current thinking. no problem with filling it out earlier if the info happens to be available (or in the case of thinks like TCP, RELP, etc where it really is static). I'm thinking of things like my central server that receives the aggregated feed from several relays. everything that it does is based on the name in the log, and since the logs come from other rsyslog relay boxes, those boxes will have cleaned things up, so there is never (outside of debugging) a need for that central box to do _any_ lookups, no matter how fast they end up being. David Lang From anichols at trumped.org Fri Oct 2 01:52:22 2009 From: anichols at trumped.org (Aaron Nichols) Date: Thu, 1 Oct 2009 17:52:22 -0600 Subject: [rsyslog] "too many tcp sessions - dropping incoming request" - tried increasing max sessions to no avail Message-ID: Hi all, I am seeing the following in my logs running rsyslog 4.4.1: Oct 1 23:29:24 host rsyslogd-2079: too many tcp sessions - dropping incoming request [try http://www.rsyslog.com/e/2079 ] Oct 1 23:29:29 host rsyslogd-2079: too many tcp sessions - dropping incoming request [try http://www.rsyslog.com/e/2079 ] Oct 1 23:29:38 host rsyslogd-2079: too many tcp sessions - dropping incoming request [try http://www.rsyslog.com/e/2079 ] I went to the URL in the message and the answer makes sense so I added the following to my configuration: $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) I added it before the imtcp module is loaded so the config looks like: # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) $ModLoad imtcp.so # load module $InputTCPServerRun 514 # start up TCP listener at port 514 I am still seeing the "too many tcp sessions" in the logs. When I take a look at netstat (assuming this is the relevant data) I see less than 1000 *connections* - but I'm not sure if each of these is considered a "session" $ netstat -an | grep :514 | wc -l 269 If I look for only ESTABLISHED connections I see less: [anichols at inlwr01p1 inlwr01p1]$ netstat -an | grep :514 | grep ESTABLISHED | wc -l 231 Many of the connections show a SYN_RECV state on the server side - I assume these are cases where we aren't completing the tcp handshake due to too many connections. Are there other factors which may be impacting the max connections or is my configuration incorrect? Thanks, Aaron From anichols at trumped.org Fri Oct 2 04:44:50 2009 From: anichols at trumped.org (Aaron Nichols) Date: Thu, 1 Oct 2009 20:44:50 -0600 Subject: [rsyslog] "too many tcp sessions - dropping incoming request" - tried increasing max sessions to no avail In-Reply-To: References: Message-ID: I found the cause of my problem I believe - the $InputTCPMaxSessions value needs to come after the imtcp module is loaded. This makes sense in hindsight. Of course, the evidence was right in front of me had I looked in the right place: Oct 2 02:34:54 host rsyslogd-3003: invalid or yet-unknown config file command - have you forgotten to load a module? [try http://www.rsyslog.com/e/3003 ] Oct 2 02:34:54 host rsyslogd: the last error occured in /etc/rsyslog.d/rsyslog-server.conf, line 4 I have to thank you guys for making an effort to provide informative errors with URL references. Thanks, Aaron On Thu, Oct 1, 2009 at 5:52 PM, Aaron Nichols wrote: > Hi all, > > I am seeing the following in my logs running rsyslog 4.4.1: > > Oct 1 23:29:24 host rsyslogd-2079: too many tcp sessions - dropping > incoming request [try http://www.rsyslog.com/e/2079 ] > Oct 1 23:29:29 host rsyslogd-2079: too many tcp sessions - dropping > incoming request [try http://www.rsyslog.com/e/2079 ] > Oct 1 23:29:38 host rsyslogd-2079: too many tcp sessions - dropping > incoming request [try http://www.rsyslog.com/e/2079 ] > > I went to the URL in the message and the answer makes sense so I added the > following to my configuration: > > $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) > > I added it before the imtcp module is loaded so the config looks like: > > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) > $ModLoad imtcp.so # load module > $InputTCPServerRun 514 # start up TCP listener at port 514 > > I am still seeing the "too many tcp sessions" in the logs. When I take a > look at netstat (assuming this is the relevant data) I see less than 1000 > *connections* - but I'm not sure if each of these is considered a "session" > > $ netstat -an | grep :514 | wc -l > 269 > > If I look for only ESTABLISHED connections I see less: > > [anichols at inlwr01p1 inlwr01p1]$ netstat -an | grep :514 | grep ESTABLISHED > | wc -l > 231 > > Many of the connections show a SYN_RECV state on the server side - I assume > these are cases where we aren't completing the tcp handshake due to too many > connections. > > Are there other factors which may be impacting the max connections or is my > configuration incorrect? > > Thanks, > Aaron > From tbergfeld at hq.adiscon.com Mon Oct 5 15:13:52 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Mon, 5 Oct 2009 15:13:52 +0200 Subject: [rsyslog] rsyslog 5.3.1 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710315D@GRFEXC.intern.adiscon.com> Hi all, We have just released 5.3.1, a member of the devel branch. This is a bug-fixing release, providing some important fixes from the v4-stable and the current v4-beta, like a bugfix for a potential segfault in stream writer on destruction. This is a recommended update for all users of the devel branch. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article407.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-178.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From salim.engin at gmail.com Wed Oct 7 15:23:21 2009 From: salim.engin at gmail.com (Salim Engin) Date: Wed, 7 Oct 2009 16:23:21 +0300 Subject: [rsyslog] Sender IP or Hostname Message-ID: <43660480910070623s6eeb20b3gf200de18cff7d553@mail.gmail.com> Hello all, I have just installed Debain 5.0.3 with its default syslog server which is rsyslog 3.18.6 . and I configured it to receive external syslog messages from our network devices. What the problem is that, the sender IP or hostname of external devices which send syslog messages is not displayed in the log . Please see the example below; 2009-10-07T16:09:39.407770+03:00 debian dhcpd: DHCPACK on 172.10.10.101 to 00:24:7e:11:eb:49 (stajerstac) via vlan110 --------> "debian" hostname is OK 2009-09-06T14:04:53+03:00 - rCT0[184]: craft(8) : RTRV-HDR:::P6099; -----------------------------------------------------------------------------------> Neither IP nor hostname is printed 2009-09-06T14:04:56+03:00 - clij[258]: isadmin at 10.5.5.23(21) : info configure system sntp ----------------------------------------------------> Neither IP nor hostname is printed. Instead there is dash after the timestamp. I tried many things but was not able to make it work. Can someone help me to fix this ? Thanks Salim Engin From rgerhards at hq.adiscon.com Wed Oct 7 15:33:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 7 Oct 2009 15:33:17 +0200 Subject: [rsyslog] Sender IP or Hostname References: <43660480910070623s6eeb20b3gf200de18cff7d553@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103191@GRFEXC.intern.adiscon.com> Hi Salim, sounds like a malformed message is causing this: http://www.rsyslog.com/doc-syslog_parsing.html Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Salim Engin > Sent: Wednesday, October 07, 2009 3:23 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Sender IP or Hostname > > Hello all, > > I have just installed Debain 5.0.3 with its default syslog server which > is > rsyslog 3.18.6 . and I configured it to receive external syslog > messages > from our network devices. > > What the problem is that, the sender IP or hostname of external devices > which send syslog messages is not displayed in the log . Please see the > example below; > > > 2009-10-07T16:09:39.407770+03:00 debian dhcpd: DHCPACK on 172.10.10.101 > to > 00:24:7e:11:eb:49 (stajerstac) via vlan110 --------> "debian" hostname > is > OK > 2009-09-06T14:04:53+03:00 - rCT0[184]: craft(8) : RTRV-HDR:::P6099; > ----------------------------------------------------------------------- > ------------> > Neither IP nor hostname is printed > 2009-09-06T14:04:56+03:00 - clij[258]: isadmin at 10.5.5.23(21) : info > configure system sntp ------------------------------------------------- > ---> > Neither IP nor hostname is printed. Instead there is dash after the > timestamp. > > > I tried many things but was not able to make it work. > > Can someone help me to fix this ? > > Thanks > > Salim Engin > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From iamsayan at gmail.com Wed Oct 7 16:35:55 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Wed, 7 Oct 2009 20:05:55 +0530 Subject: [rsyslog] limiting message size while forwarding messages Message-ID: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> Hello All,RFC 3164 says that max rsyslog message length is 1024 bytes. Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 and has a default max message size of 2K. Now, I don't want to restrict the size while receiving messages.However when I forward message out of the box, I want to truncate it to 1K, as I am not sure what implementations I will have to interop with. I could not find an easy way to do this. $MaxMessageSize seems to control both sending and receiving. Is there a way in which I can achieve this? Regards, Sayan From rgerhards at hq.adiscon.com Wed Oct 7 16:43:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 7 Oct 2009 16:43:56 +0200 Subject: [rsyslog] limiting message size while forwarding messages References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> you need to look at the property replacer, and do something like %msg:1:950% in your template. I am not sure, but I think the default template already does this. Did you experience oversize messages on a receiver? > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Wednesday, October 07, 2009 4:36 PM > To: rsyslog-users > Subject: [rsyslog] limiting message size while forwarding messages > > Hello All,RFC 3164 says that max rsyslog message length is 1024 bytes. > Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 and > has a > default max message size of 2K. > Now, I don't want to restrict the size while receiving messages.However > when > I forward message out of the box, I want to truncate it > to 1K, as I am not sure what implementations I will have to interop > with. > I could not find an easy way to do this. $MaxMessageSize seems to > control > both sending and receiving. > Is there a way in which I can achieve this? > Regards, > Sayan > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From dirk.schulz at kinzesberg.de Wed Oct 7 16:31:34 2009 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Wed, 07 Oct 2009 16:31:34 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG Message-ID: <4ACCA646.8050209@kinzesberg.de> Hi folks, I am not sure if this is the right list to ask, but I hope that there is more of the related knowledge here. I have several Syslog-NG servers reporting to Rsyslog central servers (just because nobody could tell me how to run Rsyslog on NetBSD ...). Now I fould that the facility is not transferred to/evaluated on the Rsyslog servers; e.g. mail messages are written to syslog instead of maillog on the Rsyslog machines. Is there something special I have to configure on the Rsyslog servers or the Syslog-NG servers to fix that? Any hint or help is appreciated. Dirk From iamsayan at gmail.com Wed Oct 7 16:53:08 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Wed, 7 Oct 2009 20:23:08 +0530 Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> thanks Rainer, I will try that.So that means if I use the default template for forwarding, i think it should just work then. I did actually get a message from an application running on my box(uses standard syslog api) which was around 4K in size rsyslog did the right thing and truncated it to 2K.and logged into the file as per the configured rule. now I have requirement to support messages up to 5K while logging to the local file, but truncating it to 1K while sending it out. On Wed, Oct 7, 2009 at 8:13 PM, Rainer Gerhards wrote: > you need to look at the property replacer, and do something like > %msg:1:950% > in your template. I am not sure, but I think the default template already > does this. Did you experience oversize messages on a receiver? > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > Sent: Wednesday, October 07, 2009 4:36 PM > > To: rsyslog-users > > Subject: [rsyslog] limiting message size while forwarding messages > > > > Hello All,RFC 3164 says that max rsyslog message length is 1024 bytes. > > Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 and > > has a > > default max message size of 2K. > > Now, I don't want to restrict the size while receiving messages.However > > when > > I forward message out of the box, I want to truncate it > > to 1K, as I am not sure what implementations I will have to interop > > with. > > I could not find an easy way to do this. $MaxMessageSize seems to > > control > > both sending and receiving. > > Is there a way in which I can achieve this? > > Regards, > > Sayan > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 7 18:32:52 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 7 Oct 2009 18:32:52 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG References: <4ACCA646.8050209@kinzesberg.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dirk H. Schulz > Sent: Wednesday, October 07, 2009 4:32 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Getting Facility from Synlog-NG > > Hi folks, > > I am not sure if this is the right list to ask, but I hope that there > is > more of the related knowledge here. > > I have several Syslog-NG servers reporting to Rsyslog central servers > (just because nobody could tell me how to run Rsyslog on NetBSD ...). > Now I fould that the facility is not transferred to/evaluated on the > Rsyslog servers; e.g. mail messages are written to syslog instead of > maillog on the Rsyslog machines. This is strange. Facility is a fairly standard field. Could you tell me the content of %rawmwg%. This is done along these lines (if it does not work, check doc for exact syntax): $template raw,"%rawmsg%\n" *.* /path/to/file;raw Rainer > > Is there something special I have to configure on the Rsyslog servers > or > the Syslog-NG servers to fix that? > > Any hint or help is appreciated. > > Dirk > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 7 18:37:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 7 Oct 2009 18:37:17 +0200 Subject: [rsyslog] limiting message size while forwarding messages References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> I just checked, and the default is to send the full message. See: http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/syslogd.c;h=0f4f8a233c97 a44e72c97a555ff6f95eb367573a;hb=HEAD#l382 But all you need to do is use %msg:1:950...% in a template that otherwise is the same as the canned template. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Wednesday, October 07, 2009 4:53 PM > To: rsyslog-users > Subject: Re: [rsyslog] limiting message size while forwarding messages > > thanks Rainer, I will try that.So that means if I use the default > template > for forwarding, i think it should just work then. I did actually get a > message from an application running on my box(uses standard syslog api) > which was around 4K in size > rsyslog did the right thing and truncated it to 2K.and logged into the > file > as per the configured rule. > > now I have requirement to support messages up to 5K while logging to > the > local file, but truncating it to 1K while sending it out. > > > On Wed, Oct 7, 2009 at 8:13 PM, Rainer Gerhards > wrote: > > > you need to look at the property replacer, and do something like > > %msg:1:950% > > in your template. I am not sure, but I think the default template > already > > does this. Did you experience oversize messages on a receiver? > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > Sent: Wednesday, October 07, 2009 4:36 PM > > > To: rsyslog-users > > > Subject: [rsyslog] limiting message size while forwarding messages > > > > > > Hello All,RFC 3164 says that max rsyslog message length is 1024 > bytes. > > > Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 > and > > > has a > > > default max message size of 2K. > > > Now, I don't want to restrict the size while receiving > messages.However > > > when > > > I forward message out of the box, I want to truncate it > > > to 1K, as I am not sure what implementations I will have to interop > > > with. > > > I could not find an easy way to do this. $MaxMessageSize seems to > > > control > > > both sending and receiving. > > > Is there a way in which I can achieve this? > > > Regards, > > > Sayan > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From salim.engin at gmail.com Wed Oct 7 19:00:38 2009 From: salim.engin at gmail.com (Salim Engin) Date: Wed, 7 Oct 2009 20:00:38 +0300 Subject: [rsyslog] Sender IP or Hostname In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103191@GRFEXC.intern.adiscon.com> References: <43660480910070623s6eeb20b3gf200de18cff7d553@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103191@GRFEXC.intern.adiscon.com> Message-ID: <43660480910071000o6dbd760cn66ee567ccb844c28@mail.gmail.com> That is right. I configured an other networking device to use rsyslog and saw that the sender IP is displayed in the log file. But the WA mentioned in the document did not work ... Thanks On Wed, Oct 7, 2009 at 4:33 PM, Rainer Gerhards wrote: > Hi Salim, > > sounds like a malformed message is causing this: > > http://www.rsyslog.com/doc-syslog_parsing.html > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Salim Engin > > Sent: Wednesday, October 07, 2009 3:23 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] Sender IP or Hostname > > > > Hello all, > > > > I have just installed Debain 5.0.3 with its default syslog server which > > is > > rsyslog 3.18.6 . and I configured it to receive external syslog > > messages > > from our network devices. > > > > What the problem is that, the sender IP or hostname of external devices > > which send syslog messages is not displayed in the log . Please see the > > example below; > > > > > > 2009-10-07T16:09:39.407770+03:00 debian dhcpd: DHCPACK on 172.10.10.101 > > to > > 00:24:7e:11:eb:49 (stajerstac) via vlan110 --------> "debian" hostname > > is > > OK > > 2009-09-06T14:04:53+03:00 - rCT0[184]: craft(8) : RTRV-HDR:::P6099; > > ----------------------------------------------------------------------- > > ------------> > > Neither IP nor hostname is printed > > 2009-09-06T14:04:56+03:00 - clij[258]: isadmin at 10.5.5.23(21) : info > > configure system sntp ------------------------------------------------- > > ---> > > Neither IP nor hostname is printed. Instead there is dash after the > > timestamp. > > > > > > I tried many things but was not able to make it work. > > > > Can someone help me to fix this ? > > > > Thanks > > > > Salim Engin > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 7 19:05:28 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 7 Oct 2009 10:05:28 -0700 (PDT) Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> Message-ID: On Wed, 7 Oct 2009, Sayan Chowdhury wrote: > thanks Rainer, I will try that.So that means if I use the default template > for forwarding, i think it should just work then. I did actually get a > message from an application running on my box(uses standard syslog api) > which was around 4K in size > rsyslog did the right thing and truncated it to 2K.and logged into the file > as per the configured rule. > > now I have requirement to support messages up to 5K while logging to the > local file, but truncating it to 1K while sending it out. RFC 3164 says that a syslog implementation should truncate any oversized messages that it gets, so it should be safe to send a 2k message to a syslog daemon that truncates it to 1k note that in rsyslog the max message size is configrable. you may want to just set it to 1k until everything in your infrastructure handles larger messages (unless you want rsyslog to get the full message and send a portion of it, in which case you may want to set it larger) David Lang > > On Wed, Oct 7, 2009 at 8:13 PM, Rainer Gerhards wrote: > >> you need to look at the property replacer, and do something like >> %msg:1:950% >> in your template. I am not sure, but I think the default template already >> does this. Did you experience oversize messages on a receiver? >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury >>> Sent: Wednesday, October 07, 2009 4:36 PM >>> To: rsyslog-users >>> Subject: [rsyslog] limiting message size while forwarding messages >>> >>> Hello All,RFC 3164 says that max rsyslog message length is 1024 bytes. >>> Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 and >>> has a >>> default max message size of 2K. >>> Now, I don't want to restrict the size while receiving messages.However >>> when >>> I forward message out of the box, I want to truncate it >>> to 1K, as I am not sure what implementations I will have to interop >>> with. >>> I could not find an easy way to do this. $MaxMessageSize seems to >>> control >>> both sending and receiving. >>> Is there a way in which I can achieve this? >>> Regards, >>> Sayan >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iamsayan at gmail.com Wed Oct 7 20:11:49 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Wed, 7 Oct 2009 23:41:49 +0530 Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> Yes, I saw that too ... I made the template based out of Rsyslog traditional forward format and then used the property replacer as you mentioned. It works like a charm. Thanks a bunch! Regards, Sayan On Wed, Oct 7, 2009 at 10:07 PM, Rainer Gerhards wrote: > I just checked, and the default is to send the full message. See: > > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/syslogd.c;h=0f4f8a233c97 > a44e72c97a555ff6f95eb367573a;hb=HEAD#l382 > > But all you need to do is use %msg:1:950...% in a template that otherwise > is > the same as the canned template. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > Sent: Wednesday, October 07, 2009 4:53 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] limiting message size while forwarding messages > > > > thanks Rainer, I will try that.So that means if I use the default > > template > > for forwarding, i think it should just work then. I did actually get a > > message from an application running on my box(uses standard syslog api) > > which was around 4K in size > > rsyslog did the right thing and truncated it to 2K.and logged into the > > file > > as per the configured rule. > > > > now I have requirement to support messages up to 5K while logging to > > the > > local file, but truncating it to 1K while sending it out. > > > > > > On Wed, Oct 7, 2009 at 8:13 PM, Rainer Gerhards > > wrote: > > > > > you need to look at the property replacer, and do something like > > > %msg:1:950% > > > in your template. I am not sure, but I think the default template > > already > > > does this. Did you experience oversize messages on a receiver? > > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > > Sent: Wednesday, October 07, 2009 4:36 PM > > > > To: rsyslog-users > > > > Subject: [rsyslog] limiting message size while forwarding messages > > > > > > > > Hello All,RFC 3164 says that max rsyslog message length is 1024 > > bytes. > > > > Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 > > and > > > > has a > > > > default max message size of 2K. > > > > Now, I don't want to restrict the size while receiving > > messages.However > > > > when > > > > I forward message out of the box, I want to truncate it > > > > to 1K, as I am not sure what implementations I will have to interop > > > > with. > > > > I could not find an easy way to do this. $MaxMessageSize seems to > > > > control > > > > both sending and receiving. > > > > Is there a way in which I can achieve this? > > > > Regards, > > > > Sayan > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From jbondc at openmv.com Thu Oct 8 03:25:21 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Wed, 7 Oct 2009 21:25:21 -0400 Subject: [rsyslog] Queue System Message-ID: <007901ca47b6$34ac0870$9e041950$@com> Hi everyone, I'm been experimenting with rsyslogd to see if I could avoid losing messages/events when the central server is rebooted. I'll start with what works, then what doesn't L Both the server & client are running rsyslogd 3.22.1 The client has the following configuration: $ModLoad imklog.so # kernel logging $ModLoad imuxsock.so # provides support for local system logging $WorkDirectory /var/spool/rsyslog $ActionQueueType Disk $ActionQueueFileName srvrfwd $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on *.err;*.crit;*.emerg;*.warn;auth.*;user.notice @@server.domain.com # Stop the central server [root at server] $ service rsyslogd stop # Start client logger [root at client] $ service rsyslogd start ** (client) syslog(test a, NOTICE) ** (client) syslog(test b, NOTICE) Since the central server is down I expect that the client will keep messages in the disk queue and send them when the server is back online. # Start the central server [root at server] $ service rsyslogd start . 2009-10-08T00:44:21-04:00 client php: test a 2009-10-08T00:45:39-04:00 client php: test b . Great it works! ****** Now the problem, when client and server are both connected and exchanging messages # Stop the central server [root at server] $ service rsyslogd stop ** (client) syslog(test c - lost message, NOTICE) With some debugging: 4241.782860452:action 10 queue:Reg/w0: action 10 queue: entry deleted, state 0, size now 0 entries 4241.782860452:action 10 queue:Reg/w0: server 4241.782860452:action 10 queue:Reg/w0: server:10514/tcp 4241.782860452:action 10 queue:Reg/w0: TCP sent 36 bytes, requested 36 The client 'sends the message over tcp' and actually empties the queue! Is this expected behavior? rsyslogd was shutdown 'cleanly' on the server. Shouldn't it force the client to close the TCP connection? Using netstat, the two servers are : Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 client.10514 server..60726 FIN_WAIT_2 Proto Recv-Q Send-Q Local Address Foreign Address State tcp 1 36 server:60726 client:10514 CLOSE_WAIT Have I misunderstood something? I've read Rainer's blog and issues with TCP "unreliability" / case of a power failure but don't think this related From david at lang.hm Thu Oct 8 06:58:21 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 7 Oct 2009 21:58:21 -0700 (PDT) Subject: [rsyslog] Queue System In-Reply-To: <007901ca47b6$34ac0870$9e041950$@com> References: <007901ca47b6$34ac0870$9e041950$@com> Message-ID: On Wed, 7 Oct 2009, Jonathan Bond-Caron wrote: > The client 'sends the message over tcp' and actually empties the queue! > > Is this expected behavior? rsyslogd was shutdown 'cleanly' on the server. > Shouldn't it force the client to close the TCP connection? > > > > Using netstat, the two servers are : > > > > Proto Recv-Q Send-Q Local Address Foreign Address (state) > > tcp4 0 0 client.10514 server..60726 FIN_WAIT_2 > > > > Proto Recv-Q Send-Q Local Address Foreign Address > State > > tcp 1 36 server:60726 client:10514 CLOSE_WAIT > > > > Have I misunderstood something? I've read Rainer's blog and issues with TCP > "unreliability" / case of a power failure but don't think this related I think it's the same problem, one side thinks the connection is closed, the other doesn't and so the one that doesn't happily sends the data out. try reconfiguring to use relp and things should work as expected. now, I don't know why the close isn't getting through from the one system to the other rapidly. is there any sort of firewall in between them? David Lang From rgerhards at hq.adiscon.com Thu Oct 8 09:27:30 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 09:27:30 +0200 Subject: [rsyslog] limiting message size while forwarding messages References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com><87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Wednesday, October 07, 2009 8:12 PM > To: rsyslog-users > Subject: Re: [rsyslog] limiting message size while forwarding messages > > Yes, I saw that too ... I made the template based out of Rsyslog > traditional > forward format and then used the property replacer as you mentioned. > It works like a charm. Thanks a bunch! I am glad it works, but I also tend to think that the default template is probably not correct. As it claims to be "traditional" format, I think it should really limit itself to 1K message size, so that another template must be picked if messages of "non-traditional large size" are to be transmitted. However, changing that default would potentially break a number of existing deployments. Does anybody else have an opinion on that? Thanks, Rainer > Regards, > Sayan > > > On Wed, Oct 7, 2009 at 10:07 PM, Rainer Gerhards > wrote: > > > I just checked, and the default is to send the full message. See: > > > > > > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/syslogd.c;h=0f4f8a > 233c97 > > a44e72c97a555ff6f95eb367573a;hb=HEAD#l382 > > > > But all you need to do is use %msg:1:950...% in a template that > otherwise > > is > > the same as the canned template. > > > > Rainer > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > Sent: Wednesday, October 07, 2009 4:53 PM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] limiting message size while forwarding > messages > > > > > > thanks Rainer, I will try that.So that means if I use the default > > > template > > > for forwarding, i think it should just work then. I did actually > get a > > > message from an application running on my box(uses standard syslog > api) > > > which was around 4K in size > > > rsyslog did the right thing and truncated it to 2K.and logged into > the > > > file > > > as per the configured rule. > > > > > > now I have requirement to support messages up to 5K while logging > to > > > the > > > local file, but truncating it to 1K while sending it out. > > > > > > > > > On Wed, Oct 7, 2009 at 8:13 PM, Rainer Gerhards > > > wrote: > > > > > > > you need to look at the property replacer, and do something like > > > > %msg:1:950% > > > > in your template. I am not sure, but I think the default template > > > already > > > > does this. Did you experience oversize messages on a receiver? > > > > > > > > > -----Original Message----- > > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > > > Sent: Wednesday, October 07, 2009 4:36 PM > > > > > To: rsyslog-users > > > > > Subject: [rsyslog] limiting message size while forwarding > messages > > > > > > > > > > Hello All,RFC 3164 says that max rsyslog message length is 1024 > > > bytes. > > > > > Rsyslog version 4.2.0 which I am using, is compliant to rfc > 5424 > > > and > > > > > has a > > > > > default max message size of 2K. > > > > > Now, I don't want to restrict the size while receiving > > > messages.However > > > > > when > > > > > I forward message out of the box, I want to truncate it > > > > > to 1K, as I am not sure what implementations I will have to > interop > > > > > with. > > > > > I could not find an easy way to do this. $MaxMessageSize seems > to > > > > > control > > > > > both sending and receiving. > > > > > Is there a way in which I can achieve this? > > > > > Regards, > > > > > Sayan > > > > > _______________________________________________ > > > > > rsyslog mailing list > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > http://www.rsyslog.com > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Thu Oct 8 09:41:16 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 8 Oct 2009 00:41:16 -0700 (PDT) Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com><87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 8 Oct 2009, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury >> >> Yes, I saw that too ... I made the template based out of Rsyslog >> traditional >> forward format and then used the property replacer as you mentioned. >> It works like a charm. Thanks a bunch! > > I am glad it works, but I also tend to think that the default template is > probably not correct. As it claims to be "traditional" format, I think it > should really limit itself to 1K message size, so that another template must > be picked if messages of "non-traditional large size" are to be transmitted. > > However, changing that default would potentially break a number of existing > deployments. > > Does anybody else have an opinion on that? existing syslog implementations already need to be able to deal with oversized messages (almost nothing checks the size before it gets sent), so I don't see a big benifit in changing rsyslog to limit what it outputs. David Lang From rgerhards at hq.adiscon.com Thu Oct 8 09:43:47 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 09:43:47 +0200 Subject: [rsyslog] Queue System References: <007901ca47b6$34ac0870$9e041950$@com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710319B@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, October 08, 2009 6:58 AM > To: rsyslog-users > Subject: Re: [rsyslog] Queue System > > On Wed, 7 Oct 2009, Jonathan Bond-Caron wrote: > > > The client 'sends the message over tcp' and actually empties the > queue! > > > > Is this expected behavior? rsyslogd was shutdown 'cleanly' on the > server. > > Shouldn't it force the client to close the TCP connection? > > > > > > > > Using netstat, the two servers are : > > > > > > > > Proto Recv-Q Send-Q Local Address Foreign Address > (state) > > > > tcp4 0 0 client.10514 server..60726 FIN_WAIT_2 > > > > > > > > Proto Recv-Q Send-Q Local Address Foreign Address > > State > > > > tcp 1 36 server:60726 client:10514 > CLOSE_WAIT > > > > > > > > Have I misunderstood something? I've read Rainer's blog and issues > with TCP > > "unreliability" / case of a power failure but don't think this > related > > I think it's the same problem, one side thinks the connection is > closed, > the other doesn't and so the one that doesn't happily sends the data > out. > > try reconfiguring to use relp and things should work as expected. > > > now, I don't know why the close isn't getting through from the one > system > to the other rapidly. is there any sort of firewall in between them? The close may get through. It is a (kind of) race condition, inside the tcp stack. Assume the following happens (System S being the server, system C being the client): S : closes connection, sends close info C1: rsyslog writes data (not yet sent, but acknowledged by tcp stack) C2: receives close request C3: discards data The race is in C1/C2. In this order, data is lost. If C2 happens before C1, no data is lost. No chance to solve that with plain tcp without app-level acks. Rainer From david at lang.hm Thu Oct 8 09:48:08 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 8 Oct 2009 00:48:08 -0700 (PDT) Subject: [rsyslog] Queue System In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710319B@GRFEXC.intern.adiscon.com> References: <007901ca47b6$34ac0870$9e041950$@com> <9B6E2A8877C38245BFB15CC491A11DA710319B@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 8 Oct 2009, Rainer Gerhards wrote: >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >> On Wed, 7 Oct 2009, Jonathan Bond-Caron wrote: >> >>> The client 'sends the message over tcp' and actually empties the >> queue! >>> >>> Is this expected behavior? rsyslogd was shutdown 'cleanly' on the >> server. >>> Shouldn't it force the client to close the TCP connection? >>> >>> >>> >>> Using netstat, the two servers are : >>> >>> >>> >>> Proto Recv-Q Send-Q Local Address Foreign Address >> (state) >>> >>> tcp4 0 0 client.10514 server..60726 FIN_WAIT_2 >>> >>> >>> >>> Proto Recv-Q Send-Q Local Address Foreign Address >>> State >>> >>> tcp 1 36 server:60726 client:10514 >> CLOSE_WAIT >>> >>> >>> >>> Have I misunderstood something? I've read Rainer's blog and issues >> with TCP >>> "unreliability" / case of a power failure but don't think this >> related >> >> I think it's the same problem, one side thinks the connection is >> closed, >> the other doesn't and so the one that doesn't happily sends the data >> out. >> >> try reconfiguring to use relp and things should work as expected. >> >> >> now, I don't know why the close isn't getting through from the one >> system >> to the other rapidly. is there any sort of firewall in between them? > > The close may get through. It is a (kind of) race condition, inside the tcp > stack. Assume the following happens (System S being the server, system C > being the client): > > S : closes connection, sends close info > C1: rsyslog writes data (not yet sent, but acknowledged by tcp stack) > C2: receives close request > C3: discards data > > The race is in C1/C2. In this order, data is lost. If C2 happens before C1, > no data is lost. No chance to solve that with plain tcp without app-level > acks. but that window should be very short, for it to last long enough to show up in the netstat commands sounds odd to me. David Lang From rgerhards at hq.adiscon.com Thu Oct 8 09:51:39 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 09:51:39 +0200 Subject: [rsyslog] Queue System References: <007901ca47b6$34ac0870$9e041950$@com><9B6E2A8877C38245BFB15CC491A11DA710319B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710319D@GRFEXC.intern.adiscon.com> > -----Original Message----- > > The close may get through. It is a (kind of) race condition, inside > the tcp > > stack. Assume the following happens (System S being the server, > system C > > being the client): > > > > S : closes connection, sends close info > > C1: rsyslog writes data (not yet sent, but acknowledged by tcp stack) > > C2: receives close request > > C3: discards data > > > > The race is in C1/C2. In this order, data is lost. If C2 happens > before C1, > > no data is lost. No chance to solve that with plain tcp without app- > level > > acks. > > but that window should be very short, for it to last long enough to > show > up in the netstat commands sounds odd to me. Ah, I didn't pay attention to the netstat. You are right, in *this* case the close looks like it does not go through. HOWEVER, for the argument I have given, this race exists in general. The window is extremely short (at least on a local LAN), but I have learned if there is a potential for a race, it will happen sooner or later. Chances go up very soon as quickly as you have millions and millions of cases... Just yesterday I was able to find a race with a much lower probability in v5-beta during shutdown... And it really happened (thankfully only in my lab, where I set parameters to make such races more probable). Rainer From martinmie at PartyGaming.com Thu Oct 8 10:50:13 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 8 Oct 2009 10:50:13 +0200 Subject: [rsyslog] URGENT - rsyslog stops sshd Message-ID: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Hi all, odd behavior here... Today one of our core systems stopped accepting ssh connections. -- $ telnet sshserver 22 Trying 1.2.3.4... Connected to sshserver. Escape character is '^]'. Connection closed. -- Using a pre-established session rsyslog was shutdown: -- # /etc/init.d/rsyslog stop Shutting down system logger: [ OK ][/quote] -- ..and the system started accepting connections again: -- $ ssh sshserver Last login: Thu Oct 8 01:12:59 2009 from foo.bar.tld -- We are just one step before dropping rsyslog and use syslog-ng again (which was dismissed some time ago to favor rsyslog) but I would like to know what's the reason for this and, most important, what's the solution :-) Some additional info: -- # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 32767 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 32767 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited -- Thanks! Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From david at lang.hm Thu Oct 8 11:16:44 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 8 Oct 2009 02:16:44 -0700 (PDT) Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Message-ID: On Thu, 8 Oct 2009, Martin Mielke wrote: > Hi all, > > odd behavior here... > > Today one of our core systems stopped accepting ssh connections. > -- > $ telnet sshserver 22 > Trying 1.2.3.4... > Connected to sshserver. > Escape character is '^]'. > Connection closed. > -- > > Using a pre-established session rsyslog was shutdown: > -- > # /etc/init.d/rsyslog stop > Shutting down system logger: [ OK > ][/quote] > -- > > ..and the system started accepting connections again: > -- > $ ssh sshserver > Last login: Thu Oct 8 01:12:59 2009 from foo.bar.tld > -- > > We are just one step before dropping rsyslog and use syslog-ng again > (which was dismissed some time ago to favor rsyslog) but I would like to > know what's the reason for this and, most important, what's the solution > :-) ssh logs connections, if the syslog process cannot process the message, ssh is designed to stop and wait until it does (it deems the log so important that it refuses to do anything until the log is written) conole logins do the same thing. so something causes rsyslog to stop accepting messages. the same thing can happen to syslog-ng or to plain sysklog, just under different conditions. so the question is why rsyslog stopped. what is rsyslog configured to do with log messages? is there any chance that it was unable to do something with a message and so would have had to keep it in it's queue until the queue filled up? (logging via TCP to a remote server that stops responding will do this, so will writing to a full filesystem) rsyslog can be configured to accept and discard log entries when the queue is full, doing this can avoid this sort of situation. David Lang > Some additional info: > -- > # ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 32767 > max locked memory (kbytes, -l) 32 > max memory size (kbytes, -m) unlimited > open files (-n) 1024 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 10240 > cpu time (seconds, -t) unlimited > max user processes (-u) 32767 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > -- > > > Thanks! > > Martin > > > This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. > > Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iamsayan at gmail.com Thu Oct 8 11:16:14 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Thu, 8 Oct 2009 05:16:14 -0400 Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540910080216y1caef6begd013ff6b722e76ef@mail.gmail.com> I agree ... As long as rsyslog has a mechanism to do the truncation if somebody wants on the egress(which it has), it is fine. I don't think the default template should change. BTW, there are implementations out there which behaves badly when they receive large messages/control characters etc. I want to be defensive about what send out and hence my requirement. On Thu, Oct 8, 2009 at 3:41 AM, wrote: > On Thu, 8 Oct 2009, Rainer Gerhards wrote: > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > >> > >> Yes, I saw that too ... I made the template based out of Rsyslog > >> traditional > >> forward format and then used the property replacer as you mentioned. > >> It works like a charm. Thanks a bunch! > > > > I am glad it works, but I also tend to think that the default template is > > probably not correct. As it claims to be "traditional" format, I think it > > should really limit itself to 1K message size, so that another template > must > > be picked if messages of "non-traditional large size" are to be > transmitted. > > > > However, changing that default would potentially break a number of > existing > > deployments. > > > > Does anybody else have an opinion on that? > > existing syslog implementations already need to be able to deal with > oversized messages (almost nothing checks the size before it gets sent), > so I don't see a big benifit in changing rsyslog to limit what it outputs. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu Oct 8 11:26:15 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 8 Oct 2009 02:26:15 -0700 (PDT) Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <87a1ae540910080216y1caef6begd013ff6b722e76ef@mail.gmail.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> <87a1ae540910080216y1caef6begd013ff6b722e76ef@mail.gmail.com> Message-ID: On Thu, 8 Oct 2009, Sayan Chowdhury wrote: > I agree ... As long as rsyslog has a mechanism to do the truncation if > somebody wants on the egress(which it has), it is fine. I don't think the > default template should change. > BTW, there are implementations out there which behaves badly when they > receive large messages/control characters etc. I want to be defensive about > what send out and hence my requirement. which implementation caused you grief? (so I know what to avoid) David Lang > > > > On Thu, Oct 8, 2009 at 3:41 AM, wrote: > >> On Thu, 8 Oct 2009, Rainer Gerhards wrote: >> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury >>>> >>>> Yes, I saw that too ... I made the template based out of Rsyslog >>>> traditional >>>> forward format and then used the property replacer as you mentioned. >>>> It works like a charm. Thanks a bunch! >>> >>> I am glad it works, but I also tend to think that the default template is >>> probably not correct. As it claims to be "traditional" format, I think it >>> should really limit itself to 1K message size, so that another template >> must >>> be picked if messages of "non-traditional large size" are to be >> transmitted. >>> >>> However, changing that default would potentially break a number of >> existing >>> deployments. >>> >>> Does anybody else have an opinion on that? >> >> existing syslog implementations already need to be able to deal with >> oversized messages (almost nothing checks the size before it gets sent), >> so I don't see a big benifit in changing rsyslog to limit what it outputs. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Oct 8 11:28:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 11:28:56 +0200 Subject: [rsyslog] limiting message size while forwarding messages References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com><87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com><87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com><87a1ae540910080216y1caef6begd013ff6b722e76ef@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031A2@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, October 08, 2009 11:26 AM > To: rsyslog-users > Subject: Re: [rsyslog] limiting message size while forwarding messages > > On Thu, 8 Oct 2009, Sayan Chowdhury wrote: > > > I agree ... As long as rsyslog has a mechanism to do the truncation > if > > somebody wants on the egress(which it has), it is fine. I don't think > the > > default template should change. > > BTW, there are implementations out there which behaves badly when > they > > receive large messages/control characters etc. I want to be defensive > about > > what send out and hence my requirement. > > which implementation caused you grief? (so I know what to avoid) > The one I know about is darn old Solaris syslogd, which reliably died with messages of 1025 bytes and above. However, I think this is fixed for 10 years or so now ;) I've seen no other problems with implementations, but I definitely have not seen everything ;) Rainer > David Lang > > > > > > > > > On Thu, Oct 8, 2009 at 3:41 AM, wrote: > > > >> On Thu, 8 Oct 2009, Rainer Gerhards wrote: > >> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > >>>> > >>>> Yes, I saw that too ... I made the template based out of Rsyslog > >>>> traditional > >>>> forward format and then used the property replacer as you > mentioned. > >>>> It works like a charm. Thanks a bunch! > >>> > >>> I am glad it works, but I also tend to think that the default > template is > >>> probably not correct. As it claims to be "traditional" format, I > think it > >>> should really limit itself to 1K message size, so that another > template > >> must > >>> be picked if messages of "non-traditional large size" are to be > >> transmitted. > >>> > >>> However, changing that default would potentially break a number of > >> existing > >>> deployments. > >>> > >>> Does anybody else have an opinion on that? > >> > >> existing syslog implementations already need to be able to deal with > >> oversized messages (almost nothing checks the size before it gets > sent), > >> so I don't see a big benifit in changing rsyslog to limit what it > outputs. > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From dag at wieers.com Thu Oct 8 12:36:38 2009 From: dag at wieers.com (Dag Wieers) Date: Thu, 8 Oct 2009 12:36:38 +0200 (CEST) Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Message-ID: On Thu, 8 Oct 2009, david at lang.hm wrote: > rsyslog can be configured to accept and discard log entries when the queue > is full, doing this can avoid this sort of situation. Hi Martin, We were in a similar situation recently (because of a limit of open file descriptors) and I am very interested to learn how you can do the above. Thanks in advance, -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] From martinmie at PartyGaming.com Thu Oct 8 13:28:36 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 8 Oct 2009 13:28:36 +0200 Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Message-ID: <0B1B9163138571439EAF804646F3F6AA08771181@GIBSVWIN004X.partygaming.local> Hi Dag, The number of open file descriptors wasn't the issue here... and I'm also interested in how to configure rsyslog as described by David ;-) Cheers, Martin > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dag Wieers > Sent: 08 October 2009 12:37 > To: rsyslog-users > Subject: Re: [rsyslog] URGENT - rsyslog stops sshd > > On Thu, 8 Oct 2009, david at lang.hm wrote: > > > rsyslog can be configured to accept and discard log entries when the > queue > > is full, doing this can avoid this sort of situation. > > Hi Martin, > > We were in a similar situation recently (because of a limit of open file > descriptors) and I am very interested to learn how you can do the above. > > Thanks in advance, > -- > -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- > [Any errors in spelling, tact or fact are transmission errors] > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > nfo/rsyslog > http://www.rsyslog.com This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From martinmie at PartyGaming.com Thu Oct 8 13:32:23 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 8 Oct 2009 13:32:23 +0200 Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Message-ID: <0B1B9163138571439EAF804646F3F6AA08771183@GIBSVWIN004X.partygaming.local> Hi, > > ssh logs connections, if the syslog process cannot process the message, > ssh is designed to stop and wait until it does (it deems the log so > important that it refuses to do anything until the log is written) > > conole logins do the same thing. Is there any way I can test this? Strace? Ltrace? Sshd configuration directive?... > > so something causes rsyslog to stop accepting messages. the same thing can > happen to syslog-ng or to plain sysklog, just under different conditions. > > so the question is why rsyslog stopped. > > what is rsyslog configured to do with log messages? > > is there any chance that it was unable to do something with a message and > so would have had to keep it in it's queue until the queue filled up? > (logging via TCP to a remote server that stops responding will do this, so > will writing to a full filesystem) The system sends its logs over TCP to a remote logserver. Initial analysis yield a routing problem which should have been fixed so I hope not to see such a problem due to this. But, my question now is: shouldn't logfiles be spooled under $WorkDirectory /var/spool/rsyslog and sent over to the logserver once it becomes available again? There's plenty of space on /var ... > > rsyslog can be configured to accept and discard log entries when the queue > is full, doing this can avoid this sort of situation. Could you please tell us how? :-) Regards, Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From rgerhards at hq.adiscon.com Thu Oct 8 14:11:45 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 14:11:45 +0200 Subject: [rsyslog] URGENT - rsyslog stops sshd References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031A5@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dag Wieers > Sent: Thursday, October 08, 2009 12:37 PM > To: rsyslog-users > Subject: Re: [rsyslog] URGENT - rsyslog stops sshd > > On Thu, 8 Oct 2009, david at lang.hm wrote: > > > rsyslog can be configured to accept and discard log entries when the > queue > > is full, doing this can avoid this sort of situation. > > Hi Martin, > > We were in a similar situation recently (because of a limit of open > file > descriptors) and I am very interested to learn how you can do the > above. Well, it depends on your configuration. If you use a plain default config, this situation is expected to never happen, because we assume that log files can be written and discard messages when this is not the case. However, if you begin to configure the system to be reliable, you need to think about the implications. In almost all cases where I have seen such a problem (assuming it was not a real bug, of course), the configuration demanded reliable delivery, but it was not thought about the fact that if the message could not be delivered, the system would stall. Some organizations actually prefer this mode, so it may be useful. If you post your config, I could check if there is something in it that demands such reliability and prevents the engine from discarding messages when needed. Rainer From martinmie at PartyGaming.com Thu Oct 8 14:24:06 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 8 Oct 2009 14:24:06 +0200 Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031A5@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> <9B6E2A8877C38245BFB15CC491A11DA71031A5@GRFEXC.intern.adiscon.com> Message-ID: <0B1B9163138571439EAF804646F3F6AA087711B7@GIBSVWIN004X.partygaming.local> Hi Rainer, This is the rsyslog.conf used - quite simple as you can see: -- $ cat /etc/rsyslog.conf # rsyslog v3: load input modules # If you do not load inputs, nothing happens! # You may need to set the module load path if modules are not found. $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # provides kernel logging support (previously done by rklogd) # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none -/var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* -/var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit -/var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. $WorkDirectory /var/spool/rsyslog # where to place spool files $ActionQueueFileName uniqName # unique name prefix for spool files $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) $ActionQueueSaveOnShutdown on # save messages to disk on shutdown $ActionQueueType LinkedList # run asynchronously $ActionResumeRetryCount 5 # five retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional *.* @@logserver -- Do you see anything suspicious? Thanks, Martin > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: 08 October 2009 14:12 > To: rsyslog-users > Subject: Re: [rsyslog] URGENT - rsyslog stops sshd > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Dag Wieers > > Sent: Thursday, October 08, 2009 12:37 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] URGENT - rsyslog stops sshd > > > > On Thu, 8 Oct 2009, david at lang.hm wrote: > > > > > rsyslog can be configured to accept and discard log entries when the > > queue > > > is full, doing this can avoid this sort of situation. > > > > Hi Martin, > > > > We were in a similar situation recently (because of a limit of open > > file > > descriptors) and I am very interested to learn how you can do the > > above. > > Well, it depends on your configuration. If you use a plain default config, > this situation is expected to never happen, because we assume that log > files > can be written and discard messages when this is not the case. However, if > you begin to configure the system to be reliable, you need to think about > the > implications. In almost all cases where I have seen such a problem > (assuming > it was not a real bug, of course), the configuration demanded reliable > delivery, but it was not thought about the fact that if the message could > not > be delivered, the system would stall. Some organizations actually prefer > this > mode, so it may be useful. > > If you post your config, I could check if there is something in it that > demands such reliability and prevents the engine from discarding messages > when needed. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > /rsyslog > http://www.rsyslog.com This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From jbondc at openmv.com Thu Oct 8 15:17:44 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Thu, 8 Oct 2009 09:17:44 -0400 Subject: [rsyslog] Queue System In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710319D@GRFEXC.intern.adiscon.com> References: <007901ca47b6$34ac0870$9e041950$@com><9B6E2A8877C38245BFB15CC491A11DA710319B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710319D@GRFEXC.intern.adiscon.com> Message-ID: <000e01ca4819$b942b760$2bc82620$@com> On Thu Oct 8 03:51 AM, Rainer Gerhards wrote: > > -----Original Message----- > > > The close may get through. It is a (kind of) race condition, > > > inside > > the tcp > > > stack. Assume the following happens (System S being the server, > > system C > > > being the client): > > Ah, I didn't pay attention to the netstat. You are right, in *this* > case the close looks like it does not go through. HOWEVER, for the > argument I have given, this race exists in general. The window is > extremely short (at least on a local LAN), but I have learned if there > is a potential for a race, it will happen sooner or later. Chances go > up very soon as quickly as you have millions and millions of cases... > Just yesterday I was able to find a race with a much lower probability > in v5-beta during shutdown... And it really happened (thankfully only > in my lab, where I set parameters to make such races more probable). > I decided to look at the code and I must say, wow documented open source code! I haven't written c in a while but easily found my way, so big congrats on keeping the code so clean. After some googling, I found that whenever TCP is in status CLOSE_WAIT, the host is expected to close the socket sometime soon. It might not want to close it right away for example to keep-alive the connection... So I searched for all close() calls on the socket. I found sockClose() then static void CheckConnection(nsd_t *pNsd) -- runtime/nsd_ptcp.c http://blog.gerhards.net/2008/06/getting-bit-more-reliability-from-plain.htm l With debug mode I found: 5543.432507331:action 10 queue:Reg/w0: server:10514/tcp 5543.432507331:action 10 queue:Reg/w0: TCP sent 36 bytes, requested 36 It turns out I was pointing to the SSL config which had: $DefaultNetstreamDriver gtls For this, I found: void CheckConnection(nsd_t *pNsd) -- runtime/nsd_gtls.c /* This function checks if the connection is still alive - well, kind of... * This is a dummy here. For details, check function common in ptcp driver. * rgerhards, 2008-06-09 */ static void CheckConnection(nsd_t *pNsd) { dbgprintf("CheckConnection SSL - do something\n"); nsd_gtls_t *pThis = (nsd_gtls_t*) pNsd; ISOBJ_TYPE_assert(pThis, nsd_gtls); nsd_ptcp.CheckConnection(pThis->pTcp); } 5649.998580185:action 10 queue:Reg/w0: server:10514/tcp 5649.998580185:action 10 queue:Reg/w0: CheckConnection SSL - do something 5649.998580185:action 10 queue:Reg/w0: CheckConnection detected broken connection - closing it Wonderful! It works as I would expect. But, is there any reason why the socket check wasn't added for SSL? I'm currently testing this 'patch' with a couple of 'live' servers and randomly stopping the central logging server. From dirk.schulz at kinzesberg.de Thu Oct 8 17:58:51 2009 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Thu, 08 Oct 2009 17:58:51 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> References: <4ACCA646.8050209@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> Message-ID: <4ACE0C3B.7090207@kinzesberg.de> Rainer Gerhards schrieb: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Dirk H. Schulz >> Sent: Wednesday, October 07, 2009 4:32 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] Getting Facility from Synlog-NG >> >> Hi folks, >> >> I am not sure if this is the right list to ask, but I hope that there >> is >> more of the related knowledge here. >> >> I have several Syslog-NG servers reporting to Rsyslog central servers >> (just because nobody could tell me how to run Rsyslog on NetBSD ...). >> Now I fould that the facility is not transferred to/evaluated on the >> Rsyslog servers; e.g. mail messages are written to syslog instead of >> maillog on the Rsyslog machines. >> > > This is strange. Facility is a fairly standard field. Could you tell me the > content of %rawmwg%. This is done along these lines (if it does not work, > check doc for exact syntax): > > $template raw,"%rawmsg%\n" > *.* /path/to/file;raw > I did not find these terms, neither in rsyslog.conf on the central logserver servers nor in syslog-ng.conf on the log client servers. Rsyslog is 3.18.6-4 on Debian 5. Dirk > > Rainer > >> Is there something special I have to configure on the Rsyslog servers >> or >> the Syslog-NG servers to fix that? >> >> Any hint or help is appreciated. >> >> Dirk >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Oct 8 18:03:54 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 18:03:54 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG References: <4ACCA646.8050209@kinzesberg.de><9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> <4ACE0C3B.7090207@kinzesberg.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031AC@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dirk H. Schulz > Sent: Thursday, October 08, 2009 5:59 PM > To: rsyslog-users > Subject: Re: [rsyslog] Getting Facility from Synlog-NG > > Rainer Gerhards schrieb: > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Dirk H. Schulz > >> Sent: Wednesday, October 07, 2009 4:32 PM > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] Getting Facility from Synlog-NG > >> > >> Hi folks, > >> > >> I am not sure if this is the right list to ask, but I hope that > there > >> is > >> more of the related knowledge here. > >> > >> I have several Syslog-NG servers reporting to Rsyslog central > servers > >> (just because nobody could tell me how to run Rsyslog on NetBSD > ...). > >> Now I fould that the facility is not transferred to/evaluated on the > >> Rsyslog servers; e.g. mail messages are written to syslog instead of > >> maillog on the Rsyslog machines. > >> > > > > This is strange. Facility is a fairly standard field. Could you tell > me the > > content of %rawmwg%. This is done along these lines (if it does not > work, > > check doc for exact syntax): > > > > $template raw,"%rawmsg%\n" > > *.* /path/to/file;raw > > > > I did not find these terms, neither in rsyslog.conf on the central > logserver servers nor in syslog-ng.conf on the log client servers. > Rsyslog is 3.18.6-4 on Debian 5. Well, these are the config statements for the central rsyslog instance... Rainer From dirk.schulz at kinzesberg.de Thu Oct 8 18:22:49 2009 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Thu, 08 Oct 2009 18:22:49 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031AC@GRFEXC.intern.adiscon.com> References: <4ACCA646.8050209@kinzesberg.de><9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> <4ACE0C3B.7090207@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA71031AC@GRFEXC.intern.adiscon.com> Message-ID: <4ACE11D9.1020806@kinzesberg.de> Rainer Gerhards schrieb: >>> check doc for exact syntax): >>> >>> $template raw,"%rawmsg%\n" >>> *.* /path/to/file;raw >>> >>> >> I did not find these terms, neither in rsyslog.conf on the central >> logserver servers nor in syslog-ng.conf on the log client servers. >> Rsyslog is 3.18.6-4 on Debian 5. >> > > Well, these are the config statements for the central rsyslog instance... > Does that mean that I am lacking a template statement in the central rsyslog config? Dirk > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iamsayan at gmail.com Thu Oct 8 18:48:33 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Thu, 8 Oct 2009 22:18:33 +0530 Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031A2@GRFEXC.intern.adiscon.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> <87a1ae540910080216y1caef6begd013ff6b722e76ef@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71031A2@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540910080948t1859f0dav90e3cd15c1f951b7@mail.gmail.com> it was an in house properietary "data mining" implementation. I don't know why people try to reinvent the wheel :) On Thu, Oct 8, 2009 at 2:58 PM, Rainer Gerhards wrote: > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Thursday, October 08, 2009 11:26 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] limiting message size while forwarding messages > > > > On Thu, 8 Oct 2009, Sayan Chowdhury wrote: > > > > > I agree ... As long as rsyslog has a mechanism to do the truncation > > if > > > somebody wants on the egress(which it has), it is fine. I don't think > > the > > > default template should change. > > > BTW, there are implementations out there which behaves badly when > > they > > > receive large messages/control characters etc. I want to be defensive > > about > > > what send out and hence my requirement. > > > > which implementation caused you grief? (so I know what to avoid) > > > > The one I know about is darn old Solaris syslogd, which reliably died with > messages of 1025 bytes and above. However, I think this is fixed for 10 > years > or so now ;) I've seen no other problems with implementations, but I > definitely have not seen everything ;) > > Rainer > > > David Lang > > > > > > > > > > > > > > On Thu, Oct 8, 2009 at 3:41 AM, wrote: > > > > > >> On Thu, 8 Oct 2009, Rainer Gerhards wrote: > > >> > > >>>> -----Original Message----- > > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >>>> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > >>>> > > >>>> Yes, I saw that too ... I made the template based out of Rsyslog > > >>>> traditional > > >>>> forward format and then used the property replacer as you > > mentioned. > > >>>> It works like a charm. Thanks a bunch! > > >>> > > >>> I am glad it works, but I also tend to think that the default > > template is > > >>> probably not correct. As it claims to be "traditional" format, I > > think it > > >>> should really limit itself to 1K message size, so that another > > template > > >> must > > >>> be picked if messages of "non-traditional large size" are to be > > >> transmitted. > > >>> > > >>> However, changing that default would potentially break a number of > > >> existing > > >>> deployments. > > >>> > > >>> Does anybody else have an opinion on that? > > >> > > >> existing syslog implementations already need to be able to deal with > > >> oversized messages (almost nothing checks the size before it gets > > sent), > > >> so I don't see a big benifit in changing rsyslog to limit what it > > outputs. > > >> > > >> David Lang > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iamsayan at gmail.com Thu Oct 8 18:51:45 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Thu, 8 Oct 2009 22:21:45 +0530 Subject: [rsyslog] Getting Facility from Synlog-NG In-Reply-To: <4ACE11D9.1020806@kinzesberg.de> References: <4ACCA646.8050209@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> <4ACE0C3B.7090207@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA71031AC@GRFEXC.intern.adiscon.com> <4ACE11D9.1020806@kinzesberg.de> Message-ID: <87a1ae540910080951g6c7106fby53979808642cb537@mail.gmail.com> Hello Dirk, I think Rainer wants you to add this to your rsyslog.conf file, and send him the file that gets saved because of adding this statement. So as to debug the issue. You will need to restart rsyslog to pick up the config change Hope this helps. Sayan On Thu, Oct 8, 2009 at 9:52 PM, Dirk H. Schulz wrote: > Rainer Gerhards schrieb: > >>> check doc for exact syntax): > >>> > >>> $template raw,"%rawmsg%\n" > >>> *.* /path/to/file;raw > >>> > >>> > >> I did not find these terms, neither in rsyslog.conf on the central > >> logserver servers nor in syslog-ng.conf on the log client servers. > >> Rsyslog is 3.18.6-4 on Debian 5. > >> > > > > Well, these are the config statements for the central rsyslog instance... > > > Does that mean that I am lacking a template statement in the central > rsyslog config? > > Dirk > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu Oct 8 20:31:09 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 8 Oct 2009 11:31:09 -0700 (PDT) Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: <0B1B9163138571439EAF804646F3F6AA087711B7@GIBSVWIN004X.partygaming.local> References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> <9B6E2A8877C38245BFB15CC491A11DA71031A5@GRFEXC.intern.adiscon.com> <0B1B9163138571439EAF804646F3F6AA087711B7@GIBSVWIN004X.partygaming.local> Message-ID: On Thu, 8 Oct 2009, Martin Mielke wrote: > Hi Rainer, > > This is the rsyslog.conf used - quite simple as you can see: > -- > $ cat /etc/rsyslog.conf > # rsyslog v3: load input modules > # If you do not load inputs, nothing happens! > # You may need to set the module load path if modules are not found. > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # provides kernel logging support (previously done by > rklogd) > > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none > -/var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* > -/var/log/maillog > > > # Log cron stuff > cron.* -/var/log/cron > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit > -/var/log/spooler > > # Save boot messages also to boot.log > local7.* > /var/log/boot.log > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/spool/rsyslog # where to place spool files > $ActionQueueFileName uniqName # unique name prefix for spool files > $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > possible) > $ActionQueueSaveOnShutdown on # save messages to disk on shutdown > $ActionQueueType LinkedList # run asynchronously > $ActionResumeRetryCount 5 # five retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > *.* @@logserver > -- > > Do you see anything suspicious? this TCP logging would cause it. you have it limited to 1g of disk space, how large did it get? also, disk queues are pretty slow, it may be that you had messages arriving faster than they could be put into the queue also, you still have it using the default main queue size (something tiny like 1000 messages IIRC) I would bump the main queue size up significantly (assuming you can afford the ram) also, look at the high and low watermark config variables, they (along with one other variable I don't remember offhand) tell the system what to do in crisis mode and how to define that mode. unfortunantly I don't have time until late today to go through the config page and dig up the exact details. David Lang From dag at wieers.com Thu Oct 8 21:10:22 2009 From: dag at wieers.com (Dag Wieers) Date: Thu, 8 Oct 2009 21:10:22 +0200 (CEST) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole Message-ID: Hi, We have been looking at why we don't see messages from netconsole ending up in our logfiles. netconsole is a kernel module that sends information of kernel panics to a syslog server. The documentation indicates it should work with syslog as an endpoint but that is not true with rsyslog, apparently :) https://bugzilla.redhat.com/show_bug.cgi?id=432160 Also, I have noticed that our central rsyslog-server (more than 400 systems log to it) have directories with names like: Detected/, exiting/, ext3_abort/, EXT3-fs/, journal/, last/, martian/, program/, Remounting/, Restarting/, ... So it is obvious that something is not working as expected coming from the kernel. >From information I found online, a message formatted like: <4>Testing through netcat should work for testing the syslog connectivity, but it is clear that rsyslog only accepts: <4>Oct 7 12:34:56 hostname Testing but of course I cannot influence our production kernels to do the right thing. What can I do to have rsyslog accept the "wrong" thing ? :) Advice is appreciated :) -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] From aoz.syn at gmail.com Thu Oct 8 21:28:38 2009 From: aoz.syn at gmail.com (RB) Date: Thu, 8 Oct 2009 13:28:38 -0600 Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: References: Message-ID: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> On Thu, Oct 8, 2009 at 13:10, Dag Wieers wrote: > We have been looking at why we don't see messages from netconsole ending > up in our logfiles. netconsole is a kernel module that sends information > of kernel panics to a syslog server. The documentation indicates it should > work with syslog as an endpoint but that is not true with rsyslog, > apparently :) Mostly because said syslog daemons are not only very loose in their interpretation of a valid syslog entry, but in their default configurations they also tend to silently interpolate data sloppy senders leave out. > Also, I have noticed that our central rsyslog-server (more than 400 > systems log to it) have directories with names like: Detected/, exiting/, > ext3_abort/, EXT3-fs/, journal/, last/, martian/, program/, Remounting/, > Restarting/, ... So it is obvious that something is not working as > expected coming from the kernel. More than likely because you're trying to make directories based on the %hostname% property, which rsyslog assumes is a specific field. With the sloppier daemons (FreeBSD in particular), I've had far more luck using the %fromhost-ip% property (as well as the $ system properties for timestamps). Of course, that breaks down if you're doing relaying, but relying on values the other end sends you to create filesystem artifacts is dangerous at best anyway > but of course I cannot influence our production kernels to do the right > thing. What can I do to have rsyslog accept the "wrong" thing ? :) Use %fromhost% or %fromhost-ip% to make the directory structures/filenames, and make a custom format if you need to handle the remaning lack of data (again, timestamp & host). I'm sure there are many other ways to approach it, but that's the way I've solved it. From mrdemeanour at jackpot.uk.net Fri Oct 9 13:59:52 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 09 Oct 2009 12:59:52 +0100 Subject: [rsyslog] Deviant PRI values causing dropped messages Message-ID: <4ACF25B8.3040002@jackpot.uk.net> Hi, I have a Speedtouch router that produces firewall events whose PRI value is <84> (auth.notice, according to RFC3164). I have a script that retrieves events from the router via Telnet, and passes them on to rsyslog via UDP. The firewall events with PRI=<84> are dropped. If I modify the script so that they are submitted to rsyslog with a PRI of <36> (facility changed from 10 to 4), they are logged. They are also logged if the facility is 13 ("audit"?). According to RFC3614, facilities numbers 4 and 10 are both "auth". I haven't inspected the source, but I suspect that rsyslog doesn't like facility number 10, and is ignoring it. Is this a correct interpretation? Why should it object to this facility in particular? Regards, -- Jack. From mrdemeanour at jackpot.uk.net Fri Oct 9 14:08:38 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 09 Oct 2009 13:08:38 +0100 Subject: [rsyslog] Deviant PRI values causing dropped messages In-Reply-To: <4ACF25B8.3040002@jackpot.uk.net> References: <4ACF25B8.3040002@jackpot.uk.net> Message-ID: <4ACF27C6.5080705@jackpot.uk.net> Mr. Demeanour wrote: > Hi, > > I have a Speedtouch router that produces firewall events whose PRI value > is <84> (auth.notice, according to RFC3164). Sorry; that should be "auth.warn". -- Jack. From rgerhards at hq.adiscon.com Fri Oct 9 14:10:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 9 Oct 2009 14:10:27 +0200 Subject: [rsyslog] Deviant PRI values causing dropped messages References: <4ACF25B8.3040002@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031BB@GRFEXC.intern.adiscon.com> Hi Jack, rsyslog does not do any processing per se on the message. It only does what you instruct it to do. So there is probably something in your rsyslog.conf that does either not process messages with this facility or drops them. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Friday, October 09, 2009 2:00 PM > To: rsyslog-users > Subject: [rsyslog] Deviant PRI values causing dropped messages > > Hi, > > I have a Speedtouch router that produces firewall events whose PRI > value > is <84> (auth.notice, according to RFC3164). I have a script that > retrieves events from the router via Telnet, and passes them on to > rsyslog via UDP. > > The firewall events with PRI=<84> are dropped. If I modify the script > so > that they are submitted to rsyslog with a PRI of <36> (facility changed > from 10 to 4), they are logged. They are also logged if the facility is > 13 ("audit"?). > > According to RFC3614, facilities numbers 4 and 10 are both "auth". I > haven't inspected the source, but I suspect that rsyslog doesn't like > facility number 10, and is ignoring it. Is this a correct > interpretation? Why should it object to this facility in particular? From mrdemeanour at jackpot.uk.net Fri Oct 9 14:15:21 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 09 Oct 2009 13:15:21 +0100 Subject: [rsyslog] Deviant PRI values causing dropped messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031BB@GRFEXC.intern.adiscon.com> References: <4ACF25B8.3040002@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71031BB@GRFEXC.intern.adiscon.com> Message-ID: <4ACF2959.2020707@jackpot.uk.net> Rainer Gerhards wrote: > Hi Jack, > > rsyslog does not do any processing per se on the message. It only > does what you instruct it to do. So there is probably something in > your rsyslog.conf that does either not process messages with this > facility or drops them. Oh, thanks, Rainer. I had this entry in the config: authpriv.info ~ Commenting it out fixed the problem. -- Jack. From tbergfeld at hq.adiscon.com Fri Oct 9 14:34:49 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 9 Oct 2009 14:34:49 +0200 Subject: [rsyslog] rsyslog 4.4.2 (v4-stable) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031BE@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.4.2, a member of the v4-stable branch. This is a bug-fixing release containing some important fixes. It is a recommended update for all users of the v4-stabled. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article409.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-179.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From dag at wieers.com Fri Oct 9 14:38:01 2009 From: dag at wieers.com (Dag Wieers) Date: Fri, 9 Oct 2009 14:38:01 +0200 (CEST) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> Message-ID: On Thu, 8 Oct 2009, RB wrote: > On Thu, Oct 8, 2009 at 13:10, Dag Wieers wrote: > >> Also, I have noticed that our central rsyslog-server (more than 400 >> systems log to it) have directories with names like: Detected/, exiting/, >> ext3_abort/, EXT3-fs/, journal/, last/, martian/, program/, Remounting/, >> Restarting/, ... So it is obvious that something is not working as >> expected coming from the kernel. > > More than likely because you're trying to make directories based on > the %hostname% property, which rsyslog assumes is a specific field. > With the sloppier daemons (FreeBSD in particular), I've had far more > luck using the %fromhost-ip% property (as well as the $ system > properties for timestamps). Of course, that breaks down if you're > doing relaying, but relying on values the other end sends you to > create filesystem artifacts is dangerous at best anyway You are correct, that is exactly what we do. However with rsyslog v2.0.6 it seems there is no %FROMHOST-IP% and the %FROMHOST% property only contains the IP address. Maybe there is something else I need to do to get the short hostname from DNS, rather than an IP on rsyslog v2 ? >> but of course I cannot influence our production kernels to do the right >> thing. What can I do to have rsyslog accept the "wrong" thing ? :) > > Use %fromhost% or %fromhost-ip% to make the directory > structures/filenames, and make a custom format if you need to handle > the remaning lack of data (again, timestamp & host). I'm sure there > are many other ways to approach it, but that's the way I've solved it. Thanks for the feedback. I hope more people can chime into this. -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] From rgerhards at hq.adiscon.com Fri Oct 9 14:47:39 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 9 Oct 2009 14:47:39 +0200 Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031BF@GRFEXC.intern.adiscon.com> Unfortunately, 2.0.6 is a very old and deprecated version. I really don't dig any longer into that code base (except, of course, for folks with support contracts). It really is not worth the hassle. You'll run into a couple of issues in the long term. I suggest upgrading at least to v3-stable. Project status is here: http://www.rsyslog.com/doc-status.html Sorry I have no better reply, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dag Wieers > Sent: Friday, October 09, 2009 2:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] Wrongly formatted messages from > kernel/netconsole > > On Thu, 8 Oct 2009, RB wrote: > > > On Thu, Oct 8, 2009 at 13:10, Dag Wieers wrote: > > > >> Also, I have noticed that our central rsyslog-server (more than 400 > >> systems log to it) have directories with names like: Detected/, > exiting/, > >> ext3_abort/, EXT3-fs/, journal/, last/, martian/, program/, > Remounting/, > >> Restarting/, ... So it is obvious that something is not working as > >> expected coming from the kernel. > > > > More than likely because you're trying to make directories based on > > the %hostname% property, which rsyslog assumes is a specific field. > > With the sloppier daemons (FreeBSD in particular), I've had far more > > luck using the %fromhost-ip% property (as well as the $ system > > properties for timestamps). Of course, that breaks down if you're > > doing relaying, but relying on values the other end sends you to > > create filesystem artifacts is dangerous at best anyway > > You are correct, that is exactly what we do. However with rsyslog > v2.0.6 > it seems there is no %FROMHOST-IP% and the %FROMHOST% property only > contains the IP address. Maybe there is something else I need to do to > get > the short hostname from DNS, rather than an IP on rsyslog v2 ? > > > >> but of course I cannot influence our production kernels to do the > right > >> thing. What can I do to have rsyslog accept the "wrong" thing ? :) > > > > Use %fromhost% or %fromhost-ip% to make the directory > > structures/filenames, and make a custom format if you need to handle > > the remaning lack of data (again, timestamp & host). I'm sure there > > are many other ways to approach it, but that's the way I've solved > it. > > Thanks for the feedback. I hope more people can chime into this. > > -- > -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- > [Any errors in spelling, tact or fact are transmission errors] > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From dag at wieers.com Fri Oct 9 16:17:27 2009 From: dag at wieers.com (Dag Wieers) Date: Fri, 9 Oct 2009 16:17:27 +0200 (CEST) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031BF@GRFEXC.intern.adiscon.com> References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71031BF@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 9 Oct 2009, Rainer Gerhards wrote: > Unfortunately, 2.0.6 is a very old and deprecated version. I really don't dig > any longer into that code base (except, of course, for folks with support > contracts). It really is not worth the hassle. You'll run into a couple of > issues in the long term. I suggest upgrading at least to v3-stable. Project > status is here: > > http://www.rsyslog.com/doc-status.html > > Sorry I have no better reply, Don't feel sorry, I understand. We are looking with Red Hat to rebase rsyslog to a newer version asap. We expect by RHEL5.5 to have either v3 or v4. No word yet from Red Hat what their target will be for RHEL5.5. (Hopefully the same as RHEL6) This is related to the bugzilla reports I send to the list recently. But it would have been nice to know whether there was a solution for this on v2 as well, even when I accept that it is old and unsupported here :-) Kind regards, -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] From aoz.syn at gmail.com Fri Oct 9 16:19:30 2009 From: aoz.syn at gmail.com (RB) Date: Fri, 9 Oct 2009 08:19:30 -0600 Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> Message-ID: <4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> On Fri, Oct 9, 2009 at 06:38, Dag Wieers wrote: > You are correct, that is exactly what we do. However with rsyslog v2.0.6 > it seems there is no %FROMHOST-IP% and the %FROMHOST% property only > contains the IP address. Maybe there is something else I need to do to get > the short hostname from DNS, rather than an IP on rsyslog v2 ? This usually has more to do with the host's DNS configuration than the syslog daemon - I've not looked at the 2.x code (ever), but if your host is configured with a resolver that will perform reverse lookups of the IPs and with an appropriate domain search list, you should be able to get the short hostnames. That can be shortcut by placing entries in /etc/hosts, but that obviously doesn't scale very well. From dag at wieers.com Fri Oct 9 16:36:23 2009 From: dag at wieers.com (Dag Wieers) Date: Fri, 9 Oct 2009 16:36:23 +0200 (CEST) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: <4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> <4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> Message-ID: On Fri, 9 Oct 2009, RB wrote: > On Fri, Oct 9, 2009 at 06:38, Dag Wieers wrote: >> You are correct, that is exactly what we do. However with rsyslog v2.0.6 >> it seems there is no %FROMHOST-IP% and the %FROMHOST% property only >> contains the IP address. Maybe there is something else I need to do to get >> the short hostname from DNS, rather than an IP on rsyslog v2 ? > > This usually has more to do with the host's DNS configuration than the > syslog daemon - I've not looked at the 2.x code (ever), but if your > host is configured with a resolver that will perform reverse lookups > of the IPs and with an appropriate domain search list, you should be > able to get the short hostnames. That can be shortcut by placing > entries in /etc/hosts, but that obviously doesn't scale very well. We do have a working DNS resolver and reverse lookups work fine. I assume that the problem is that with rsyslog v2 %FROMHOST% simply is not resolved. To be sure, I will have to dig into the sources. -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] From david at lang.hm Sat Oct 10 08:06:17 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 9 Oct 2009 23:06:17 -0700 (PDT) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: <4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> <4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> Message-ID: On Fri, 9 Oct 2009, RB wrote: > On Fri, Oct 9, 2009 at 06:38, Dag Wieers wrote: >> You are correct, that is exactly what we do. However with rsyslog v2.0.6 >> it seems there is no %FROMHOST-IP% and the %FROMHOST% property only >> contains the IP address. Maybe there is something else I need to do to get >> the short hostname from DNS, rather than an IP on rsyslog v2 ? > > This usually has more to do with the host's DNS configuration than the > syslog daemon - I've not looked at the 2.x code (ever), but if your > host is configured with a resolver that will perform reverse lookups > of the IPs and with an appropriate domain search list, you should be > able to get the short hostnames. That can be shortcut by placing > entries in /etc/hosts, but that obviously doesn't scale very well. even the 4.x and 5.x do not populate the fromhost-ip if you have -x on the command line. I think this is a bug, but there are bigger bugs that are being worked on. David Lang From rgerhards at hq.adiscon.com Mon Oct 12 16:01:23 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 12 Oct 2009 16:01:23 +0200 Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com><4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031DA@GRFEXC.intern.adiscon.com> David, All, > even the 4.x and 5.x do not populate the fromhost-ip if you have -x on > the > command line. > > I think this is a bug, but there are bigger bugs that are being worked > on. Does anybody have a debug log where this happens? I do not see it in my lab (but I did not try hard). For the DNS cache, I am shuffling code in v5, and that would be a good time to address the bug. Raienr From marc.schiffbauer at mightycare.de Tue Oct 13 13:20:45 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 13 Oct 2009 13:20:45 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? Message-ID: <200910131320.45629.marc.schiffbauer@mightycare.de> Hello all, I am running a rsyslog server version 4.5.2 I tested the message queuing capabilities recently and noticed that queue files do not get cleaned up. Evertime rsyslog has to spool something (e.g. when the database server ist stopped) it creates new spoolfiles so the spool area will grow over time. Is this a bug or how can I handle those old queue files? This is the relevant part from my config: # enable message spooling $WorkDirectory /data/syslog-spool # default location for work (spool) files $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName dbq # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure TIA -Marc Schiffbauer -- Senior Consultant :: Solution Architect IT-Security :: Free Software :: GNU/Linux MightyCare Solutions GmbH http://www.mightycare.de Firmenangaben unter http://www.mightycare.de/#/de/impressum/ From rgerhards at hq.adiscon.com Tue Oct 13 14:16:47 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 13 Oct 2009 14:16:47 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com> This looks like a dirty shutdown. In general, rsyslog cleans up when processing completes. Do you have a .qi file inside that directory? Absence of a .qi file - when the instance is not running - is indication of an issue during shutdown. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, October 13, 2009 1:21 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Hello all, > > I am running a rsyslog server version 4.5.2 > > I tested the message queuing capabilities recently and noticed that > queue > files do not get cleaned up. Evertime rsyslog has to spool something > (e.g. > when the database server ist stopped) it creates new spoolfiles so the > spool > area will grow over time. > > Is this a bug or how can I handle those old queue files? > > This is the relevant part from my config: > > # enable message spooling > $WorkDirectory /data/syslog-spool # default location for work (spool) > files > $ActionQueueType LinkedList # use asynchronous processing > $ActionQueueFileName dbq # set file name, also enables disk > mode > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > TIA > -Marc Schiffbauer > -- > Senior Consultant :: Solution Architect > IT-Security :: Free Software :: GNU/Linux > MightyCare Solutions GmbH http://www.mightycare.de > Firmenangaben unter http://www.mightycare.de/#/de/impressum/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Oct 13 15:31:44 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 13 Oct 2009 15:31:44 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com> Message-ID: <200910131531.44817.marc.schiffbauer@mightycare.de> Hello Rainer, Am Dienstag, 13. Oktober 2009 14:16:47 schrieb Rainer Gerhards: > This looks like a dirty shutdown. I am not aware of any... > In general, rsyslog cleans up when > processing completes. OK. This is good to know. So something went wrong here: I produced a lot of logging traffic using the logger shell command. Then I stopped that postgres server to test that spooling works. After a while I started it again. After a while rsyslog reconnected to the db and logging to the DB was working again. > Do you have a .qi file inside that directory? Absence > of a .qi file - when the instance is not running - is indication of an > issue during shutdown. There is a .qi file when rsyslog is NOT running and ist being removed when I start it again. I will remove stop rsyslog now, then remove *everything* from the spool- directory and start it up again. Then I will repeat the spool-test. Thanks for helping -Marc > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > > Sent: Tuesday, October 13, 2009 1:21 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > > Hello all, > > > > I am running a rsyslog server version 4.5.2 > > > > I tested the message queuing capabilities recently and noticed that > > queue > > files do not get cleaned up. Evertime rsyslog has to spool something > > (e.g. > > when the database server ist stopped) it creates new spoolfiles so the > > spool > > area will grow over time. > > > > Is this a bug or how can I handle those old queue files? > > > > This is the relevant part from my config: > > > > # enable message spooling > > $WorkDirectory /data/syslog-spool # default location for work (spool) > > files > > $ActionQueueType LinkedList # use asynchronous processing > > $ActionQueueFileName dbq # set file name, also enables disk > > mode > > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > > > > TIA > > -Marc Schiffbauer -- Senior Consultant :: Solution Architect IT-Security :: Free Software :: GNU/Linux Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 MightyCare Solutions GmbH http://www.mightycare.de Firmenangaben unter http://www.mightycare.de/#/de/impressum/ From marc.schiffbauer at mightycare.de Tue Oct 13 15:37:52 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 13 Oct 2009 15:37:52 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <200910131531.44817.marc.schiffbauer@mightycare.de> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com> <200910131531.44817.marc.schiffbauer@mightycare.de> Message-ID: <200910131537.52917.marc.schiffbauer@mightycare.de> Am Dienstag, 13. Oktober 2009 15:31:44 schrieb Marc Schiffbauer: > > Do you have a .qi file inside that directory? Absence > > of a .qi file - when the instance is not running - is indication of an > > issue during shutdown. > > There is a .qi file when rsyslog is NOT running and ist being removed when > I start it again. > Now this is strange: To prepare the new test I stopped rsyslog again, and now it did NOT create the .qi file. Another restart: same result. This rsyslog is idle ATM BTW. -Marc -- Senior Consultant :: Solution Architect IT-Security :: Free Software :: GNU/Linux Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 MightyCare Solutions GmbH http://www.mightycare.de Firmenangaben unter http://www.mightycare.de/#/de/impressum/ From rgerhards at hq.adiscon.com Tue Oct 13 15:40:07 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 13 Oct 2009 15:40:07 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com><200910131531.44817.marc.schiffbauer@mightycare.de> <200910131537.52917.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031E6@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, October 13, 2009 3:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Am Dienstag, 13. Oktober 2009 15:31:44 schrieb Marc Schiffbauer: > > > Do you have a .qi file inside that directory? Absence > > > of a .qi file - when the instance is not running - is indication of > an > > > issue during shutdown. > > > > There is a .qi file when rsyslog is NOT running and ist being removed > when > > I start it again. > > > > Now this is strange: To prepare the new test I stopped rsyslog again, > and now > it did NOT create the .qi file. Another restart: same result. > > This rsyslog is idle ATM BTW. Both sounds like there simply was nothing in the queue... > > -Marc > -- > Senior Consultant :: Solution Architect > IT-Security :: Free Software :: GNU/Linux > Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 > MightyCare Solutions GmbH http://www.mightycare.de > Firmenangaben unter http://www.mightycare.de/#/de/impressum/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Oct 13 15:41:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 13 Oct 2009 15:41:17 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com> <200910131531.44817.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031E7@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, October 13, 2009 3:32 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > Am Dienstag, 13. Oktober 2009 14:16:47 schrieb Rainer Gerhards: > > This looks like a dirty shutdown. > > I am not aware of any... > > > In general, rsyslog cleans up when > > processing completes. > > OK. This is good to know. So something went wrong here: I produced a > lot of > logging traffic using the logger shell command. Then I stopped that > postgres > server to test that spooling works. After a while I started it again. > After a while rsyslog reconnected to the db and logging to the DB was > working > again. > > > > Do you have a .qi file inside that directory? Absence > > of a .qi file - when the instance is not running - is indication of > an > > issue during shutdown. > > There is a .qi file when rsyslog is NOT running and ist being removed > when I > start it again. That is OK, because then rsyslog keeps the information in memory once it runs, to safe i/o time. You can modify that by specifying a different queue configuration, but you need to understand how the queue system works in order to do so. A good read is the queue doc. Rainer > > I will remove stop rsyslog now, then remove *everything* from the > spool- > directory and start it up again. > > Then I will repeat the spool-test. > > Thanks for helping > -Marc > > > > > Rainer > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > > > Sent: Tuesday, October 13, 2009 1:21 PM > > > To: rsyslog at lists.adiscon.com > > > Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > > > > Hello all, > > > > > > I am running a rsyslog server version 4.5.2 > > > > > > I tested the message queuing capabilities recently and noticed that > > > queue > > > files do not get cleaned up. Evertime rsyslog has to spool > something > > > (e.g. > > > when the database server ist stopped) it creates new spoolfiles so > the > > > spool > > > area will grow over time. > > > > > > Is this a bug or how can I handle those old queue files? > > > > > > This is the relevant part from my config: > > > > > > # enable message spooling > > > $WorkDirectory /data/syslog-spool # default location for work > (spool) > > > files > > > $ActionQueueType LinkedList # use asynchronous processing > > > $ActionQueueFileName dbq # set file name, also enables > disk > > > mode > > > $ActionResumeRetryCount -1 # infinite retries on insert > failure > > > > > > > > > TIA > > > -Marc Schiffbauer > > -- > Senior Consultant :: Solution Architect > IT-Security :: Free Software :: GNU/Linux > Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 > MightyCare Solutions GmbH http://www.mightycare.de > Firmenangaben unter http://www.mightycare.de/#/de/impressum/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Oct 13 16:27:44 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 13 Oct 2009 16:27:44 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031E7@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <200910131531.44817.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA71031E7@GRFEXC.intern.adiscon.com> Message-ID: <200910131627.44897.marc.schiffbauer@mightycare.de> Am Dienstag, 13. Oktober 2009 15:41:17 schrieb Rainer Gerhards: > > > > There is a .qi file when rsyslog is NOT running and ist being removed > > when I > > start it again. > > That is OK, because then rsyslog keeps the information in memory once it > runs, to safe i/o time. You can modify that by specifying a different queue > configuration, but you need to understand how the queue system works in > order to do so. A good read is the queue doc. > > Rainer > > > I will remove stop rsyslog now, then remove *everything* from the > > spool- > > directory and start it up again. > > > > Then I will repeat the spool-test. > > As it seems, rsyslog will not write a .qi file in all cases. New tests were not all successful (with rsyslog under load): OK => spooling while DB is offline OK => reconnect to DB OK => despooling while still under load and spooling to disk Now the following produced "stale" queue files and a loss of messages I guess: NOT OK => despooling while under load and while spooling to disk, then stopping rsyslogd (stopped via /etc/init.d/syslog stop) -> no .qi file has been created! after making sure there are no more rsyslog processes I started it again. The spool files will not be cleared (no load anymore and DB started of course) bug? TIA -Marc -- Senior Consultant :: Solution Architect IT-Security :: Free Software :: GNU/Linux Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 MightyCare Solutions GmbH http://www.mightycare.de Firmenangaben unter http://www.mightycare.de/#/de/impressum/ From rgerhards at hq.adiscon.com Tue Oct 13 16:35:49 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 13 Oct 2009 16:35:49 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><200910131531.44817.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA71031E7@GRFEXC.intern.adiscon.com> <200910131627.44897.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031EA@GRFEXC.intern.adiscon.com> > As it seems, rsyslog will not write a .qi file in all cases. not always, but it should always write them when necessary ;) > > New tests were not all successful (with rsyslog under load): > OK => spooling while DB is offline > OK => reconnect to DB > OK => despooling while still under load and spooling to disk > > Now the following produced "stale" queue files and a loss of messages I > guess: > NOT OK => despooling while under load and while spooling to disk, > then stopping rsyslogd > (stopped via /etc/init.d/syslog stop) > -> no .qi file has been created! > after making sure there are no more rsyslog processes I started it > again. > The spool files will not be cleared (no load anymore and DB started of > course) > > bug? Smells like one. I re-checked your config, I think it does not include a directive to tell the engine to persist messages on shutdown. Even if it does not do that, it should clean up the files. A debug log would be most useful. Note that the v4 engine, and more so the v5 engine, has had a number of important changes, and people only gradually begin to utilize it in practice. The past couple of month, I had comparatively few bug reports, but the past three weeks or so people tend to adopt the new features and consequently the "bug rate" goes up. This is good, as it helps iron out things, but it is also somewhat bad, because I need to prioritize work and for some bugs that means I have not to touch too many things at once. Looking forward for additional information. Rainer PS: please all keep contributing bug reports! It is really useful to have them (even better if they are timely ;)) as my lab can not cover everything practice does ;) > > TIA > -Marc > -- > Senior Consultant :: Solution Architect > IT-Security :: Free Software :: GNU/Linux > Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 > MightyCare Solutions GmbH http://www.mightycare.de > Firmenangaben unter http://www.mightycare.de/#/de/impressum/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Oct 14 00:40:14 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 13 Oct 2009 15:40:14 -0700 (PDT) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031DA@GRFEXC.intern.adiscon.com> References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com><4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71031DA@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 12 Oct 2009, Rainer Gerhards wrote: > David, All, > >> even the 4.x and 5.x do not populate the fromhost-ip if you have -x on >> the >> command line. >> >> I think this is a bug, but there are bigger bugs that are being worked >> on. > > Does anybody have a debug log where this happens? I do not see it in my lab > (but I did not try hard). For the DNS cache, I am shuffling code in v5, and > that would be a good time to address the bug. you should have it in the debug logs I've made available for the segfault problems. Those instances are running with -x David Lang From rgerhards at hq.adiscon.com Wed Oct 14 07:14:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 14 Oct 2009 07:14:44 +0200 Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com><4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71031DA@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031F1@GRFEXC.intern.adiscon.com> thanks, then I know where to look at :) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, October 14, 2009 12:40 AM > To: rsyslog-users > Subject: Re: [rsyslog] Wrongly formatted messages from > kernel/netconsole > > On Mon, 12 Oct 2009, Rainer Gerhards wrote: > > > David, All, > > > >> even the 4.x and 5.x do not populate the fromhost-ip if you have -x > on > >> the > >> command line. > >> > >> I think this is a bug, but there are bigger bugs that are being > worked > >> on. > > > > Does anybody have a debug log where this happens? I do not see it in > my lab > > (but I did not try hard). For the DNS cache, I am shuffling code in > v5, and > > that would be a good time to address the bug. > > you should have it in the debug logs I've made available for the > segfault > problems. Those instances are running with -x > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From tbergfeld at hq.adiscon.com Thu Oct 15 10:59:17 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Thu, 15 Oct 2009 10:59:17 +0200 Subject: [rsyslog] rsyslog 5.1.6 (v5-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103210@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.1.6, a member of the v5-beta branch. This is a bug-fixing release containing some important fixes. It is a recommended update for all users of the v5-beta. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article413.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-180.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From marc.schiffbauer at mightycare.de Thu Oct 15 14:14:33 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Thu, 15 Oct 2009 14:14:33 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? Message-ID: <200910151414.33860.marc.schiffbauer@mightycare.de> [ sorry, attachement in first try was too big, next try with shortened debug log ] Am Dienstag, 13. Oktober 2009 16:35:49 schrieb Rainer Gerhards: > > As it seems, rsyslog will not write a .qi file in all cases. > > not always, but it should always write them when necessary ;) > > > New tests were not all successful (with rsyslog under load): > > OK => spooling while DB is offline > > OK => reconnect to DB > > OK => despooling while still under load and spooling to disk > > > > Now the following produced "stale" queue files and a loss of messages I > > guess: > > NOT OK => despooling while under load and while spooling to disk, > > then stopping rsyslogd > > (stopped via /etc/init.d/syslog stop) > > -> no .qi file has been created! > > after making sure there are no more rsyslog processes I started it > > again. > > The spool files will not be cleared (no load anymore and DB started of > > course) > > > > bug? > > Smells like one. I re-checked your config, I think it does not include a > directive to tell the engine to persist messages on shutdown. Oh, I did not know that there is a directive for it.. good to know. > Even if it > does not do that, it should clean up the files. A debug log would be most > useful. Trying to produce one it seems a difficult task to me because rsyslogd seems to behave completely different when in debug mode... In one console I started: /sbin/rsyslogd -c 4 -f /etc/rsyslog.conf -d &> rsyslog-debug.log Then I stopped postgres: /etc/init.d/postgresql stop Now I started a logger loop: while true; do logger -t spool-test "no real message here... (PID=$$)"; done After the first spool file is being created I stop the loop again. Now I want to stop rsyslogd by calling "killall rsyslogd" I had to call this several times before rsyslogd really did exit. After the first attemots it seemed that it did try to reach the DB in a loop and did not attempt to prepare for exit. See the log attached. Is this helpful? -Marc > > Note that the v4 engine, and more so the v5 engine, has had a number of > important changes, and people only gradually begin to utilize it in > practice. The past couple of month, I had comparatively few bug reports, > but the past three weeks or so people tend to adopt the new features and > consequently the "bug rate" goes up. This is good, as it helps iron out > things, but it is also somewhat bad, because I need to prioritize work and > for some bugs that means I have not to touch too many things at once. > > Looking forward for additional information. > > Rainer > PS: please all keep contributing bug reports! It is really useful to have > them (even better if they are timely ;)) as my lab can not cover everything > practice does ;) > -- Senior Consultant :: Solution Architect IT-Security :: Free Software :: GNU/Linux From rgerhards at hq.adiscon.com Thu Oct 15 14:21:42 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 15 Oct 2009 14:21:42 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910151414.33860.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710321F@GRFEXC.intern.adiscon.com> Hi Marc, the mailing list is set to be very restrictive in what it accepts. So the attachment did not went through ;) You can mail it to me at rgerhards at gmail.com - but please let me know when you have done, I do not regularly check that mailbox. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Thursday, October 15, 2009 2:15 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > [ sorry, attachement in first try was too big, next try with shortened > debug > log ] > > Am Dienstag, 13. Oktober 2009 16:35:49 schrieb Rainer Gerhards: > > > As it seems, rsyslog will not write a .qi file in all cases. > > > > not always, but it should always write them when necessary ;) > > > > > New tests were not all successful (with rsyslog under load): > > > OK => spooling while DB is offline > > > OK => reconnect to DB > > > OK => despooling while still under load and spooling to disk > > > > > > Now the following produced "stale" queue files and a loss of > messages I > > > guess: > > > NOT OK => despooling while under load and while spooling to disk, > > > then stopping rsyslogd > > > (stopped via /etc/init.d/syslog stop) > > > -> no .qi file has been created! > > > after making sure there are no more rsyslog processes I started it > > > again. > > > The spool files will not be cleared (no load anymore and DB started > of > > > course) > > > > > > bug? > > > > Smells like one. I re-checked your config, I think it does not > include a > > directive to tell the engine to persist messages on shutdown. > > Oh, I did not know that there is a directive for it.. good to know. > > > Even if it > > does not do that, it should clean up the files. A debug log would be > most > > useful. > > Trying to produce one it seems a difficult task to me because rsyslogd > seems > to behave completely different when in debug mode... > > In one console I started: > /sbin/rsyslogd -c 4 -f /etc/rsyslog.conf -d &> rsyslog-debug.log > > Then I stopped postgres: > /etc/init.d/postgresql stop > > Now I started a logger loop: > while true; do logger -t spool-test "no real message here... > (PID=$$)"; > done > > After the first spool file is being created I stop the loop again. > > Now I want to stop rsyslogd by calling "killall rsyslogd" > > I had to call this several times before rsyslogd really did exit. After > the > first attemots it seemed that it did try to reach the DB in a loop and > did not > attempt to prepare for exit. > > See the log attached. Is this helpful? > > -Marc > > > > > Note that the v4 engine, and more so the v5 engine, has had a number > of > > important changes, and people only gradually begin to utilize it in > > practice. The past couple of month, I had comparatively few bug > reports, > > but the past three weeks or so people tend to adopt the new features > and > > consequently the "bug rate" goes up. This is good, as it helps iron > out > > things, but it is also somewhat bad, because I need to prioritize > work and > > for some bugs that means I have not to touch too many things at > once. > > > > Looking forward for additional information. > > > > Rainer > > PS: please all keep contributing bug reports! It is really useful to > have > > them (even better if they are timely ;)) as my lab can not cover > everything > > practice does ;) > > > -- > Senior Consultant :: Solution Architect > IT-Security :: Free Software :: GNU/Linux From marc.schiffbauer at mightycare.de Thu Oct 15 19:33:11 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Thu, 15 Oct 2009 19:33:11 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710321F@GRFEXC.intern.adiscon.com> References: <200910151414.33860.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA710321F@GRFEXC.intern.adiscon.com> Message-ID: <200910151933.11296.marc.schiffbauer@mightycare.de> Am Donnerstag, 15. Oktober 2009 14:21:42 schrieb Rainer Gerhards: > Hi Marc, > > the mailing list is set to be very restrictive in what it accepts. So the > attachment did not went through ;) You can mail it to me at > rgerhards at gmail.com - but please let me know when you have done, I do not > regularly check that mailbox. Hi Rainer, mail sent! TIA -Marc From marc.schiffbauer at mightycare.de Fri Oct 16 12:52:26 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Fri, 16 Oct 2009 12:52:26 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031EA@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <200910131627.44897.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA71031EA@GRFEXC.intern.adiscon.com> Message-ID: <200910161252.26765.marc.schiffbauer@mightycare.de> Hi Rainer, Am Dienstag, 13. Oktober 2009 16:35:49 schrieb Rainer Gerhards: > > > > bug? > > Smells like one. I re-checked your config, I think it does not include a > directive to tell the engine to persist messages on shutdown. Even if it > does not do that, it should clean up the files. A debug log would be most > useful. I now added "$ActionQueueSaveOnShutdown on" to the configuration. But I still get queuefiles without a .qi file after rsyslog has quit by SIGTERM -Marc From rgerhards at hq.adiscon.com Fri Oct 16 14:07:23 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 16 Oct 2009 14:07:23 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><200910131627.44897.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA71031EA@GRFEXC.intern.adiscon.com> <200910161252.26765.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103231@GRFEXC.intern.adiscon.com> busy day today... I had a quick look at the debug log, it looks like there indeed is a shutdown issue. Out of curiosity, would you be willing to test a current (v5) devel version? I am currently working on an enhanced queue engine, which has been greatly simplified and speeded up. During that work, I notice that there exist some potential issue in v4, that I had thought to not surface in practice. Queue shutdown is very, very complicated. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Friday, October 16, 2009 12:52 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > Hi Rainer, > > Am Dienstag, 13. Oktober 2009 16:35:49 schrieb Rainer Gerhards: > > > > > > bug? > > > > Smells like one. I re-checked your config, I think it does not > include a > > directive to tell the engine to persist messages on shutdown. Even if > it > > does not do that, it should clean up the files. A debug log would be > most > > useful. > > I now added "$ActionQueueSaveOnShutdown on" to the configuration. But I > still > get queuefiles without a .qi file after rsyslog has quit by SIGTERM > > -Marc > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Fri Oct 16 15:55:40 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Fri, 16 Oct 2009 15:55:40 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103231@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <200910161252.26765.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103231@GRFEXC.intern.adiscon.com> Message-ID: <200910161555.40812.marc.schiffbauer@mightycare.de> Am Freitag, 16. Oktober 2009 14:07:23 schrieb Rainer Gerhards: > busy day today... I had a quick look at the debug log, it looks like there > indeed is a shutdown issue. > Do you think you can get it fixed in 4.5.4? I would be pleased to test patches... > Out of curiosity, would you be willing to test a current (v5) devel > version? As this is at a customers system I can do only a short test to see whether it works or not. > I am currently working on an enhanced queue engine, which has > been greatly simplified and speeded up. During that work, I notice that > there exist some potential issue in v4, that I had thought to not surface > in practice. Queue shutdown is very, very complicated. > I see. Sounds good for future versions.... Are these DA-queuing features are present in the 3.xx version already? Maybe I should consider using 3.xx the instead if its more robust. Thanks -Marc > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > > Sent: Friday, October 16, 2009 12:52 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > > > > Hi Rainer, > > > > Am Dienstag, 13. Oktober 2009 16:35:49 schrieb Rainer Gerhards: > > > > bug? > > > > > > Smells like one. I re-checked your config, I think it does not > > > > include a > > > > > directive to tell the engine to persist messages on shutdown. Even if > > > > it > > > > > does not do that, it should clean up the files. A debug log would be > > > > most > > > > > useful. > > > > I now added "$ActionQueueSaveOnShutdown on" to the configuration. But I > > still > > get queuefiles without a .qi file after rsyslog has quit by SIGTERM > > > > -Marc > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Oct 16 15:59:18 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 16 Oct 2009 15:59:18 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><200910161252.26765.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA7103231@GRFEXC.intern.adiscon.com> <200910161555.40812.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103234@GRFEXC.intern.adiscon.com> just quickly > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Friday, October 16, 2009 3:56 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Am Freitag, 16. Oktober 2009 14:07:23 schrieb Rainer Gerhards: > > busy day today... I had a quick look at the debug log, it looks like > there > > indeed is a shutdown issue. > > > > Do you think you can get it fixed in 4.5.4? I would be pleased to test > patches... > > > > Out of curiosity, would you be willing to test a current (v5) devel > > version? > > As this is at a customers system I can do only a short test to see > whether it > works or not. > > > > I am currently working on an enhanced queue engine, which has > > been greatly simplified and speeded up. During that work, I notice > that > > there exist some potential issue in v4, that I had thought to not > surface > > in practice. Queue shutdown is very, very complicated. > > > > I see. Sounds good for future versions.... > > > Are these DA-queuing features are present in the 3.xx version already? > Maybe I should consider using 3.xx the instead if its more robust. Indeed, they are. Raienr From marc.schiffbauer at mightycare.de Fri Oct 16 16:20:50 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Fri, 16 Oct 2009 16:20:50 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <200910161555.40812.marc.schiffbauer@mightycare.de> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103231@GRFEXC.intern.adiscon.com> <200910161555.40812.marc.schiffbauer@mightycare.de> Message-ID: <200910161620.50460.marc.schiffbauer@mightycare.de> Am Freitag, 16. Oktober 2009 15:55:40 schrieb Marc Schiffbauer: > Am Freitag, 16. Oktober 2009 14:07:23 schrieb Rainer Gerhards: > > busy day today... I had a quick look at the debug log, it looks like > > there indeed is a shutdown issue. > > Do you think you can get it fixed in 4.5.4? I would be pleased to test > patches... I meant 4.4.3 of course ;-) From marc.schiffbauer at mightycare.de Fri Oct 16 16:32:20 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Fri, 16 Oct 2009 16:32:20 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103234@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <200910161555.40812.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103234@GRFEXC.intern.adiscon.com> Message-ID: <200910161632.20392.marc.schiffbauer@mightycare.de> Am Freitag, 16. Oktober 2009 15:59:18 schrieb Rainer Gerhards: > just quickly > > > > > Are these DA-queuing features are present in the 3.xx version already? > > Maybe I should consider using 3.xx the instead if its more robust. > > Indeed, they are. But 3.22.1 seems to have the same issues with DA-spooling :-( I will do some more testing on monday. -Marc From rgerhards at hq.adiscon.com Fri Oct 16 16:39:16 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 16 Oct 2009 16:39:16 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><200910161555.40812.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA7103234@GRFEXC.intern.adiscon.com> <200910161632.20392.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103235@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Friday, October 16, 2009 4:32 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Am Freitag, 16. Oktober 2009 15:59:18 schrieb Rainer Gerhards: > > just quickly > > > > > > > > Are these DA-queuing features are present in the 3.xx version > already? > > > Maybe I should consider using 3.xx the instead if its more robust. > > > > Indeed, they are. > > But 3.22.1 seems to have the same issues with DA-spooling :-( > I will do some more testing on monday. One problem is that most people didn't use the enhanced queing modes until lately. So even while the versions were out for quite some while, and there were no bug reports, there seem to be some hidden issues. I have been receiving bug reports the past three month or so and I am working to get these things straight. I consider these reports to be a good sign, as it indicates the functionality gradually is adopted in practice. Right now, I am working on a much more streamlined queue engine in v5. Queue shutdown is a very complex operation, and I have been able to greatly simplify processing. Also, there has been a tremendous amount of work gone into performance optimization, where the simplified engine will also set new marks, I think. Of course, that doesn't help you immediately. I just thought I try to convey at least some important bits of "the big picture". Rainer From david at lang.hm Fri Oct 16 22:05:14 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 16 Oct 2009 13:05:14 -0700 (PDT) Subject: [rsyslog] disabling doc installation Message-ID: is there a configure option that will skip populating everything under the DATAROOT directory? while I see the value of having everything there by default, on my server builds I don't want to waste the space of having everything there, so I end up having to create the package, and then manually modify it. David Lang From sovrevage at gmail.com Mon Oct 19 12:46:17 2009 From: sovrevage at gmail.com (=?ISO-8859-1?Q?Stian_=D8vrev=E5ge?=) Date: Mon, 19 Oct 2009 12:46:17 +0200 Subject: [rsyslog] timegenerated and timereported Message-ID: timegenerated - timestamp when the message was RECEIVED. Always in high resolution timereported - timestamp from the message. Resolution depends on what was provided in the message (in most cases, only seconds) Neither of these fit into either a mysql datetime nor unixtime field type! They come in the format "Oct 19 12:32:54", exacly what is high resolution about that I don't know. So, does rsyslog provide a way to easily fit these into a time-aware mysql field type? My template looks like this: $template SyslogAllInsert,"INSERT INTO `Syslog`.`AllMessages` (`ID` ,`timeGenerated` ,`timeReported` ,`source` ,`programName` ,`syslogTag` ,`syslogFacility` ,`syslogSeverity` ,`message`)VALUES ( NULL , '%timegenerated%', '%timereported%', '%source%', '%programname%', '%syslogtag%', '%syslogfacility%', '%syslogseverity%', '%msg%');",sql TIA Cheers, Stian ?vrev?ge From dirk.schulz at kinzesberg.de Mon Oct 19 13:51:15 2009 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Mon, 19 Oct 2009 13:51:15 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG In-Reply-To: <87a1ae540910080951g6c7106fby53979808642cb537@mail.gmail.com> References: <4ACCA646.8050209@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> <4ACE0C3B.7090207@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA71031AC@GRFEXC.intern.adiscon.com> <4ACE11D9.1020806@kinzesberg.de> <87a1ae540910080951g6c7106fby53979808642cb537@mail.gmail.com> Message-ID: <4ADC52B3.4050806@kinzesberg.de> I fould the problem - I have to admit it was me. :-( Had a configuration error (cleverly combined with a research error). It is working fine now. Thanks a lot for you help. Dirk Sayan Chowdhury schrieb: > Hello Dirk, I think Rainer wants you to add this to your rsyslog.conf file, > and send him the file that gets saved because of adding this statement. So > as to debug the issue. > You will need to restart rsyslog to pick up the config change > Hope this helps. > Sayan > > On Thu, Oct 8, 2009 at 9:52 PM, Dirk H. Schulz wrote: > > >> Rainer Gerhards schrieb: >> >>>>> check doc for exact syntax): >>>>> >>>>> $template raw,"%rawmsg%\n" >>>>> *.* /path/to/file;raw >>>>> >>>>> >>>>> >>>> I did not find these terms, neither in rsyslog.conf on the central >>>> logserver servers nor in syslog-ng.conf on the log client servers. >>>> Rsyslog is 3.18.6-4 on Debian 5. >>>> >>>> >>> Well, these are the config statements for the central rsyslog instance... >>> >>> >> Does that mean that I am lacking a template statement in the central >> rsyslog config? >> >> Dirk >> >>> Rainer >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Oct 20 11:32:55 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 11:32:55 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><200910161555.40812.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA7103234@GRFEXC.intern.adiscon.com> <200910161632.20392.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103259@GRFEXC.intern.adiscon.com> Marc, I finally had some time to look into the issue. The problem actually is rooted in the shutdown sequence, which is rather complex in v3/v4. I am right now working on great simplification in v5, which will lead to more robust and even better performing code. Thanks to that work, I think I was able to quickly find the culprit and also to develop a patch. In my lab it works, and in theory it works as well, but practice is always another beast... So I would appreciate if you could give it a try. It is a pre-release of what once will become 4.4.3 and is available at: http://download.rsyslog.com/rsyslog/pre/rsyslog-4.4.3.tar.gz FYI, the actual patch is small: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=8e45fd5c8e7df45532c5d239 676cd92107c9e942 Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Friday, October 16, 2009 4:32 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Am Freitag, 16. Oktober 2009 15:59:18 schrieb Rainer Gerhards: > > just quickly > > > > > > > > Are these DA-queuing features are present in the 3.xx version > already? > > > Maybe I should consider using 3.xx the instead if its more robust. > > > > Indeed, they are. > > But 3.22.1 seems to have the same issues with DA-spooling :-( > I will do some more testing on monday. > > -Marc > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Oct 20 12:50:45 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 20 Oct 2009 12:50:45 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103259@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <200910161632.20392.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103259@GRFEXC.intern.adiscon.com> Message-ID: <200910201250.45820.marc.schiffbauer@mightycare.de> Am Dienstag, 20. Oktober 2009 11:32:55 schrieb Rainer Gerhards: > Marc, Hi Rainer, > > I finally had some time to look into the issue. The problem actually is > rooted in the shutdown sequence, which is rather complex in v3/v4. I am > right now working on great simplification in v5, which will lead to more > robust and even better performing code. > > Thanks to that work, I think I was able to quickly find the culprit and > also to develop a patch. In my lab it works, and in theory it works as > well, but practice is always another beast... > > So I would appreciate if you could give it a try. It is a pre-release of > what once will become 4.4.3 and is available at: > > http://download.rsyslog.com/rsyslog/pre/rsyslog-4.4.3.tar.gz Thanks very much! Now it works as expected with one exception: In my test that I did now I put an ID into each log-message to see if no messages get lost. That way I found that one message always gets lost when I do the following: * DB and rsyslogd started * Start logger loop with ID (slow message rate): X=1; while true; do logger -t spool-test "no real message here... (ID=$X)"; X=$((X+1)); sleep 3; done & * stop DB server, spooling to mem starts * start another logger loop to force DA spooling: while true; do logger -t spool-test "no real message here..."; done & * watch spool dir: as soon as a queue file is being created: stop both loops ("fg" followeg by ctrl-c, two times) * stop rsyslogd: qi is file being created now! now there should be no lost message, right? * start DB server * start rsyslogd * check the log: *one* of those ID=$X messages is lost. I did that test two times. -Marc From rgerhards at hq.adiscon.com Tue Oct 20 13:36:48 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 13:36:48 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? Message-ID: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> Can you mail me a debug log where the message loss can be seen? rainer ----- Urspr?ngliche Nachricht ----- Von: "Marc Schiffbauer" An: "rsyslog-users" Gesendet: 20.10.09 12:51 Betreff: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? Am Dienstag, 20. Oktober 2009 11:32:55 schrieb Rainer Gerhards: > Marc, Hi Rainer, > > I finally had some time to look into the issue. The problem actually is > rooted in the shutdown sequence, which is rather complex in v3/v4. I am > right now working on great simplification in v5, which will lead to more > robust and even better performing code. > > Thanks to that work, I think I was able to quickly find the culprit and > also to develop a patch. In my lab it works, and in theory it works as > well, but practice is always another beast... > > So I would appreciate if you could give it a try. It is a pre-release of > what once will become 4.4.3 and is available at: > > http://download.rsyslog.com/rsyslog/pre/rsyslog-4.4.3.tar.gz Thanks very much! Now it works as expected with one exception: In my test that I did now I put an ID into each log-message to see if no messages get lost. That way I found that one message always gets lost when I do the following: * DB and rsyslogd started * Start logger loop with ID (slow message rate): X=1; while true; do logger -t spool-test "no real message here... (ID=$X)"; X=$((X+1)); sleep 3; done & * stop DB server, spooling to mem starts * start another logger loop to force DA spooling: while true; do logger -t spool-test "no real message here..."; done & * watch spool dir: as soon as a queue file is being created: stop both loops ("fg" followeg by ctrl-c, two times) * stop rsyslogd: qi is file being created now! now there should be no lost message, right? * start DB server * start rsyslogd * check the log: *one* of those ID=$X messages is lost. I did that test two times. -Marc _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Oct 20 14:48:16 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 20 Oct 2009 14:48:16 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> Message-ID: <200910201448.16203.marc.schiffbauer@mightycare.de> Am Dienstag, 20. Oktober 2009 13:36:48 schrieb Rainer Gerhards: > Can you mail me a debug log where the message loss can be seen? Hi Rainer, I sent the logs to your gmail address. The lost message contains "ID=12". TIA -Marc From rgerhards at hq.adiscon.com Tue Oct 20 15:23:31 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 15:23:31 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200910201448.16203.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> interesting... the log looks like all works ok (of course, without the missing message ;)). Can you send me the contents of the .qi file after the shutdown? I am asking because it should contain the missing message (with that version of the queue engine). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, October 20, 2009 2:48 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Am Dienstag, 20. Oktober 2009 13:36:48 schrieb Rainer Gerhards: > > Can you mail me a debug log where the message loss can be seen? > > Hi Rainer, > > I sent the logs to your gmail address. > > The lost message contains "ID=12". > > > TIA > -Marc > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Oct 20 16:12:59 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 20 Oct 2009 16:12:59 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200910201448.16203.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> Message-ID: <200910201612.59316.marc.schiffbauer@mightycare.de> Am Dienstag, 20. Oktober 2009 15:23:31 schrieb Rainer Gerhards: > interesting... the log looks like all works ok (of course, without the > missing message ;)). Can you send me the contents of the .qi file after the > shutdown? I am asking because it should contain the missing message (with > that version of the queue engine). > > Rainer > Hi Rainer, I sent you another gmail. I repeated the test and this time I saved everything from the queue-Directory (one queue file + the dbq.qi file) before restarting rsyslogd The missing log entry in this try: ID=7 TIA -Marc From marc.schiffbauer at mightycare.de Tue Oct 20 16:39:42 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 20 Oct 2009 16:39:42 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200910201448.16203.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> Message-ID: <200910201639.43143.marc.schiffbauer@mightycare.de> Hi Rainer, new mail on gmail, now with attachment ;) TIA -Marc From anichols at trumped.org Tue Oct 20 17:32:07 2009 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 20 Oct 2009 09:32:07 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase in write volume (bytes/sec) to NAS - can someone help shed light? Message-ID: Hello, We have two rsyslog "writer" instances which are receiving syslog messages and writing them to an NFS mount on a NAS. The NAS is connected via 10gb ethernet but has a number of other applications writing to it as well. Typical write volume for our log writers is 2-3Mbps to the NAS. Yesterday I attempted to resolve some complaints with logs being "bursty" - events appear to be written in batches which sounded consistent with buffering on the rsyslog daemon or not syncing the file with every message. These bursts could come as infrequently as every 30 seconds however, which seems excessive. I took a number of steps to remedy this but ultimately had to reverse the changes because write volume (measured at the disks) to the NAS went from 2Mbps up to 9Mbps and iops went from 30 up to 900 at the peak. I'm sure there is an explanation for this - but there are some subtle configuration changes which had an impact and I don't understand exactly why. First, the main change was to set "$ActionFileEnableSync on" in each logging servers configuration. From what I can tell this globally enables the ability to turn on file sync and it defaults to "off". This change, combined with removing the "-" from all the actions in my configuration should have resulted in all files being synced all the time - probably the worst case scenario and I'm not all that suprised this caused problems. Second, before rolling the change back 100% - I removed the "$ActionFileEnableSync on" configuration from the servers but left the action configurations without the "-" - assuming that the global configuration option would completely disable file sync and using a "-" would have no effect. This did not seem to be the case - it did dramatically decrease the volume of traffic to the NAS but still the volume was about double (5Mbps) the normal volume. Third, I added "-" before all the action lines in the configuration and this brought volume back down to the 2-3Mbps we are used to. This leaves me with a few questions: 1) I would expect file sync to increase transactions with the NAS but not increase the volume of data being written to the NAS as measured at the disk. Can someone shed light on why this would so profoundly impact write volume? Load on the rsyslog servers also went down substantially after making this change - presumably because there were fewer queued transactions - not sure. 2) Is there some difference between what "$ActionFileEnableSync on" does vs. adding a "-" before an action other than one is global and one is per-action? I thought these were just two different levels of granularity for configuring the same thing but that doesn't appear to be the case. 3) Is it expected to have rsyslog take up to 30 seconds to flush messages to a logfile? These are BUSY logs - being written to multiple times per second, so it's not a delay on the client. I can observe these same clients logging messages to another logging server (the old server) running syslog-ng and messages are observable in near real-time. Ultimate - #3 is the problem I am trying to solve, but I'm very curious why these changes had the impact they did and why globally disabling file sync using "$ActionFileEnableSync" would not entire decrease log volume until I add "-" to all the actions. Let me know if I can add info that would help shed light on this. Thanks, Aaron From rgerhards at hq.adiscon.com Tue Oct 20 18:32:30 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 18:32:30 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase inwrite volume (bytes/sec) to NAS - can someone help shed light? References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> Hi Aaron, first and very important question: which version of rsyslog are you using? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Tuesday, October 20, 2009 5:32 PM > To: rsyslog-users > Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive > increase inwrite volume (bytes/sec) to NAS - can someone help shed > light? > > Hello, > We have two rsyslog "writer" instances which are receiving syslog > messages and writing them to an NFS mount on a NAS. The NAS is > connected via > 10gb ethernet but has a number of other applications writing to it as > well. > Typical write volume for our log writers is 2-3Mbps to the NAS. > Yesterday I > attempted to resolve some complaints with logs being "bursty" - events > appear to be written in batches which sounded consistent with buffering > on > the rsyslog daemon or not syncing the file with every message. These > bursts > could come as infrequently as every 30 seconds however, which seems > excessive. I took a number of steps to remedy this but ultimately had > to > reverse the changes because write volume (measured at the disks) to the > NAS > went from 2Mbps up to 9Mbps and iops went from 30 up to 900 at the > peak. I'm > sure there is an explanation for this - but there are some subtle > configuration changes which had an impact and I don't understand > exactly > why. > > First, the main change was to set "$ActionFileEnableSync on" in each > logging > servers configuration. From what I can tell this globally enables the > ability to turn on file sync and it defaults to "off". This change, > combined > with removing the "-" from all the actions in my configuration should > have > resulted in all files being synced all the time - probably the worst > case > scenario and I'm not all that suprised this caused problems. > > Second, before rolling the change back 100% - I removed the > "$ActionFileEnableSync on" configuration from the servers but left the > action configurations without the "-" - assuming that the global > configuration option would completely disable file sync and using a "-" > would have no effect. This did not seem to be the case - it did > dramatically > decrease the volume of traffic to the NAS but still the volume was > about > double (5Mbps) the normal volume. > > Third, I added "-" before all the action lines in the configuration and > this > brought volume back down to the 2-3Mbps we are used to. > > This leaves me with a few questions: > > 1) I would expect file sync to increase transactions with the NAS but > not > increase the volume of data being written to the NAS as measured at the > disk. Can someone shed light on why this would so profoundly impact > write > volume? Load on the rsyslog servers also went down substantially after > making this change - presumably because there were fewer queued > transactions > - not sure. > > 2) Is there some difference between what "$ActionFileEnableSync on" > does vs. > adding a "-" before an action other than one is global and one is > per-action? I thought these were just two different levels of > granularity > for configuring the same thing but that doesn't appear to be the case. > > 3) Is it expected to have rsyslog take up to 30 seconds to flush > messages to > a logfile? These are BUSY logs - being written to multiple times per > second, > so it's not a delay on the client. I can observe these same clients > logging > messages to another logging server (the old server) running syslog-ng > and > messages are observable in near real-time. > > Ultimate - #3 is the problem I am trying to solve, but I'm very curious > why > these changes had the impact they did and why globally disabling file > sync > using "$ActionFileEnableSync" would not entire decrease log volume > until I > add "-" to all the actions. > > Let me know if I can add info that would help shed light on this. > > Thanks, > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Oct 20 18:38:16 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 18:38:16 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com><200910201448.16203.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> <200910201639.43143.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> thanks! Interesting, I see that only one of the messages that should be in the .qi are actually present. I wonder if this is related to the fact that ompgsql emits an error message itself while being down (the others don't do this AFIK). Will try to dig down to this (but I have to do so as time permits). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, October 20, 2009 4:40 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > Hi Rainer, > > new mail on gmail, now with attachment ;) > > TIA > -Marc > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From anichols at trumped.org Tue Oct 20 18:54:02 2009 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 20 Oct 2009 10:54:02 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase inwrite volume (bytes/sec) to NAS - can someone help shed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> Message-ID: On Tue, Oct 20, 2009 at 10:32 AM, Rainer Gerhards wrote: > Hi Aaron, > > first and very important question: which version of rsyslog are you using? > > Of course - I forgot to include that, sorry - 4.4.1. Running on RHEL 5.3. From rgerhards at hq.adiscon.com Tue Oct 20 18:58:19 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 18:58:19 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincrease inwrite volume (bytes/sec) to NAS - can someone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Tuesday, October 20, 2009 6:54 PM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling $ActionFileEnableSync caused > massiveincrease inwrite volume (bytes/sec) to NAS - can someone > helpshed light? > > On Tue, Oct 20, 2009 at 10:32 AM, Rainer Gerhards > wrote: > > > Hi Aaron, > > > > first and very important question: which version of rsyslog are you > using? > > > > > Of course - I forgot to include that, sorry - 4.4.1. Running on RHEL > 5.3. Hmmm... OK, so this is a version without output batching support (v5+) and without background writer (current v4-beta). So I do not really see how rsyslog could request this writing (other, of course, than by using the sync calls). Could you create a debug log? I'd try to dig through it. But I'd actually think that it has something to do with the way NFS parameters are set. Rainer From anichols at trumped.org Tue Oct 20 20:41:44 2009 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 20 Oct 2009 12:41:44 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincrease inwrite volume (bytes/sec) to NAS - can someone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> Message-ID: On Tue, Oct 20, 2009 at 10:58 AM, Rainer Gerhards wrote: > Hmmm... OK, so this is a version without output batching support (v5+) and > without background writer (current v4-beta). So I do not really see how > rsyslog could request this writing (other, of course, than by using the > sync > calls). Could you create a debug log? I'd try to dig through it. But I'd > actually think that it has something to do with the way NFS parameters are > set. > Here are the NFS mount options for the nodes writing to the NFS volume. The NFS server is an EMC Celerra - I can get any specific parameters there are questions about but can't easily provide a dump of the configured options: hostname:/log/syslog /log/syslog nfs rw,vers=3,rsize=32768,wsize=32768,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=hostname 0 0 I will work on a debug log but as these are production systems it'll take a little time. Thanks, Aaron From anichols at trumped.org Tue Oct 20 22:37:43 2009 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 20 Oct 2009 14:37:43 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincrease inwrite volume (bytes/sec) to NAS - can someone helpshed light? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> Message-ID: Also - can someone just confirm that my assumption is correct about the expected behavior? Should the "$ActionFileEnableSync [on/off]" option enable the ABILITY to turn on file sync per action using "-" or does it do something different? If I have "$ActionFileEnableSync off" set - should I expect adding or removing "-" before an action to have any impact at all? The documentation just associates these two options with enabling/disabling "file sync" but it seems like there are some additional subtleties that aren't clear. Thanks, Aaron On Tue, Oct 20, 2009 at 12:41 PM, Aaron Nichols wrote: > > > On Tue, Oct 20, 2009 at 10:58 AM, Rainer Gerhards < > rgerhards at hq.adiscon.com> wrote: > >> Hmmm... OK, so this is a version without output batching support (v5+) >> and >> without background writer (current v4-beta). So I do not really see how >> rsyslog could request this writing (other, of course, than by using the >> sync >> calls). Could you create a debug log? I'd try to dig through it. But I'd >> actually think that it has something to do with the way NFS parameters are >> set. >> > > From david at lang.hm Wed Oct 21 07:12:39 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 20 Oct 2009 22:12:39 -0700 (PDT) Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase in write volume (bytes/sec) to NAS - can someone help shed light? In-Reply-To: References: Message-ID: hmm, how large are the log entries? I'm wondering if the fact that every log entry then updates the metadata (size, time modified, etc) could account for most, if not all, of your extra traffic. the other thing that you could be seeing is if the NFS server records write traffic based on # of blocks written * block size. when doing a sync after each write you will do a lot of blocks (but re-write the same block many times), if it counts each of these writes as full block that would greatly magnify the write totals. I think you will really like the message batching in v5. you configure it to output every X messages or Y ms, whichever comes first. doing a sync after every write will have _far_ less effect. David Lang On Tue, 20 Oct 2009, Aaron Nichols wrote: > Date: Tue, 20 Oct 2009 09:32:07 -0600 > From: Aaron Nichols > Reply-To: rsyslog-users > To: rsyslog-users > Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase in > write volume (bytes/sec) to NAS - can someone help shed light? > > Hello, > We have two rsyslog "writer" instances which are receiving syslog > messages and writing them to an NFS mount on a NAS. The NAS is connected via > 10gb ethernet but has a number of other applications writing to it as well. > Typical write volume for our log writers is 2-3Mbps to the NAS. Yesterday I > attempted to resolve some complaints with logs being "bursty" - events > appear to be written in batches which sounded consistent with buffering on > the rsyslog daemon or not syncing the file with every message. These bursts > could come as infrequently as every 30 seconds however, which seems > excessive. I took a number of steps to remedy this but ultimately had to > reverse the changes because write volume (measured at the disks) to the NAS > went from 2Mbps up to 9Mbps and iops went from 30 up to 900 at the peak. I'm > sure there is an explanation for this - but there are some subtle > configuration changes which had an impact and I don't understand exactly > why. > > First, the main change was to set "$ActionFileEnableSync on" in each logging > servers configuration. From what I can tell this globally enables the > ability to turn on file sync and it defaults to "off". This change, combined > with removing the "-" from all the actions in my configuration should have > resulted in all files being synced all the time - probably the worst case > scenario and I'm not all that suprised this caused problems. > > Second, before rolling the change back 100% - I removed the > "$ActionFileEnableSync on" configuration from the servers but left the > action configurations without the "-" - assuming that the global > configuration option would completely disable file sync and using a "-" > would have no effect. This did not seem to be the case - it did dramatically > decrease the volume of traffic to the NAS but still the volume was about > double (5Mbps) the normal volume. > > Third, I added "-" before all the action lines in the configuration and this > brought volume back down to the 2-3Mbps we are used to. > > This leaves me with a few questions: > > 1) I would expect file sync to increase transactions with the NAS but not > increase the volume of data being written to the NAS as measured at the > disk. Can someone shed light on why this would so profoundly impact write > volume? Load on the rsyslog servers also went down substantially after > making this change - presumably because there were fewer queued transactions > - not sure. > > 2) Is there some difference between what "$ActionFileEnableSync on" does vs. > adding a "-" before an action other than one is global and one is > per-action? I thought these were just two different levels of granularity > for configuring the same thing but that doesn't appear to be the case. > > 3) Is it expected to have rsyslog take up to 30 seconds to flush messages to > a logfile? These are BUSY logs - being written to multiple times per second, > so it's not a delay on the client. I can observe these same clients logging > messages to another logging server (the old server) running syslog-ng and > messages are observable in near real-time. > > Ultimate - #3 is the problem I am trying to solve, but I'm very curious why > these changes had the impact they did and why globally disabling file sync > using "$ActionFileEnableSync" would not entire decrease log volume until I > add "-" to all the actions. > > Let me know if I can add info that would help shed light on this. > > Thanks, > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From anichols at trumped.org Wed Oct 21 07:28:48 2009 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 20 Oct 2009 23:28:48 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase in write volume (bytes/sec) to NAS - can someone help shed light? In-Reply-To: References: Message-ID: On Tue, Oct 20, 2009 at 11:12 PM, wrote: > hmm, how large are the log entries? I'm wondering if the fact that every > log entry then updates the metadata (size, time modified, etc) could > account for most, if not all, of your extra traffic. > Most log entries are small (under 2k) but there is one log destination which produces very large log entries. We have had to increase the max message size to 64k to accommodate these messages. This is only one destination but comes from multiple sources (collated into a single file) - so with a better understanding of how the "$ActionFileEnableSync" option vs. using "-" on a particular action works, I may be able to enable sync for log destinations other than that one high volume, large message size destination. > the other thing that you could be seeing is if the NFS server records > write traffic based on # of blocks written * block size. > > when doing a sync after each write you will do a lot of blocks (but > re-write the same block many times), if it counts each of these writes as > full block that would greatly magnify the write totals. > > I think you will really like the message batching in v5. you configure it > to output every X messages or Y ms, whichever comes first. doing a sync > after every write will have _far_ less effect. > > I probably will enjoy v5 but am not yet ready to run it in production. I need to find a good balance between log update frequency and keeping NAS performance sane. I did pass this feedback along to our storage guys to see if there is any tuning we can do around this. I will also be gathering a debug log soon - awaiting approvals to run the test. I assume the debug output would show some indication of where this might be causing problems? Thanks again for the feedback. I'm pushing for a support contract as I get the feeling it'll come in handy as well and I want to support the project, but I appreciate the feedback in this forum, I know everyone is hard at work. Thanks, Aaron From rgerhards at hq.adiscon.com Wed Oct 21 08:59:48 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 21 Oct 2009 08:59:48 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase in write volume (bytes/sec) to NAS - can someone help shed light? References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103267@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, October 21, 2009 7:13 AM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling $ActionFileEnableSync caused massive > increase in write volume (bytes/sec) to NAS - can someone help shed > light? > > hmm, how large are the log entries? I'm wondering if the fact that > every > log entry then updates the metadata (size, time modified, etc) could > account for most, if not all, of your extra traffic. rsyslog does an fdatasync() to avoid this problem. However, that API is not available on all platforms... Rainer From rgerhards at hq.adiscon.com Wed Oct 21 09:33:22 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 21 Oct 2009 09:33:22 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincrease in write volume (bytes/sec) to NAS - can someonehelp shed light? References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710326A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Wednesday, October 21, 2009 7:29 AM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling $ActionFileEnableSync caused > massiveincrease in write volume (bytes/sec) to NAS - can someonehelp > shed light? > > On Tue, Oct 20, 2009 at 11:12 PM, wrote: > > > hmm, how large are the log entries? I'm wondering if the fact that > every > > log entry then updates the metadata (size, time modified, etc) could > > account for most, if not all, of your extra traffic. > > > > Most log entries are small (under 2k) but there is one log destination > which > produces very large log entries. We have had to increase the max > message > size to 64k to accommodate these messages. This is only one destination > but > comes from multiple sources (collated into a single file) - so with a > better > understanding of how the "$ActionFileEnableSync" option vs. using "-" > on a > particular action works, I may be able to enable sync for log > destinations > other than that one high volume, large message size destination. Sorry, I forgot to answer that in my posting yesterday. $ActionFileEnableSync is position dependent. So I assume that it is giving right at the top of rsyslog.conf, before any action. If set to on, it *enables* the sync facility. So you can use the dash in front of the file to specify whether or not a sync should happen. If it is set to "off" (the default), the "dash-rule" is always ignored and a sync never happens. This was done because many default configs have the sync option set just out of habit and the performance toll, as you have seen, is immense. > > > > the other thing that you could be seeing is if the NFS server records > > write traffic based on # of blocks written * block size. > > > > when doing a sync after each write you will do a lot of blocks (but > > re-write the same block many times), if it counts each of these > writes as > > full block that would greatly magnify the write totals. > > > > I think you will really like the message batching in v5. you > configure it > > to output every X messages or Y ms, whichever comes first. doing a > sync > > after every write will have _far_ less effect. > > > > > I probably will enjoy v5 but am not yet ready to run it in production. > I > need to find a good balance between log update frequency and keeping > NAS > performance sane. I did pass this feedback along to our storage guys to > see > if there is any tuning we can do around this. > > I will also be gathering a debug log soon - awaiting approvals to run > the > test. I assume the debug output would show some indication of where > this > might be causing problems? My primary interest in the debug log is to see if something works other than I would expect it. If that is not the case, you need to dig down into what NFS does to cause this behavior. Doing an strace may also be a good idea, depending on the output of the debug log. The core problem from my POV is that in 4.4.1 there is no code that could create the behavior you describe (at least not to the best of my knowledge, and this is what the debug log helps verify ;)). Rainer From rgerhards at hq.adiscon.com Wed Oct 21 09:38:19 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 21 Oct 2009 09:38:19 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincreaseinwrite volume (bytes/sec) to NAS - can someone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Tuesday, October 20, 2009 10:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling $ActionFileEnableSync caused > massiveincreaseinwrite volume (bytes/sec) to NAS - can someone helpshed > light? > > Also - can someone just confirm that my assumption is correct about the > expected behavior? Should the "$ActionFileEnableSync [on/off]" option > enable > the ABILITY to turn on file sync per action using "-" or does it do > something different? See my other note, but there may be a subtle misunderstanding: The presence of "-" turns OFF file syncing. Absence of "-" causes a sync! So in *.* /path/to/fileWithSync *.* -/path/to/fileWithoutSync fileWithSync is being synced and fileWithoutSync is not! This is traditional config file syntax and probably the main reason why *a lot* of real-world config files do syncs... Rainer From tbergfeld at hq.adiscon.com Wed Oct 21 11:57:33 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 21 Oct 2009 11:57:33 +0200 Subject: [rsyslog] rsyslog 4.5.5 (v4-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103272@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.5.5, a member of the v4-beta branch. This release offers a new config directive $InputTCPServerNotifyOnConnectionClose. Also there are also some bug fixes. See Changelog for more details. This is a recommended update for all users of the devel branch. ChangeLog: http://www.rsyslog.com/Article416.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-182.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com _______________________________________________ From tbergfeld at hq.adiscon.com Wed Oct 21 11:56:58 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 21 Oct 2009 11:56:58 +0200 Subject: [rsyslog] rsyslog 5.3.2 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103271@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.3.2, a member of the devel branch. This release offers a lot of enhancement like the enhanced omfile to support transactional interface. Also there are also some bug fixes. See Changelog for more details. This is a recommended update for all users of the devel branch. ChangeLog: http://www.rsyslog.com/Article415.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-181.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com _______________________________________________ From anichols at trumped.org Wed Oct 21 15:17:49 2009 From: anichols at trumped.org (Aaron Nichols) Date: Wed, 21 Oct 2009 07:17:49 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincreaseinwrite volume (bytes/sec) to NAS - can someone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> Message-ID: On Wed, Oct 21, 2009 at 1:38 AM, Rainer Gerhards wrote: > See my other note, but there may be a subtle misunderstanding: The presence > of "-" turns OFF file syncing. Absence of "-" causes a sync! So in > > *.* /path/to/fileWithSync > *.* -/path/to/fileWithoutSync > > fileWithSync is being synced and fileWithoutSync is not! > > This is traditional config file syntax and probably the main reason why *a > lot* of real-world config files do syncs... > > I understood correctly and now I'm certain I know how it should work. I need to do some more testing and I think I have enough information to do that and will come back here when I have something interesting to share. An observation I did make when this was happening was that if $ActionFileEnableSync was disabled (commented out) but the "-" was missing before the actions in the configuration, write volume to the NAS was higher than when we had $ActionFileEnableSync disabled and had the "-" added before the action (write volume was 100% higher if $ActionFileEnableSync was OFF and the "-" option was missing from all log actions than if $ActionFileEnableSync was OFF and "-" was included in all log actions.) This led to my original suspicion that the two options somehow change sync behavior independently. If they shouldn't, then I need to get the debug log and do some analysis. I appreciate the guidance - I'll gather some more details. Thanks, Aaron From rgerhards at hq.adiscon.com Wed Oct 21 15:22:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 21 Oct 2009 15:22:29 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync causedmassiveincreaseinwrite volume (bytes/sec) to NAS - cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Wednesday, October 21, 2009 3:18 PM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling $ActionFileEnableSync > causedmassiveincreaseinwrite volume (bytes/sec) to NAS - cansomeone > helpshed light? > > On Wed, Oct 21, 2009 at 1:38 AM, Rainer Gerhards > wrote: > > > See my other note, but there may be a subtle misunderstanding: The > presence > > of "-" turns OFF file syncing. Absence of "-" causes a sync! So in > > > > *.* /path/to/fileWithSync > > *.* -/path/to/fileWithoutSync > > > > fileWithSync is being synced and fileWithoutSync is not! > > > > This is traditional config file syntax and probably the main reason > why *a > > lot* of real-world config files do syncs... > > > > > I understood correctly and now I'm certain I know how it should work. I > need > to do some more testing and I think I have enough information to do > that and > will come back here when I have something interesting to share. An > observation I did make when this was happening was that if > $ActionFileEnableSync was disabled (commented out) but the "-" was > missing > before the actions in the configuration, write volume to the NAS was > higher > than when we had $ActionFileEnableSync disabled and had the "-" added > before > the action (write volume was 100% higher if $ActionFileEnableSync was > OFF > and the "-" option was missing from all log actions than if > $ActionFileEnableSync was OFF and "-" was included in all log actions.) That indeed sounds odd. Are you sure the $ActionFileEnableSync is in front of all write statements? On some systems, write statements may be included via files and if these are in front of the directive, those would be unaffected by it... Rainer From anichols at trumped.org Wed Oct 21 16:25:21 2009 From: anichols at trumped.org (Aaron Nichols) Date: Wed, 21 Oct 2009 08:25:21 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync causedmassiveincreaseinwrite volume (bytes/sec) to NAS - cansomeone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> Message-ID: On Wed, Oct 21, 2009 at 7:22 AM, Rainer Gerhards wrote: > That indeed sounds odd. Are you sure the $ActionFileEnableSync is in front > of > all write statements? On some systems, write statements may be included via > files and if these are in front of the directive, those would be unaffected > by it... > We do also use includes but in this case $ActionFileEnableSync was at the top of the included file with all write statements below it. I will also note that the write volume changed significantly with two different config changes. To uber-simplify my original email: "normal" volume - $ActionFileEnableSync off & all actions include "-" in front : 2-3mbps ~30 iops $ActionFileEnableSync off & all actions exclude "-" in front : 5mbps (~100% increase) (iops unknown) $ActionFileEnableSync on & all actions exclude "-" in front : 9mbps (~300-400% increase) ~900 iops I am using a mix of static and dynamic filename actions and some actions use format templates as well. I suppose there could be one of those combinations which is not honoring the global $ActionFileEnableSync configuration. Again, I need to do some testing because at this point I'm just making guesses without much information. Thanks, Aaron From rgerhards at hq.adiscon.com Wed Oct 21 17:32:59 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 21 Oct 2009 17:32:59 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Wednesday, October 21, 2009 4:25 PM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling > $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to > NAS -cansomeone helpshed light? > > On Wed, Oct 21, 2009 at 7:22 AM, Rainer Gerhards > wrote: > > > That indeed sounds odd. Are you sure the $ActionFileEnableSync is in > front > > of > > all write statements? On some systems, write statements may be > included via > > files and if these are in front of the directive, those would be > unaffected > > by it... > > > > We do also use includes but in this case $ActionFileEnableSync was at > the > top of the included file with all write statements below it. I will > also > note that the write volume changed significantly with two different > config > changes. To uber-simplify my original email: > > "normal" volume - $ActionFileEnableSync off & all actions include "-" > in > front : 2-3mbps ~30 iops > $ActionFileEnableSync off & all actions exclude "-" in front : 5mbps > (~100% > increase) (iops unknown) > $ActionFileEnableSync on & all actions exclude "-" in front : 9mbps > (~300-400% increase) ~900 iops > > I am using a mix of static and dynamic filename actions and some > actions use > format templates as well. I suppose there could be one of those > combinations > which is not honoring the global $ActionFileEnableSync configuration. I may be overlooking something here, but the check is very straightforward and I don't see why some should be affected but others not. This is the file (and spot) in question: http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/omfile.c;h=bb12b4b61dacb 84167b9b0d17e49a597342dddc0;hb=07b076ddcfed4ea9b447d0be574d1dcdb799bc2f#l623 Do a search for "bEnableSync" to see all occurences. But the debug log will show... Rainer > Again, > I need to do some testing because at this point I'm just making guesses > without much information. > > Thanks, > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From wirelessdreamer at gmail.com Wed Oct 21 18:09:36 2009 From: wirelessdreamer at gmail.com (Nathanael Anderson) Date: Wed, 21 Oct 2009 11:09:36 -0500 Subject: [rsyslog] Regex matching problems Message-ID: I've been trying to get a basic regex to work in the regex tester at: http://www.rsyslog.com/user-regex.php \.[0-9]{1,3} I'm using this regex to match one octet in this message: Killing attempted connection: tcp (192.168.129.13:48351 - 192.168.145.91:35500) This should return 6 matches that can be accessed by changing the position fied from 0 to the field I want. in both rsyslog and the testing link, this is not the case. %msg:R,ERE,0,DFLT:\.[0-9]{1,3}--end% Is this a bug, or am I misunderstanding how this should behave? Nathanael From david at lang.hm Wed Oct 21 20:04:27 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 21 Oct 2009 11:04:27 -0700 (PDT) Subject: [rsyslog] makefile tweak Message-ID: when doing 'make clean' it should not need to run configure first. David Lang From mbiebl at gmail.com Wed Oct 21 20:11:49 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 21 Oct 2009 20:11:49 +0200 Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: 2009/10/21 : > when doing 'make clean' it should not need to run configure first. Ist that from a release tarball or a git snapshot? Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From david at lang.hm Wed Oct 21 22:19:41 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 21 Oct 2009 13:19:41 -0700 (PDT) Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: On Wed, 21 Oct 2009, Michael Biebl wrote: > 2009/10/21 : >> when doing 'make clean' it should not need to run configure first. > > Ist that from a release tarball or a git snapshot? git snapshot, especially after doing the autoconf -fvi. the system spends several min doing lots of checks before it can remove the files, then I do a configure with the options that I need and it goes through and does all those checks again. David Lang From mbiebl at gmail.com Wed Oct 21 22:26:09 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 21 Oct 2009 22:26:09 +0200 Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: 2009/10/21 : > On Wed, 21 Oct 2009, Michael Biebl wrote: > >> 2009/10/21 ?: >>> when doing 'make clean' it should not need to run configure first. >> >> Ist that from a release tarball or a git snapshot? > > git snapshot, especially after doing the autoconf -fvi. the system spends That's the wrong command. You should use autoreconf -vfi in a fresh git checkout. After that ./configure && make clean works as expected for me. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From david at lang.hm Wed Oct 21 22:30:22 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 21 Oct 2009 13:30:22 -0700 (PDT) Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: On Wed, 21 Oct 2009, Michael Biebl wrote: > Date: Wed, 21 Oct 2009 22:26:09 +0200 > From: Michael Biebl > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] makefile tweak > > 2009/10/21 : >> On Wed, 21 Oct 2009, Michael Biebl wrote: >> >>> 2009/10/21 ?: >>>> when doing 'make clean' it should not need to run configure first. >>> >>> Ist that from a release tarball or a git snapshot? >> >> git snapshot, especially after doing the autoconf -fvi. the system spends > > That's the wrong command. > > You should use autoreconf -vfi in a fresh git checkout. sorry, I typed the wrong thing in the e-mail > After that ./configure && make clean works as expected for me. I do autoreconf -fvi make clean ./configure --enable-imfile make the make clean forces a configure David Lang From mbiebl at gmail.com Wed Oct 21 22:39:36 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 21 Oct 2009 22:39:36 +0200 Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: 2009/10/21 : > On Wed, 21 Oct 2009, Michael Biebl wrote: > >> Date: Wed, 21 Oct 2009 22:26:09 +0200 >> From: Michael Biebl >> Reply-To: rsyslog-users >> To: rsyslog-users >> Subject: Re: [rsyslog] makefile tweak >> >> 2009/10/21 ?: >>> >>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>> >>>> 2009/10/21 ?: >>>>> >>>>> when doing 'make clean' it should not need to run configure first. >>>> >>>> Ist that from a release tarball or a git snapshot? >>> >>> git snapshot, especially after doing the autoconf -fvi. the system spends >> >> That's the wrong command. >> >> You should use autoreconf -vfi in a fresh git checkout. > > sorry, I typed the wrong thing in the e-mail > >> After that ./configure && make clean works as expected for me. > > I do > autoreconf -fvi > make clean How do you expect "make" to work without running ./configure before? If you have a Makefile's from a previous ./configure run, it's absolutely correct behaviour that after a autoreconf -vfi, a configure run is first done before you can run make. To sum up, everything works as it should. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From david at lang.hm Wed Oct 21 22:52:01 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 21 Oct 2009 13:52:01 -0700 (PDT) Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: On Wed, 21 Oct 2009, Michael Biebl wrote: > 2009/10/21 : >> On Wed, 21 Oct 2009, Michael Biebl wrote: >> >>> 2009/10/21 ?: >>>> >>>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>>> >>>>> 2009/10/21 ?: >>>>>> >>>>>> when doing 'make clean' it should not need to run configure first. >>>>> >>>>> Ist that from a release tarball or a git snapshot? >>>> >>>> git snapshot, especially after doing the autoconf -fvi. the system spends >>> >>> That's the wrong command. >>> >>> You should use autoreconf -vfi in a fresh git checkout. >> >> sorry, I typed the wrong thing in the e-mail >> >>> After that ./configure && make clean works as expected for me. >> >> I do >> autoreconf -fvi >> make clean > > How do you expect "make" to work without running ./configure before? > If you have a Makefile's from a previous ./configure run, it's > absolutely correct behaviour that after a autoreconf -vfi, a configure > run is first done before you can run make. > > To sum up, everything works as it should. I would not expect to be able to do a normal make, but since a make clean just deletes files (and the same files, not matter what the config options are) I would expect it to work. David Lang From mbiebl at gmail.com Wed Oct 21 23:02:17 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 21 Oct 2009 23:02:17 +0200 Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: 2009/10/21 : > On Wed, 21 Oct 2009, Michael Biebl wrote: > >> 2009/10/21 ?: >>> >>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>> >>>> 2009/10/21 ?: >>>>> >>>>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>>>> >>>>>> 2009/10/21 ?: >>>>>>> >>>>>>> when doing 'make clean' it should not need to run configure first. >>>>>> >>>>>> Ist that from a release tarball or a git snapshot? >>>>> >>>>> git snapshot, especially after doing the autoconf -fvi. the system >>>>> spends >>>> >>>> That's the wrong command. >>>> >>>> You should use autoreconf -vfi in a fresh git checkout. >>> >>> sorry, I typed the wrong thing in the e-mail >>> >>>> After that ./configure && make clean works as expected for me. >>> >>> I do >>> autoreconf -fvi >>> make clean >> >> How do you expect "make" to work without running ./configure before? >> If you have a Makefile's from a previous ./configure run, it's >> absolutely correct behaviour that after a autoreconf -vfi, a configure >> run is first done before you can run make. >> >> To sum up, everything works as it should. > > I would not expect to be able to do a normal make, but since a make clean > just deletes files (and the same files, not matter what the config options > are) I would expect it to work. Well, that's how automake/autoconf works. If the timestamps of build input files change, the build system is automatically re-generated. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Wed Oct 21 23:09:35 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 21 Oct 2009 23:09:35 +0200 Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: 2009/10/21 Michael Biebl : > 2009/10/21 ?: >> On Wed, 21 Oct 2009, Michael Biebl wrote: >> >>> 2009/10/21 ?: >>>> >>>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>>> >>>>> 2009/10/21 ?: >>>>>> >>>>>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>>>>> >>>>>>> 2009/10/21 ?: >>>>>>>> >>>>>>>> when doing 'make clean' it should not need to run configure first. >>>>>>> >>>>>>> Ist that from a release tarball or a git snapshot? >>>>>> >>>>>> git snapshot, especially after doing the autoconf -fvi. the system >>>>>> spends >>>>> >>>>> That's the wrong command. >>>>> >>>>> You should use autoreconf -vfi in a fresh git checkout. >>>> >>>> sorry, I typed the wrong thing in the e-mail >>>> >>>>> After that ./configure && make clean works as expected for me. >>>> >>>> I do >>>> autoreconf -fvi >>>> make clean >>> >>> How do you expect "make" to work without running ./configure before? >>> If you have a Makefile's from a previous ./configure run, it's >>> absolutely correct behaviour that after a autoreconf -vfi, a configure >>> run is first done before you can run make. >>> >>> To sum up, everything works as it should. >> >> I would not expect to be able to do a normal make, but since a make clean >> just deletes files (and the same files, not matter what the config options >> are) I would expect it to work. > > Well, that's how automake/autoconf works. > If the timestamps of build input files change, the build system is > automatically re-generated. Btw, since you run autoreconf anyway there is a much nicer (and quicker way to get rid of the old build files): git clean -x -d -f autoreconf -vfi Will give you a clean working state ready to run ./configure && make -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Oct 22 06:40:52 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 22 Oct 2009 06:40:52 +0200 Subject: [rsyslog] makefile tweak Message-ID: <000101ca52d1$dea24313$100013ac@intern.adiscon.com> The point is that configure decides which files are to be created at all. So without a configure run, make does not even know what to delete. rainer ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 21.10.09 22:52 Betreff: Re: [rsyslog] makefile tweak On Wed, 21 Oct 2009, Michael Biebl wrote: > 2009/10/21 : >> On Wed, 21 Oct 2009, Michael Biebl wrote: >> >>> 2009/10/21 : >>>> >>>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>>> >>>>> 2009/10/21 : >>>>>> >>>>>> when doing 'make clean' it should not need to run configure first. >>>>> >>>>> Ist that from a release tarball or a git snapshot? >>>> >>>> git snapshot, especially after doing the autoconf -fvi. the system spends >>> >>> That's the wrong command. >>> >>> You should use autoreconf -vfi in a fresh git checkout. >> >> sorry, I typed the wrong thing in the e-mail >> >>> After that ./configure && make clean works as expected for me. >> >> I do >> autoreconf -fvi >> make clean > > How do you expect "make" to work without running ./configure before? > If you have a Makefile's from a previous ./configure run, it's > absolutely correct behaviour that after a autoreconf -vfi, a configure > run is first done before you can run make. > > To sum up, everything works as it should. I would not expect to be able to do a normal make, but since a make clean just deletes files (and the same files, not matter what the config options are) I would expect it to work. David Lang From marc.schiffbauer at mightycare.de Thu Oct 22 15:06:29 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Thu, 22 Oct 2009 15:06:29 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200910201639.43143.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> Message-ID: <200910221506.29850.marc.schiffbauer@mightycare.de> Am Dienstag, 20. Oktober 2009 18:38:16 schrieb Rainer Gerhards: > thanks! > > Interesting, I see that only one of the messages that should be in the .qi > are actually present. I wonder if this is related to the fact that ompgsql > emits an error message itself while being down (the others don't do this > AFIK). Will try to dig down to this (but I have to do so as time permits). > Thanks, please tell me if I can assist in any way. -Marc From david at lang.hm Thu Oct 22 21:29:48 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 22 Oct 2009 12:29:48 -0700 (PDT) Subject: [rsyslog] version numbers from git repository Message-ID: I've been doing a lot of testing and can have a dozen different copies of rsyslog that all claim to be the same version number. the ability of git to give more precise versions (as described in the message from the git mailing list below) would be handy in eliminating confusion. the nice thing is that if a person is using a tagged version, this just reports the tag with nothing extra, so changing the makefile to use this would not affect most people. David Lang ---------- Forwarded message ---------- Date: Thu, 22 Oct 2009 10:01:25 +0200 From: Martin Langhoff To: Robin Rosenberg Cc: newsgroups at catchall.shelter13.net, git at vger.kernel.org Subject: Re: Deciding between Git/Mercurial On Sun, Sep 27, 2009 at 8:01 PM, Robin Rosenberg wrote: > s?ndag 27 september 2009 14:24:32 skrev Anteru : >> Mercurial's revision number system: With git, I get an SHA1 hash for >> every commit, but it's not possible to see whether Hash1 is newer than >> Hash2, while Mecurial also adds a running number to each commit. What's > > But those numbers cannot be communicated since they are local to your > clone. You can use git-describe, which will look for the latest tag, and make a combo of latest tag, commits since the tag, short form of sha1. So you get "v1.6.3-33-g1234" - 33 commits after 1.6.3. Works very well to integrate in versioning. The git project itself uses it in the Makefile to set the versions, same as the kernel folk do -- I use it to version even RPM/DEBs. hth, m -- martin.langhoff at gmail.com martin at laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majordomo at vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html From rgerhards at hq.adiscon.com Thu Oct 22 21:32:58 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 22 Oct 2009 21:32:58 +0200 Subject: [rsyslog] version numbers from git repository References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> David, I like this idea, but I have to admit I have not even the slightest clue on how I should implement it... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, October 22, 2009 9:30 PM > To: rsyslog-users > Subject: [rsyslog] version numbers from git repository > > I've been doing a lot of testing and can have a dozen > different copies of > rsyslog that all claim to be the same version number. > > the ability of git to give more precise versions (as described in the > message from the git mailing list below) would be handy in > eliminating > confusion. > > the nice thing is that if a person is using a tagged version, > this just > reports the tag with nothing extra, so changing the makefile > to use this > would not affect most people. > > David Lang > > ---------- Forwarded message ---------- > Date: Thu, 22 Oct 2009 10:01:25 +0200 > From: Martin Langhoff > To: Robin Rosenberg > Cc: newsgroups at catchall.shelter13.net, git at vger.kernel.org > Subject: Re: Deciding between Git/Mercurial > > On Sun, Sep 27, 2009 at 8:01 PM, Robin Rosenberg > wrote: > > s?ndag 27 september 2009 14:24:32 skrev Anteru > : > >> Mercurial's revision number system: With git, I get an > SHA1 hash for > >> every commit, but it's not possible to see whether Hash1 > is newer than > >> Hash2, while Mecurial also adds a running number to each > commit. What's > > > > But those numbers cannot be communicated since they are > local to your > > clone. > > You can use git-describe, which will look for the latest tag, and make > a combo of latest tag, commits since the tag, short form of sha1. So > you get "v1.6.3-33-g1234" - 33 commits after 1.6.3. Works very well to > integrate in versioning. The git project itself uses it in the > Makefile to set the versions, same as the kernel folk do -- I use it > to version even RPM/DEBs. > > hth, > > > > m > -- > martin.langhoff at gmail.com > martin at laptop.org -- School Server Architect > - ask interesting questions > - don't get distracted with shiny stuff - working code first > - http://wiki.laptop.org/go/User:Martinlanghoff > -- > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo at vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu Oct 22 21:36:49 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 22 Oct 2009 12:36:49 -0700 (PDT) Subject: [rsyslog] version numbers from git repository In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 22 Oct 2009, Rainer Gerhards wrote: > David, > > I like this idea, but I have to admit I have not even the slightest clue on > how I should implement it... I don't know either, but I do know a couple projects that do implement it. git implements it the linux kernel implements it if LOCAL_VERSION_AUTO is enabled in the config if I get a chance I'll try to dig into their makefiles to find out how. David Lang P.S. I don't know what rsyslog does with the version number, but when I use checkinstall to create a package from the source it ends up detecting the version number many times instead of once so instead of version='5.3.2' I get version='5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2' > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Thursday, October 22, 2009 9:30 PM >> To: rsyslog-users >> Subject: [rsyslog] version numbers from git repository >> >> I've been doing a lot of testing and can have a dozen >> different copies of >> rsyslog that all claim to be the same version number. >> >> the ability of git to give more precise versions (as described in the >> message from the git mailing list below) would be handy in >> eliminating >> confusion. >> >> the nice thing is that if a person is using a tagged version, >> this just >> reports the tag with nothing extra, so changing the makefile >> to use this >> would not affect most people. >> >> David Lang >> >> ---------- Forwarded message ---------- >> Date: Thu, 22 Oct 2009 10:01:25 +0200 >> From: Martin Langhoff >> To: Robin Rosenberg >> Cc: newsgroups at catchall.shelter13.net, git at vger.kernel.org >> Subject: Re: Deciding between Git/Mercurial >> >> On Sun, Sep 27, 2009 at 8:01 PM, Robin Rosenberg >> wrote: >>> s?ndag 27 september 2009 14:24:32 skrev Anteru >> : >>>> Mercurial's revision number system: With git, I get an >> SHA1 hash for >>>> every commit, but it's not possible to see whether Hash1 >> is newer than >>>> Hash2, while Mecurial also adds a running number to each >> commit. What's >>> >>> But those numbers cannot be communicated since they are >> local to your >>> clone. >> >> You can use git-describe, which will look for the latest tag, and make >> a combo of latest tag, commits since the tag, short form of sha1. So >> you get "v1.6.3-33-g1234" - 33 commits after 1.6.3. Works very well to >> integrate in versioning. The git project itself uses it in the >> Makefile to set the versions, same as the kernel folk do -- I use it >> to version even RPM/DEBs. >> >> hth, >> >> >> >> m >> -- >> martin.langhoff at gmail.com >> martin at laptop.org -- School Server Architect >> - ask interesting questions >> - don't get distracted with shiny stuff - working code first >> - http://wiki.laptop.org/go/User:Martinlanghoff >> -- >> To unsubscribe from this list: send the line "unsubscribe git" in >> the body of a message to majordomo at vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Oct 23 07:54:28 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 07:54:28 +0200 Subject: [rsyslog] version numbers from git repository References: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103292@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, October 22, 2009 9:37 PM > To: rsyslog-users > Subject: Re: [rsyslog] version numbers from git repository > > On Thu, 22 Oct 2009, Rainer Gerhards wrote: > > > David, > > > > I like this idea, but I have to admit I have not even the slightest > clue on > > how I should implement it... > > I don't know either, but I do know a couple projects that do implement > it. > > git implements it > > the linux kernel implements it if LOCAL_VERSION_AUTO is enabled in the > config > > if I get a chance I'll try to dig into their makefiles to find out how. I've looked a bit into the issue. Neither of them use the GNU build system. In essence, it boils down to running "git describe" as part of the build process, generating a file with it. However, I do not know how I may achieve that. It should probably be run during the ./configure stage. Maybe someone can provide some advise ;) > > David Lang > > P.S. I don't know what rsyslog does with the version number, but when I > use checkinstall to create a package from the source it ends up > detecting > the version number many times instead of once > > so instead of > version='5.3.2' > I get > version='5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2' I have no real idea what may cause that other than the fact that the version number is probably present more than once in the constant pool because of rsyslog's modular strucutre. Rainer From rgerhards at hq.adiscon.com Fri Oct 23 08:21:01 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 08:21:01 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> Aaron, I am replying on-list so that others can follow (and I hope they have some advise). I have stripped the sensitive part of your message. In short for all others: I got a debug log and have now reviewed it. > During this debug run we saw NAS write volume increase from a "normal" > 2-3mbps and 35 iops up to 500 iops and 5.6mbps toward the end of the > test. I do understand how enabling file sync will increase iops but > I'm unclear on why the volume of data being written would increase so > much. If rsyslog has file sync "disabled" then I would expect bursts > of data and if file sync is "enabled" I would expect the same amount > of data to be written in much more frequent, but also much smaller > chunks. NFS could be a culprit here, but I'm curious what NFS > parameters would impact this behavior. The debug log does not show anything that looks specifically suspect. The incoming traffic is intense and the default queue size is overrun quickly, resulting in voluntary delays of the sender (via tcp). Looking at the traffic, I think this is perfectly legal and increasing the queue size would not bring any benefit (I think the end result would just be a larger memory footprint without any advantage). I do not see any unusual write behavior. Of course, the debug log could be more verbose as it does not specifically tell about the syncs (usually, the log is missing some information you'd like to see for a given instant, but you can't include everything...). But looking at the code base plus the log really does not reveal anything suspicious. One thing you might want to try is doing an strace so that we can look at it and see which exact OS calls are made and how long they last. > For the record, changing the sync behavior of rsyslog DOES fix the > original complaint. When we have file sync disabled we see up to 30 > second pauses between writes to files which get multiple messages per > second delivered to the rsyslog host. There is nothing in the log that points to rsyslog doing these delays. So my conclusion really is that you need to look at the NFS layer. To me, it looks like NFS does some local caching, and maybe 30 seconds is the timeout at which NFS writes data from files that are constantly being written in short intervals. With sync, NFS probably writes and rewrites each block as David has said, thus resulting in the far larger i/o footprint. But I am far from being a NFS expert, so it would probably make sense to ask one ;) > I have looked at the option > $OMFileFlushOnTXEnd and wonder if it would be a preferred method to > resolve this in the 4.1.1 version (or if it even works there). This makes no sense for v4 (I think it is not even available). Sorry I have no better answer. If get a strace, I'd gladly look at it. Rainer From david at lang.hm Fri Oct 23 08:32:05 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 22 Oct 2009 23:32:05 -0700 (PDT) Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 23 Oct 2009, Rainer Gerhards wrote: >> For the record, changing the sync behavior of rsyslog DOES fix the >> original complaint. When we have file sync disabled we see up to 30 >> second pauses between writes to files which get multiple messages per >> second delivered to the rsyslog host. > > There is nothing in the log that points to rsyslog doing these delays. > > So my conclusion really is that you need to look at the NFS layer. To me, it > looks like NFS does some local caching, and maybe 30 seconds is the timeout > at which NFS writes data from files that are constantly being written in > short intervals. With sync, NFS probably writes and rewrites each block as > David has said, thus resulting in the far larger i/o footprint. But I am far > from being a NFS expert, so it would probably make sense to ask one ;) this reminded me of something. a standard complient NFS implementation would require that any write to a NFS volume go all the way to the destination and be safe on disk before the write call returns to the calling program. this doesn't mean when you do a fsync, it means each individual write call. for the program to think the data has been written and it not be available on the NFS drive to other systems there has to be some non-standard caching in place here. note that unless you have a high-end NFS server that includes some battery-backed ram on it, standard complient behavior also means a very low write transaction rate. note that the caching could be on the reader machines instead of on the writer, and it could be something like caching of the metadata (including size and last modified timestamp) David Lang From aland at freeradius.org Fri Oct 23 08:39:35 2009 From: aland at freeradius.org (Alan T DeKok) Date: Fri, 23 Oct 2009 08:39:35 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> Message-ID: <4AE14FA7.4010008@freeradius.org> Rainer Gerhards wrote: > So my conclusion really is that you need to look at the NFS layer. Logs should NEVER be written to an NFS mount. David gave a good explanation as to why. But the above prohibition should be made clear in the docs, if it isn't already. If the NFS server goes away, then the write transaction rate will go to zero. This likely isn't what people want from a syslog server. There is already a way to get syslog data from one system to another: the syslog protocol. Using NFS as a replacement for syslog is wrong on many levels. Alan DeKok. From rgerhards at hq.adiscon.com Fri Oct 23 09:03:05 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 09:03:05 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> <4AE14FA7.4010008@freeradius.org> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> I think the main point of the OP was that he logs to a NAS device, so I do not see way to write to it without going through a network share. Other than that, I agree to your statement. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Alan T DeKok > Sent: Friday, October 23, 2009 8:40 AM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling > $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to > NAS -cansomeone helpshed light? > > Rainer Gerhards wrote: > > So my conclusion really is that you need to look at the NFS layer. > > Logs should NEVER be written to an NFS mount. > > David gave a good explanation as to why. But the above prohibition > should be made clear in the docs, if it isn't already. > > If the NFS server goes away, then the write transaction rate will go > to zero. This likely isn't what people want from a syslog server. > > There is already a way to get syslog data from one system to another: > the syslog protocol. Using NFS as a replacement for syslog is wrong on > many levels. > > Alan DeKok. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Oct 23 09:17:43 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 23 Oct 2009 00:17:43 -0700 (PDT) Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> <4AE14FA7.4010008@freeradius.org> <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 23 Oct 2009, Rainer Gerhards wrote: > I think the main point of the OP was that he logs to a NAS device, so I do > not see way to write to it without going through a network share. ahh, have we been assuming that this is NFS when all he said was NAS? if so, please clarify what protocol you are using to talk to the NAS. it could make a huge difference here. David Lang > Other than that, I agree to your statement. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Alan T DeKok >> Sent: Friday, October 23, 2009 8:40 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Enabling >> $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to >> NAS -cansomeone helpshed light? >> >> Rainer Gerhards wrote: >>> So my conclusion really is that you need to look at the NFS layer. >> >> Logs should NEVER be written to an NFS mount. >> >> David gave a good explanation as to why. But the above prohibition >> should be made clear in the docs, if it isn't already. >> >> If the NFS server goes away, then the write transaction rate will go >> to zero. This likely isn't what people want from a syslog server. >> >> There is already a way to get syslog data from one system to another: >> the syslog protocol. Using NFS as a replacement for syslog is wrong on >> many levels. >> >> Alan DeKok. >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From aland at freeradius.org Fri Oct 23 09:42:22 2009 From: aland at freeradius.org (Alan T DeKok) Date: Fri, 23 Oct 2009 09:42:22 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> <4AE14FA7.4010008@freeradius.org> <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> Message-ID: <4AE15E5E.2020100@freeradius.org> Rainer Gerhards wrote: > I think the main point of the OP was that he logs to a NAS device, so I do > not see way to write to it without going through a network share. Configure rsyslog to write to local disk, and rsync periodically to the network share. Or, use another instance of rsyslog to copy from local disk to the network share. Any other method will have problems. Even NetApp systems have issues from time to time. Alan DeKok. From rgerhards at hq.adiscon.com Fri Oct 23 09:53:32 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 09:53:32 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com><4AE14FA7.4010008@freeradius.org><9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103298@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 23, 2009 9:18 AM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling > $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to > NAS -cansomeone helpshed light? > > On Fri, 23 Oct 2009, Rainer Gerhards wrote: > > > I think the main point of the OP was that he logs to a NAS device, so > I do > > not see way to write to it without going through a network share. > > ahh, have we been assuming that this is NFS when all he said was NAS? No, he said NFS, at least in the last message (not sure about the beginning of this story...) Rainer > > if so, please clarify what protocol you are using to talk to the NAS. > it > could make a huge difference here. > > David Lang > > > Other than that, I agree to your statement. > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Alan T DeKok > >> Sent: Friday, October 23, 2009 8:40 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Enabling > >> $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) > to > >> NAS -cansomeone helpshed light? > >> > >> Rainer Gerhards wrote: > >>> So my conclusion really is that you need to look at the NFS layer. > >> > >> Logs should NEVER be written to an NFS mount. > >> > >> David gave a good explanation as to why. But the above > prohibition > >> should be made clear in the docs, if it isn't already. > >> > >> If the NFS server goes away, then the write transaction rate will > go > >> to zero. This likely isn't what people want from a syslog server. > >> > >> There is already a way to get syslog data from one system to > another: > >> the syslog protocol. Using NFS as a replacement for syslog is wrong > on > >> many levels. > >> > >> Alan DeKok. > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Oct 23 10:52:02 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 23 Oct 2009 01:52:02 -0700 (PDT) Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: <4AE15E5E.2020100@freeradius.org> References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> <4AE14FA7.4010008@freeradius.org> <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> <4AE15E5E.2020100@freeradius.org> Message-ID: On Fri, 23 Oct 2009, Alan T DeKok wrote: > Rainer Gerhards wrote: >> I think the main point of the OP was that he logs to a NAS device, so I do >> not see way to write to it without going through a network share. > > Configure rsyslog to write to local disk, and rsync periodically to > the network share. Or, use another instance of rsyslog to copy from > local disk to the network share. > > Any other method will have problems. Even NetApp systems have issues > from time to time. I've found that for every situation I've ever heard someone (including me) say 'never do that' that there is _some_ situaton where 'that' is exactly the right thing to do ;-) I agree that in almost all cases, writing logs to NFS is not a good idea. you really will need a high-end NFS server to have any chance of it working (and the load that the logging will put on the server is going to be significant, unless something does caching, which is what is causing your problems) so this is a _very_ expensive way to go to make logging work. if you are putting the logs on NFS to make them readable by a bunch of machines, consider running rsyslog on a linux system and let that linux system export the log directory via NFS. for read-only access from many machines, this is probably going to rivel the speed of a very expensive NFS server, while being much simpler, cheaper, and more reliable (if you need redundant hardware things get more complicated, but it's still not hard to make it reliable and it will definantly be much cheaper) David Lang From anichols at trumped.org Fri Oct 23 15:22:24 2009 From: anichols at trumped.org (Aaron Nichols) Date: Fri, 23 Oct 2009 07:22:24 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> <4AE14FA7.4010008@freeradius.org> <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> <4AE15E5E.2020100@freeradius.org> Message-ID: Thank you everyone for your responses - rather than reply to each post, here is some feedback. The storage system - by all measure we have a "high end" system - It's an EMC Clarion CX3-80 attached to a Celerra NS80 NAS head with 10gb interfaces. The network file system is NFS, the Celerra can also present CIFS shares but I doubt those would work any better. This is not easily replaced with a linux box so that I can do rsync. The logs must be made available to multiple hosts and our legacy implementation *is* a Linux box writing to direct attached disk sharing NFS... it is crumbling under our write/read load so we had to come up with another solution. This solution is understood to be a bit of a compromise - the ideal situation would be direct attached fiberchannel disk and a cluster filesystem but that wasn't an option - mostly for cost. For all the complaints about NFS it performs just fine - if we enable file sync the write load just goes through the roof, but the NAS keeps up. Client bufferring (or throttling) is still possible however because we are transitioning to this new system many of these systems are dual-logging to two destinations so their logging behavior can be observed on two different systems. On the old system running syslog-ng these messages are written to disk and visible near real-time and they don't seem to be doing any rate limiting or otherwise. The debug log that was taken however had file sync enable - and when that's enabled this delay is not seen so I'm not suprised it wasn't visible. I can produce a debug log with file sync disabled for comparison, that's probably something I should have done in the first place. I have tried an strace on the rsyslog process(es) when running normally and haven't been able to get much out of them but that could be my own limited knowledge of strace. `strace -p ` yields a line of information and then just stops, even though the rsyslog process is still working fine and is obviously doing something. I've tried attaching to the different threads as well with the same result. I will see if I can get a comparison debug log with file sync disabled & if there is some guidance on getting a good strace I'm happy to provide that output. Yesterday we had to move the server from RHEL 5.3 to RHEL 5.1 due to some discovered limits in tcp connections - either intentionally or unintentionally different in our installation of 5.3. The server is now performing (with file sync disabled) at an even higher capacity on RH5.1 however I haven't had a chance to review whether the write latency is still observed - hopefully today I can look at that. Thanks again for all the information. Aaron From rgerhards at hq.adiscon.com Fri Oct 23 15:31:18 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 15:31:18 +0200 Subject: [rsyslog] Enabling$ActionFileEnableSynccausedmassiveincreaseinwrite volume(bytes/sec) to NAS -cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com><4AE14FA7.4010008@freeradius.org><9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com><4AE15E5E.2020100@freeradius.org> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032A4@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Friday, October 23, 2009 3:22 PM > To: rsyslog-users > Subject: Re: [rsyslog] > Enabling$ActionFileEnableSynccausedmassiveincreaseinwrite > volume(bytes/sec) to NAS -cansomeone helpshed light? > > Thank you everyone for your responses - rather than reply to each > post, here is some feedback. > > The storage system - by all measure we have a "high end" system - It's > an EMC Clarion CX3-80 attached to a Celerra NS80 NAS head with 10gb > interfaces. The network file system is NFS, the Celerra can also > present CIFS shares but I doubt those would work any better. This is > not easily replaced with a linux box so that I can do rsync. The logs > must be made available to multiple hosts and our legacy implementation > *is* a Linux box writing to direct attached disk sharing NFS... it is > crumbling under our write/read load so we had to come up with another > solution. This solution is understood to be a bit of a compromise - > the ideal situation would be direct attached fiberchannel disk and a > cluster filesystem but that wasn't an option - mostly for cost. For > all the complaints about NFS it performs just fine - if we enable file > sync the write load just goes through the roof, but the NAS keeps up. > > Client bufferring (or throttling) is still possible however because we > are transitioning to this new system many of these systems are > dual-logging to two destinations so their logging behavior can be > observed on two different systems. On the old system running syslog-ng > these messages are written to disk and visible near real-time and they > don't seem to be doing any rate limiting or otherwise. Note that the debug run is *much* slower than the actual run. It's the usual problem, if you instrument the system you also affect it. Performance-wise, we have seen that a debug build can be up to ten times slower (sometimes more, depending on config), but even the regular debug log alone, if turned on, heavily changes performance and order of execution. The root cause is the many debug outputs, which require at least some mutex snyc and that obviously affects the overall system... > The debug log > that was taken however had file sync enable - and when that's enabled > this delay is not seen so I'm not suprised it wasn't visible. I can > produce a debug log with file sync disabled for comparison, that's > probably something I should have done in the first place. >From my POV, that actually doesn't help at all. The reason is that I don't see any unusual in the log, and that means all I can see is that data is being continously written... > > I have tried an strace on the rsyslog process(es) when running > normally and haven't been able to get much out of them but that could > be my own limited knowledge of strace. `strace -p ` yields a line > of information and then just stops, even though the rsyslog process is > still working fine and is obviously doing something. I've tried > attaching to the different threads as well with the same result. ... the strace will probably provide much better info, because here we see the API calls. One thing I forgot to mention is that you should NOT use it together with the debug log. The debug output api calls will make the strace output unreadable. For example, I have these line in one of my test scripts: strace -T -tt -f -F tools/rsyslogd -c..... I don't know out of my head what each option does, but that line may give you a starting point. The end result is not a single line but rather a complete log of each API called PLUS how much time the processing took. That, too, is a quite enormous log... Rainer > > I will see if I can get a comparison debug log with file sync disabled > & if there is some guidance on getting a good strace I'm happy to > provide that output. Yesterday we had to move the server from RHEL 5.3 > to RHEL 5.1 due to some discovered limits in tcp connections - either > intentionally or unintentionally different in our installation of 5.3. > The server is now performing (with file sync disabled) at an even > higher capacity on RH5.1 however I haven't had a chance to review > whether the write latency is still observed - hopefully today I can > look at that. > > Thanks again for all the information. > > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Oct 23 22:22:52 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 23 Oct 2009 13:22:52 -0700 (PDT) Subject: [rsyslog] queue configuration Message-ID: I know that you can create an action queue for a specific output. is there any way to create an action queue for multiple outputs? for example, in my configuration I have :fromhost, !isequal, "127.0.0.1" /var/log/messages;TraditionalFormat :fromhost, isequal, "127.0.0.1" @192.168.1.1;TraditionalForwardFormat *.* @192.168.1.115 *.* @192.168.1.241 *.* @192.168.1.242 *.* @192.168.1.6 *.* @192.168.1.7 *.* @192.168.1.122 :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 & @192.168.1.1;fixsnareForwardFormat2 & ~ :syslogtag, startswith, "MSWinEventLog#011" /var/log/messages;fixsnareFormat & @192.168.1.1;fixsnareForwardFormat & ~ *.* /var/log/messages;TraditionalFormat *.* @192.168.1.1 it would be nice to move the outbound relays off to a different queue, and to put the list of rules that have order dependancies (because they output in different formats to fix up formatting problem) to a seperate queue to spread the work across multiple processes and not slow down the basic writing to file. but as far as I can tell I would have to create a queue for each individual item, and I can't create a queue for the part of the config where I need to discard. am I missing something (including a better way to do everything ;-) David Lang From rgerhards at hq.adiscon.com Fri Oct 23 22:27:06 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 22:27:06 +0200 Subject: [rsyslog] queue configuration References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032AF@GRFEXC.intern.adiscon.com> In short: this can not currently be done and it would require considerable modification to the engine to do so. Part of the work I plan in moving queues to rule sets may actually help, but it is some way to there. Icluding that I first want to get a stable v5 and there are lots of smaller, but important things in front of it (like the DNS resolution issue. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 23, 2009 10:23 PM > To: rsyslog-users > Subject: [rsyslog] queue configuration > > I know that you can create an action queue for a specific output. > > is there any way to create an action queue for multiple outputs? > > for example, in my configuration I have > > :fromhost, !isequal, "127.0.0.1" > /var/log/messages;TraditionalFormat > :fromhost, isequal, "127.0.0.1" > @192.168.1.1;TraditionalForwardFormat > *.* @192.168.1.115 > *.* @192.168.1.241 > *.* @192.168.1.242 > *.* @192.168.1.6 > *.* @192.168.1.7 > *.* @192.168.1.122 > :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 > & @192.168.1.1;fixsnareForwardFormat2 > & ~ > :syslogtag, startswith, "MSWinEventLog#011" > /var/log/messages;fixsnareFormat > & @192.168.1.1;fixsnareForwardFormat > & ~ > *.* /var/log/messages;TraditionalFormat > *.* @192.168.1.1 > > it would be nice to move the outbound relays off to a > different queue, and > to put the list of rules that have order dependancies > (because they output > in different formats to fix up formatting problem) to a > seperate queue to > spread the work across multiple processes and not slow down the basic > writing to file. > > but as far as I can tell I would have to create a queue for each > individual item, and I can't create a queue for the part of > the config > where I need to discard. > > am I missing something (including a better way to do everything ;-) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Oct 23 22:33:35 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 23 Oct 2009 13:33:35 -0700 (PDT) Subject: [rsyslog] queue configuration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032AF@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71032AF@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 23 Oct 2009, Rainer Gerhards wrote: > In short: this can not currently be done and it would require considerable > modification to the engine to do so. Part of the work I plan in moving queues > to rule sets may actually help, but it is some way to there. Icluding that I > first want to get a stable v5 and there are lots of smaller, but important > things in front of it (like the DNS resolution issue. that's what I suspected, I just wanted to make sure. with the new queue engine can you have multiple worker threads? (I seem to remember that initially you couldn't with batch mode, and in almost every case there is no reason to as a single thread in batch mode can do so much more than it could before) David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Friday, October 23, 2009 10:23 PM >> To: rsyslog-users >> Subject: [rsyslog] queue configuration >> >> I know that you can create an action queue for a specific output. >> >> is there any way to create an action queue for multiple outputs? >> >> for example, in my configuration I have >> >> :fromhost, !isequal, "127.0.0.1" >> /var/log/messages;TraditionalFormat >> :fromhost, isequal, "127.0.0.1" >> @192.168.1.1;TraditionalForwardFormat >> *.* @192.168.1.115 >> *.* @192.168.1.241 >> *.* @192.168.1.242 >> *.* @192.168.1.6 >> *.* @192.168.1.7 >> *.* @192.168.1.122 >> :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 >> & @192.168.1.1;fixsnareForwardFormat2 >> & ~ >> :syslogtag, startswith, "MSWinEventLog#011" >> /var/log/messages;fixsnareFormat >> & @192.168.1.1;fixsnareForwardFormat >> & ~ >> *.* /var/log/messages;TraditionalFormat >> *.* @192.168.1.1 >> >> it would be nice to move the outbound relays off to a >> different queue, and >> to put the list of rules that have order dependancies >> (because they output >> in different formats to fix up formatting problem) to a >> seperate queue to >> spread the work across multiple processes and not slow down the basic >> writing to file. >> >> but as far as I can tell I would have to create a queue for each >> individual item, and I can't create a queue for the part of >> the config >> where I need to discard. >> >> am I missing something (including a better way to do everything ;-) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Oct 23 22:35:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 22:35:56 +0200 Subject: [rsyslog] queue configuration References: <9B6E2A8877C38245BFB15CC491A11DA71032AF@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032B1@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 23, 2009 10:34 PM > To: rsyslog-users > Subject: Re: [rsyslog] queue configuration > > On Fri, 23 Oct 2009, Rainer Gerhards wrote: > > > In short: this can not currently be done and it would > require considerable > > modification to the engine to do so. Part of the work I > plan in moving queues > > to rule sets may actually help, but it is some way to > there. Icluding that I > > first want to get a stable v5 and there are lots of > smaller, but important > > things in front of it (like the DNS resolution issue. > > that's what I suspected, I just wanted to make sure. > > with the new queue engine can you have multiple worker > threads? (I seem to > remember that initially you couldn't with batch mode, and in > almost every > case there is no reason to as a single thread in batch mode > can do so much > more than it could before) Yes, you can do that BUT not for disk queues. This was the restriction. It may be that one early stage of batching support limited all workers to one. But we can do multiple workers. Rainer > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > >> Sent: Friday, October 23, 2009 10:23 PM > >> To: rsyslog-users > >> Subject: [rsyslog] queue configuration > >> > >> I know that you can create an action queue for a specific output. > >> > >> is there any way to create an action queue for multiple outputs? > >> > >> for example, in my configuration I have > >> > >> :fromhost, !isequal, "127.0.0.1" > >> /var/log/messages;TraditionalFormat > >> :fromhost, isequal, "127.0.0.1" > >> @192.168.1.1;TraditionalForwardFormat > >> *.* @192.168.1.115 > >> *.* @192.168.1.241 > >> *.* @192.168.1.242 > >> *.* @192.168.1.6 > >> *.* @192.168.1.7 > >> *.* @192.168.1.122 > >> :hostname, contains ,"MSWinEventLog" > /var/log/messages;fixsnareFormat2 > >> & @192.168.1.1;fixsnareForwardFormat2 > >> & ~ > >> :syslogtag, startswith, "MSWinEventLog#011" > >> /var/log/messages;fixsnareFormat > >> & @192.168.1.1;fixsnareForwardFormat > >> & ~ > >> *.* /var/log/messages;TraditionalFormat > >> *.* @192.168.1.1 > >> > >> it would be nice to move the outbound relays off to a > >> different queue, and > >> to put the list of rules that have order dependancies > >> (because they output > >> in different formats to fix up formatting problem) to a > >> seperate queue to > >> spread the work across multiple processes and not slow > down the basic > >> writing to file. > >> > >> but as far as I can tell I would have to create a queue for each > >> individual item, and I can't create a queue for the part of > >> the config > >> where I need to discard. > >> > >> am I missing something (including a better way to do everything ;-) > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Sat Oct 24 21:10:32 2009 From: david at lang.hm (david at lang.hm) Date: Sat, 24 Oct 2009 12:10:32 -0700 (PDT) Subject: [rsyslog] is there a way to find currnt queue status? Message-ID: thinking back to the testing of the v5 code that I have been doing this last week, one thing that would be very useful is if there is a way to find out what the current queue lengths are. one possible way to do this would be to have rsyslog watch for a signal and when it receives it, spit out the queue stats. for the stats definantly the length of each queue, possibly timestamps of oldest and newest message in each queue when spitting them out, some possible approaches are to stderr pro: simple con: stderr may not appear if rsyslog is running as a daemon from startup add a log message to the main queue containing the data pro: still fairly simple con: if there is a long queue, or it is blocked the data may not be visable for a while have each queue walker thread generate a message internally and process it pro: the data will show up fast, through the normal log mechanism con: this is probably the most work to imeplement this is not a critical issue, but it could be something that would speed up development in the long run by making it easier to figure out the state of the system. or I could have just missed how something like this is already supported ;-) David Lang From rgerhards at hq.adiscon.com Sat Oct 24 21:14:57 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 24 Oct 2009 21:14:57 +0200 Subject: [rsyslog] is there a way to find currnt queue status? References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032B2@GRFEXC.intern.adiscon.com> It's very partially being supported right now via imdiag. That module enables to query the main message queue length (and only for the main message queue). There is also a very, very rudamentary Java app in the v5. This was created as part of the debugging/performance testing effort earlier this year, but unfortunately more important things got into the way. An interace needs to be designed (and implemented) for performance counters. IT's on the agenda, but unfortunately not on the top spot. In concur that it would be very useful in many cases. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Saturday, October 24, 2009 9:11 PM > To: rsyslog-users > Subject: [rsyslog] is there a way to find currnt queue status? > > thinking back to the testing of the v5 code that I have been > doing this > last week, one thing that would be very useful is if there is > a way to > find out what the current queue lengths are. > > one possible way to do this would be to have rsyslog watch > for a signal > and when it receives it, spit out the queue stats. > > for the stats definantly the length of each queue, possibly > timestamps of > oldest and newest message in each queue > > when spitting them out, some possible approaches are > > to stderr > pro: simple > con: stderr may not appear if rsyslog is running as a > daemon from startup > > add a log message to the main queue containing the data > pro: still fairly simple > con: if there is a long queue, or it is blocked the data > may not be > visable for a while > > have each queue walker thread generate a message internally > and process it > pro: the data will show up fast, through the normal log mechanism > con: this is probably the most work to imeplement > > this is not a critical issue, but it could be something that > would speed > up development in the long run by making it easier to figure > out the state > of the system. > > or I could have just missed how something like this is > already supported > ;-) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Mon Oct 26 08:17:23 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 26 Oct 2009 08:17:23 +0100 Subject: [rsyslog] queue configuration References: <9B6E2A8877C38245BFB15CC491A11DA71032AF@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032B5@GRFEXC.intern.adiscon.com> David, all, I have thought a bit more about this. Actually, I think almost everything is already present by now, just some glue is missing. The most important thing, ruleset-specific "main" queues was due to implement soon (I worked towards that the past two month). So, in short, I'll have a look at it this week while running the stability lab tests (they always take considerable time). I don't want to promise anything, but it may be possible that this feature could become available shortly. That would also build that basis for some considerabe performance improvement (for some environments), so it seems to be very relevant to my short-term goals. Thanks for bringing this up and making me think about it ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 23, 2009 10:34 PM > To: rsyslog-users > Subject: Re: [rsyslog] queue configuration > > On Fri, 23 Oct 2009, Rainer Gerhards wrote: > > > In short: this can not currently be done and it would > require considerable > > modification to the engine to do so. Part of the work I > plan in moving queues > > to rule sets may actually help, but it is some way to > there. Icluding that I > > first want to get a stable v5 and there are lots of > smaller, but important > > things in front of it (like the DNS resolution issue. > > that's what I suspected, I just wanted to make sure. > > with the new queue engine can you have multiple worker > threads? (I seem to > remember that initially you couldn't with batch mode, and in > almost every > case there is no reason to as a single thread in batch mode > can do so much > more than it could before) > > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > >> Sent: Friday, October 23, 2009 10:23 PM > >> To: rsyslog-users > >> Subject: [rsyslog] queue configuration > >> > >> I know that you can create an action queue for a specific output. > >> > >> is there any way to create an action queue for multiple outputs? > >> > >> for example, in my configuration I have > >> > >> :fromhost, !isequal, "127.0.0.1" > >> /var/log/messages;TraditionalFormat > >> :fromhost, isequal, "127.0.0.1" > >> @192.168.1.1;TraditionalForwardFormat > >> *.* @192.168.1.115 > >> *.* @192.168.1.241 > >> *.* @192.168.1.242 > >> *.* @192.168.1.6 > >> *.* @192.168.1.7 > >> *.* @192.168.1.122 > >> :hostname, contains ,"MSWinEventLog" > /var/log/messages;fixsnareFormat2 > >> & @192.168.1.1;fixsnareForwardFormat2 > >> & ~ > >> :syslogtag, startswith, "MSWinEventLog#011" > >> /var/log/messages;fixsnareFormat > >> & @192.168.1.1;fixsnareForwardFormat > >> & ~ > >> *.* /var/log/messages;TraditionalFormat > >> *.* @192.168.1.1 > >> > >> it would be nice to move the outbound relays off to a > >> different queue, and > >> to put the list of rules that have order dependancies > >> (because they output > >> in different formats to fix up formatting problem) to a > >> seperate queue to > >> spread the work across multiple processes and not slow > down the basic > >> writing to file. > >> > >> but as far as I can tell I would have to create a queue for each > >> individual item, and I can't create a queue for the part of > >> the config > >> where I need to discard. > >> > >> am I missing something (including a better way to do everything ;-) > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From tbergfeld at hq.adiscon.com Tue Oct 27 11:12:18 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Tue, 27 Oct 2009 11:12:18 +0100 Subject: [rsyslog] rsyslog 5.3.3 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032CC@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.3.3, a member of the devel branch. This version offers a partially rewritten queue engine. The functionality was streamlined and simplified, resulting in not only fewer bug potential but also greater speed. This engine sets an important basis on which all further work can build. Note that due to the magnitude of changes, there is some increased bug potential. But in general, the new engine will be able to better handle some situations where the previous engine ran into problems. Also there are some bug fixes. This is a recommended update for all users of the devel branch. ChangeLog: http://www.rsyslog.com/Article419.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-183.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From martinmie at PartyGaming.com Tue Oct 27 11:56:52 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Tue, 27 Oct 2009 11:56:52 +0100 Subject: [rsyslog] rsyslog in uninterruptable sleep Message-ID: <0B1B9163138571439EAF804646F3F6AA0892EE19@GIBSVWIN004X.partygaming.local> Hi, I just did some small changes to the rsyslog.conf and wanted to restart the process: # /etc/init.d/rsyslog restart Shutting down system logger: [FAILED] Starting system logger: Already running. [FAILED] # ps -elf | grep rsyslog 5 D root 2037 1 1 76 0 - 12040 sync_b Oct18 ? 03:32:10 rsyslogd -c 4 -x -f /etc/rsyslog.conf -i /var/run/syslogd.pid This is rsyslog 4.2.0 running on RHEL5 and it's not the first time it happens. Any ideas on why this might happened and how to solve this w/o rebooting the system?? Thanks, Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From martinmie at PartyGaming.com Tue Oct 27 11:59:19 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Tue, 27 Oct 2009 11:59:19 +0100 Subject: [rsyslog] rsyslog and snare Message-ID: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> Hi, We recently started to receive logs from a Snare client. I applied the configuration changes proposed here: http://wiki.rsyslog.com/index.php/Using_Snare_as_a_client_on_Windows After restarting rsyslog, I get the following error: -- 2009-10-27T06:45:39.067279-04:00 logserger rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="29021" x-info="http://www.rsyslog.com"] (re)start 2009-10-27T06:45:39.063213-04:00 logserger rsyslogd-3000: unknown priority name "" [try http://www.rsyslog.com/e/3000 ] 2009-10-27T06:45:39.063571-04:00 logserger rsyslogd: the last error occured in /etc/rsyslog.conf, line 18 2009-10-27T06:45:39.064044-04:00 logserger rsyslogd: warning: selector line without actions will be discarded 2009-10-27T06:45:39.064213-04:00 logserger rsyslogd-2123: CONFIG ERROR: could not interpret master config file '/etc/rsyslog.conf'. [try http://www.rsyslog.com/e/2123 ] -- If I comment out the "%msg:::space-cc%" part it starts up cleanly but I assume that the filtering won't work. Suggestions? Thanks, Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From rgerhards at hq.adiscon.com Tue Oct 27 16:02:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 27 Oct 2009 16:02:17 +0100 Subject: [rsyslog] rsyslog in uninterruptable sleep References: <0B1B9163138571439EAF804646F3F6AA0892EE19@GIBSVWIN004X.partygaming.local> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032D3@GRFEXC.intern.adiscon.com> I think we just recently had this on the mailing list or bug tracker and I made a fix for it. I guess the fix is included in 4.4.2. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Martin Mielke > Sent: Tuesday, October 27, 2009 11:57 AM > To: rsyslog-users > Subject: [rsyslog] rsyslog in uninterruptable sleep > > > Hi, > > I just did some small changes to the rsyslog.conf and wanted to restart > the process: > > # /etc/init.d/rsyslog restart > Shutting down system logger: [FAILED] > Starting system logger: Already running. > [FAILED] > > # ps -elf | grep rsyslog > 5 D root 2037 1 1 76 0 - 12040 sync_b Oct18 ? > 03:32:10 rsyslogd -c 4 -x -f /etc/rsyslog.conf -i /var/run/syslogd.pid > > > This is rsyslog 4.2.0 running on RHEL5 and it's not the first time it > happens. > > > Any ideas on why this might happened and how to solve this w/o > rebooting > the system?? > > > Thanks, > Martin > > > This email and any attachments are confidential, and may be legally > privileged and protected by copyright. If you are not the intended > recipient dissemination or copying of this email is prohibited. If you > have received this in error, please notify the sender by replying by > email and then delete the email completely from your system. > > Any views or opinions are solely those of the sender. This > communication is not intended to form a binding contract unless > expressly indicated to the contrary and properly authorised. Any > actions taken on the basis of this email are at the recipient's own > risk. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Oct 27 17:53:20 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 27 Oct 2009 17:53:20 +0100 Subject: [rsyslog] important enhancement in rsyslog upcoming Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032D6@GRFEXC.intern.adiscon.com> Hi all, I was finally able to add some new featueres to rsyslog (stability is still my priority), and I think it is a very exciting one. In short: we can now partition input message into different queues, dramatically decreasing lock contention potential. Please read yourself: http://blog.gerhards.net/2009/10/next-round-of-performance-enhancement.html The change is committed, but I'll probably be busy a day or two with lab testing and testbench enhancement before I continue to add more enhancements. Rainer From epiphani at gmail.com Tue Oct 27 17:59:27 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 27 Oct 2009 12:59:27 -0400 Subject: [rsyslog] important enhancement in rsyslog upcoming In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032D6@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71032D6@GRFEXC.intern.adiscon.com> Message-ID: This looks like it has a lot of potential! I look forward to trying it. -Aaron On Tue, Oct 27, 2009 at 12:53 PM, Rainer Gerhards wrote: > Hi all, > > I was finally able to add some new featueres to rsyslog (stability is still > my priority), and I think it is a very exciting one. In short: we can now > partition input message into different queues, dramatically decreasing lock > contention potential. > > Please read yourself: > > http://blog.gerhards.net/2009/10/next-round-of-performance-enhancement.html > > The change is committed, but I'll probably be busy a day or two with lab > testing and testbench enhancement before I continue to add more enhancements. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Oct 27 18:09:06 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 27 Oct 2009 18:09:06 +0100 Subject: [rsyslog] important enhancement in rsyslog upcoming References: <9B6E2A8877C38245BFB15CC491A11DA71032D6@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032D7@GRFEXC.intern.adiscon.com> As a side-note, this does not yet mean that the input module itself runs on multiple threads. So the new functionality probably works best across different inputs. Massiv-parallel inputs are on my agenda as well, but some other things need to be done first so that I will have the necessary basis to build on (I'll not use a simple "one thread per connection" model, as this does not scale well). I'll probably go for multiplexed io with potentially one queue per worker, and quick worker engagement into that queue where no wait is necessary ... but more later ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Tuesday, October 27, 2009 5:59 PM > To: rsyslog-users > Subject: Re: [rsyslog] important enhancement in rsyslog upcoming > > This looks like it has a lot of potential! I look forward to trying > it. > > -Aaron > > On Tue, Oct 27, 2009 at 12:53 PM, Rainer Gerhards > wrote: > > Hi all, > > > > I was finally able to add some new featueres to rsyslog (stability is > still > > my priority), and I think it is a very exciting one. In short: we can > now > > partition input message into different queues, dramatically > decreasing lock > > contention potential. > > > > Please read yourself: > > > > http://blog.gerhards.net/2009/10/next-round-of-performance- > enhancement.html > > > > The change is committed, but I'll probably be busy a day or two with > lab > > testing and testbench enhancement before I continue to add more > enhancements. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jbondc at openmv.com Tue Oct 27 20:40:42 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Tue, 27 Oct 2009 15:40:42 -0400 Subject: [rsyslog] Property-Based Filters, not working? Message-ID: <002101ca573d$5f417f70$1dc47e50$@com> I tried adding the following to my config file: :msg,contains,"MSWinEventLog" *.* /var/log/windows.log :msg, !contains, "MSWinEventLog" *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err /var/log/messages security.* /var/log/security auth.info;authpriv.info /var/log/auth.log However the filter doesn't seem to apply, when debugging the parse sequence goes like this: 5250.008750501:main thread: Action 0x80a8600: queue 0x8094c00 created 5250.008823471:main thread: cfline: ':msg,contains,"MSWinEventLog"' 5250.008884805:main thread: selector line successfully processed 5250.008943799:main thread: - property-based filter 5250.009059262:main thread: tried selector action for builtin-file: -2001 5250.009130893:main thread: tried selector action for builtin-fwd: -2001 5250.009195089:main thread: tried selector action for builtin-shell: -2001 5250.009259497:main thread: tried selector action for builtin-discard: -2001 5250.009324602:main thread: tried selector action for builtin-usrmsg: -2001 5250.009394008:main thread: tried selector action for ompgsql.so: -2001 5250.009454057:main thread: config line NOT successfully processed Am I missing something? rsyslogd 4.4.2, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: Yes FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: No FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: No Runtime Instrumentation (slow code): No From david at lang.hm Tue Oct 27 21:03:01 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 27 Oct 2009 13:03:01 -0700 (PDT) Subject: [rsyslog] Property-Based Filters, not working? In-Reply-To: <002101ca573d$5f417f70$1dc47e50$@com> References: <002101ca573d$5f417f70$1dc47e50$@com> Message-ID: On Tue, 27 Oct 2009, Jonathan Bond-Caron wrote: > Date: Tue, 27 Oct 2009 15:40:42 -0400 > From: Jonathan Bond-Caron > Reply-To: rsyslog-users > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Property-Based Filters, not working? > > I tried adding the following to my config file: > > > > :msg,contains,"MSWinEventLog" > > > > *.* /var/log/windows.log > > > > :msg, !contains, "MSWinEventLog" > > > > *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err > /var/log/messages this is not how the property based filters work. I made the same mistake instead of :msg,contains,"MSWinEventLog" *.* /var/log/windows.log it should be :msg,contains,"MSWinEventLog" /var/log/windows.log when you do filter rules those filters are everything, you don't also do facility/priority filters as well (or if you need to do so, you need to make a compound if (() and ()) type rule) FYI, what I do with snare is $template fixsnareFormat,"%timereported% %HOSTNAME% MSWinEventLog %syslogtag:18:$%%msg:::drop-last-lf%\n" $template fixsnareForwardFormat,"<%pri%>%timereported% %HOSTNAME% MSWinEventLog %syslogtag:18:$:%%msg:::drop-last-lf%\n" $template fixsnareFormat2,"%timereported% %fromhost-ip% broken-MSWinEventLog %HOSTNAME% %syslogtag:18:$%%msg:::drop-last-lf%\n" $template fixsnareForwardFormat2,"<%pri%>%timereported% %fromhost-ip% broken-MSWinEventLog %HOSTNAME% %syslogtag:18:$:%%msg:::drop-last-lf%\n" :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 & @192.168.210.8;fixsnareForwardFormat2 & ~ :syslogtag, startswith, "MSWinEventLog\011" /var/log/messages;fixsnareFormat & @192.168.210.8;fixsnareForwardFormat & ~ *.* /var/log/messages;TraditionalFormat *.* @192.168.210.8 the fixsnare*2 formats are to handle cases where I haven't tracked down why snare is sending things incorrectly (8 of a couple hundred servers) David Lang > security.* /var/log/security > > auth.info;authpriv.info /var/log/auth.log > > > > > > However the filter doesn't seem to apply, when debugging the parse sequence > goes like this: > > > > 5250.008750501:main thread: Action 0x80a8600: queue 0x8094c00 created > > 5250.008823471:main thread: cfline: ':msg,contains,"MSWinEventLog"' > > 5250.008884805:main thread: selector line successfully processed > > 5250.008943799:main thread: - property-based filter > > 5250.009059262:main thread: tried selector action for builtin-file: -2001 > > 5250.009130893:main thread: tried selector action for builtin-fwd: -2001 > > 5250.009195089:main thread: tried selector action for builtin-shell: -2001 > > 5250.009259497:main thread: tried selector action for builtin-discard: -2001 > > 5250.009324602:main thread: tried selector action for builtin-usrmsg: -2001 > > 5250.009394008:main thread: tried selector action for ompgsql.so: -2001 > > 5250.009454057:main thread: config line NOT successfully processed > > > > Am I missing something? > > > > rsyslogd 4.4.2, compiled with: > > FEATURE_REGEXP: Yes > > FEATURE_LARGEFILE: Yes > > FEATURE_NETZIP (message compression): Yes > > GSSAPI Kerberos 5 support: No > > FEATURE_DEBUG (debug build, slow code): No > > Atomic operations supported: No > > Runtime Instrumentation (slow code): No > > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Tue Oct 27 21:05:47 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 27 Oct 2009 13:05:47 -0700 (PDT) Subject: [rsyslog] control character escaping change? Message-ID: it appears that the 5.3 development series recently changed the default for escaping control characters from # to \ so a tab changed from #011 to \011 was this a deliberate change going forward, or a mistake? David Lang From david at lang.hm Tue Oct 27 21:32:39 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 27 Oct 2009 13:32:39 -0700 (PDT) Subject: [rsyslog] rsyslog and snare In-Reply-To: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> Message-ID: On Tue, 27 Oct 2009, Martin Mielke wrote: > Hi, > > We recently started to receive logs from a Snare client. > I applied the configuration changes proposed here: > http://wiki.rsyslog.com/index.php/Using_Snare_as_a_client_on_Windows I just got a chance to read this, and I disagree with it's recommendation the #011 is a tab. snare uses it to seperate fields from the windows event log. if you just replace it with a space you loose the ability to do things like look for field 10 because field 8 may contain spaces. In addition there are two formats of logs that snare can output to syslog 1. the snare format 2. 'syslog' format the difference seems to be that in snare format it does Jan 1 01:01:01 mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri while in syslog format it does Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri the biggest change is the space between the host and MSWinEventLog there is also sometime a "1#011" before the field with Security in it in this example and sometimes not. I have not figured out the pattern here yet (it may be the format difference, my parsing logic just looks for a '1' in the first field, and if it finds it, drops that field) what I do is to setup the following rules to reformat the syslog formatted messages to look like Jan 1 01:01:01 mail.abc.com MSWinEventLog 1#011Security#0114169#011Fri I can then filter on the program name MSWinEventLog and later parsing tools can examine specific fields of the log message I have not fully figured out the snare formatted message, but the *2 formats at lease prevent lots of garbage from ending up in the host and program fields. $template fixsnareFormat,"%timereported% %HOSTNAME% MSWinEventLog %syslogtag:18:$%%msg:::drop-last-lf%\n" $template fixsnareForwardFormat,"<%pri%>%timereported% %HOSTNAME% MSWinEventLog %syslogtag:18:$:%%msg:::drop-last-lf%\n" $template fixsnareFormat2,"%timereported% %fromhost-ip% broken-MSWinEventLog %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" $template fixsnareForwardFormat2,"<%pri%>%timereported% %fromhost-ip% broken-MSWinEventLog %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 & @192.168.1.8;fixsnareForwardFormat2 & ~ :syslogtag, startswith, "MSWinEventLog#011" /var/log/messages;fixsnareFormat & @192.168.1.8;fixsnareForwardFormat & ~ *.* /var/log/messages;TraditionalFormat *.* @192.168.1.8 I updated to wiki to show this option > After restarting rsyslog, I get the following error: > -- > 2009-10-27T06:45:39.067279-04:00 logserger rsyslogd: [origin > software="rsyslogd" swVersion="4.2.0" x-pid="29021" > x-info="http://www.rsyslog.com"] (re)start > 2009-10-27T06:45:39.063213-04:00 logserger rsyslogd-3000: unknown > priority name "" [try http://www.rsyslog.com/e/3000 ] > 2009-10-27T06:45:39.063571-04:00 logserger rsyslogd: the last error > occured in /etc/rsyslog.conf, line 18 > 2009-10-27T06:45:39.064044-04:00 logserger rsyslogd: warning: selector > line without actions will be discarded > 2009-10-27T06:45:39.064213-04:00 logserger rsyslogd-2123: CONFIG ERROR: > could not interpret master config file '/etc/rsyslog.conf'. [try > http://www.rsyslog.com/e/2123 ] > -- > > If I comment out the "%msg:::space-cc%" part it starts up cleanly but I > assume that the filtering won't work. the %msg:::space-cc% shouldn't have been listed as a seperate line, that would replace %msg% in a format string. David Lang > > Suggestions? > > > Thanks, > Martin > > > > > > This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. > > Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Oct 27 22:09:58 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 27 Oct 2009 22:09:58 +0100 Subject: [rsyslog] control character escaping change? References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032D8@GRFEXC.intern.adiscon.com> This smells like a mistake. At least I don't remember changing that, but I'll look at the changelog tomorrow... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, October 27, 2009 9:06 PM > To: rsyslog-users > Subject: [rsyslog] control character escaping change? > > it appears that the 5.3 development series recently changed > the default > for escaping control characters from # to \ so a tab changed > from #011 to > \011 > > was this a deliberate change going forward, or a mistake? > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From r.bhatia at ipax.at Wed Oct 28 11:40:09 2009 From: r.bhatia at ipax.at (Raoul Bhatia [IPAX]) Date: Wed, 28 Oct 2009 11:40:09 +0100 Subject: [rsyslog] version numbers from git repository In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103292@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103292@GRFEXC.intern.adiscon.com> Message-ID: <4AE81F89.3070409@ipax.at> On 10/23/2009 07:54 AM, Rainer Gerhards wrote: > I've looked a bit into the issue. Neither of them use the GNU build system. > In essence, it boils down to running "git describe" as part of the build > process, generating a file with it. However, I do not know how I may achieve > that. It should probably be run during the ./configure stage. Maybe someone > can provide some advise ;) i like the way drbd includes the git-hash in its version output: > # cat /proc/drbd > version: 8.3.2 (api:88/proto:86-90) > GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by phil at fat-tyre, 2009-07-03 15:35:39 > ... take a look at line 112ff at (especially line 125) http://git.drbd.org/?p=drbd-8.3.git;a=blob;f=drbd/Makefile;hb=HEAD there, a drbd_buildtag.c file is created with the help of: > git rev-parse HEAD the output of this Makefile section is: > /* automatically generated. DO NOT EDIT. */ > #include > const char * drbd_buildtag(void) > { > return "GIT-hash: 9ce425f51860f1205395bf7127cad3c427070837" > " build by raoul at dev.ipax.at, 2008-09-04 17:00:24"; > } one can use this, or something similar using e.g. git-describe. cheers, raoul -- ____________________________________________________________________ DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at Technischer Leiter IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at Barawitzkagasse 10/2/2/11 email. office at ipax.at 1190 Wien tel. +43 1 3670030 FN 277995t HG Wien fax. +43 1 3670030 15 ____________________________________________________________________ From rgerhards at hq.adiscon.com Wed Oct 28 12:06:32 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 12:06:32 +0100 Subject: [rsyslog] version numbers from git repository References: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103292@GRFEXC.intern.adiscon.com> <4AE81F89.3070409@ipax.at> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032DC@GRFEXC.intern.adiscon.com> Hi Raoul, this is useful, but unfortunately it does not tell how to do it with GNU autotools. I'll keep this on my mind when I try to tackle it, but so far it looks like I need too much time to find out the details. If someone could suggest a solution within the GNU build system, I'd love to include that ASAP. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX] > Sent: Wednesday, October 28, 2009 11:40 AM > To: rsyslog-users > Subject: Re: [rsyslog] version numbers from git repository > > On 10/23/2009 07:54 AM, Rainer Gerhards wrote: > > I've looked a bit into the issue. Neither of them use the GNU build > system. > > In essence, it boils down to running "git describe" as part of the > build > > process, generating a file with it. However, I do not know how I may > achieve > > that. It should probably be run during the ./configure stage. Maybe > someone > > can provide some advise ;) > > i like the way drbd includes the git-hash in its version output: > > # cat /proc/drbd > > version: 8.3.2 (api:88/proto:86-90) > > GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by phil at fat- > tyre, 2009-07-03 15:35:39 > > ... > > take a look at line 112ff at (especially line 125) > http://git.drbd.org/?p=drbd-8.3.git;a=blob;f=drbd/Makefile;hb=HEAD > > there, a drbd_buildtag.c file is created with the help of: > > git rev-parse HEAD > > the output of this Makefile section is: > > /* automatically generated. DO NOT EDIT. */ > > #include > > const char * drbd_buildtag(void) > > { > > return "GIT-hash: 9ce425f51860f1205395bf7127cad3c427070837" > > " build by raoul at dev.ipax.at, 2008-09-04 17:00:24"; > > } > > one can use this, or something similar using e.g. git-describe. > > cheers, > raoul > -- > ____________________________________________________________________ > DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at > Technischer Leiter > > IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at > Barawitzkagasse 10/2/2/11 email. office at ipax.at > 1190 Wien tel. +43 1 3670030 > FN 277995t HG Wien fax. +43 1 3670030 15 > ____________________________________________________________________ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jbondc at openmv.com Wed Oct 28 12:32:35 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Wed, 28 Oct 2009 07:32:35 -0400 Subject: [rsyslog] Property-Based Filters, not working? In-Reply-To: References: <002101ca573d$5f417f70$1dc47e50$@com> Message-ID: <001b01ca57c2$593248a0$0b96d9e0$@com> On Tue Oct 27 04:03 PM, david at lang.hm wrote: > On Tue, 27 Oct 2009, Jonathan Bond-Caron wrote: > > > Date: Tue, 27 Oct 2009 15:40:42 -0400 > > From: Jonathan Bond-Caron > > Reply-To: rsyslog-users > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] Property-Based Filters, not working? > > > > I tried adding the following to my config file: > > > > > > :msg,contains,"MSWinEventLog" > > > > *.* /var/log/windows.log > > > > :msg, !contains, "MSWinEventLog" > > > > *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err > > /var/log/messages > > this is not how the property based filters work. > > I made the same mistake > > instead of > > :msg,contains,"MSWinEventLog" *.* /var/log/windows.log > > it should be > > :msg,contains,"MSWinEventLog" /var/log/windows.log > > when you do filter rules those filters are everything, you don't also > do facility/priority filters as well (or if you need to do so, you > need to make a compound if (() and ()) type rule) > Thanks a lot, that works well. The docs are bit misleading: http://www.rsyslog.com/doc-rsyslog_conf_filter.html It could show something like this: :msg, contains, "error" /var/log/error.log :msg, !contains, "error" ~ All messages that contain the word "error" are logged to /var/log/error.log All messages that do not contain the word "error" are thrown away. From rgerhards at hq.adiscon.com Wed Oct 28 12:35:25 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 12:35:25 +0100 Subject: [rsyslog] Property-Based Filters, not working? References: <002101ca573d$5f417f70$1dc47e50$@com> <001b01ca57c2$593248a0$0b96d9e0$@com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032DD@GRFEXC.intern.adiscon.com> > Thanks a lot, that works well. > > The docs are bit misleading: > http://www.rsyslog.com/doc-rsyslog_conf_filter.html > > It could show something like this: > :msg, contains, "error" /var/log/error.log > :msg, !contains, "error" ~ > > All messages that contain the word "error" are logged to > /var/log/error.log > All messages that do not contain the word "error" are thrown away. The doc in general could be much improved (any volunteers?), but here I think it is right. The link you quote defines what filters are, but what you post is not only a filter but a full selector line, consisting from a filter and the associated action. I agree that it would be useful to have more scenario-based cases which contain all the pieces put together (again, any volunteers?). Rainer From jbondc at openmv.com Wed Oct 28 13:25:40 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Wed, 28 Oct 2009 08:25:40 -0400 Subject: [rsyslog] Property-Based Filters, not working? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032DD@GRFEXC.intern.adiscon.com> References: <002101ca573d$5f417f70$1dc47e50$@com> <001b01ca57c2$593248a0$0b96d9e0$@com> <9B6E2A8877C38245BFB15CC491A11DA71032DD@GRFEXC.intern.adiscon.com> Message-ID: <001c01ca57c9$c3870810$4a951830$@com> On Wed Oct 28 07:35 AM, Rainer Gerhards wrote: > > Thanks a lot, that works well. > > > > The docs are bit misleading: > > http://www.rsyslog.com/doc-rsyslog_conf_filter.html > > > > It could show something like this: > > :msg, contains, "error" /var/log/error.log > > :msg, !contains, "error" ~ > > > > All messages that contain the word "error" are logged to > > /var/log/error.log All messages that do not contain the word "error" > > are thrown away. > > The doc in general could be much improved (any volunteers?), but here > I think it is right. The link you quote defines what filters are, but > what you post is not only a filter but a full selector line, > consisting from a filter and the associated action. > > I agree that it would be useful to have more scenario-based cases > which contain all the pieces put together (again, any volunteers?). Noted ;) I'll prepare a patch for 4.4.2 this week, will also fix an issue with the syslog 'tag' parsing. It should terminate when it sees a tab or escape control character. Makes it a little more RFC3164 compliant. From a.smith at ukgrid.net Wed Oct 28 13:53:21 2009 From: a.smith at ukgrid.net (Andy Smith) Date: Wed, 28 Oct 2009 12:53:21 +0000 Subject: [rsyslog] config prob with mail.info messages Message-ID: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> Hi, I?m finding rsyslog is logging all routine (I assume mail.info) mail messages to the main messages file (FreeBSD 7.0). Can someone suggest why based on the config: *.err;kern.warning;auth.notice;mail.crit;local7.none /dev/console *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err;local7.none /var/log/messages security.* /var/log/security auth.info;authpriv.info /var/log/auth.log mail.info /var/log/maillog lpr.info /var/log/lpd-errs ftp.info /var/log/xferlog cron.* /var/log/cron *.=debug;local7.none /var/log/debug.log *.emerg * local7.* :ommysql:127.0.0.1,Syslog,syslog,lisboa08 !startslip *.* /var/log/slip.log !ppp *.* /var/log/ppp.log The messages in the messages file dont include a facility or criticality value so makes it tough to work out. For example: 2009-10-28T12:21:36.782834+00:00 ourmailserver tpop3d[90983]: connections_post_select: client [3]dave at somedomain.co.uk(1.8.1.1): disconnected; 93/602717 bytes read/written Can anyone give me a clue?? thanks, Andy. From rgerhards at hq.adiscon.com Wed Oct 28 13:56:08 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 13:56:08 +0100 Subject: [rsyslog] config prob with mail.info messages References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032E2@GRFEXC.intern.adiscon.com> I have to admit I do not really understand the question, but to record severity and facility, you can follow this guide: http://www.rsyslog.com/doc-rsyslog_recording_pri.html HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Andy Smith > Sent: Wednesday, October 28, 2009 1:53 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] config prob with mail.info messages > > Hi, > > I?m finding rsyslog is logging all routine (I assume mail.info) > mail messages to the main messages file (FreeBSD 7.0). Can someone > suggest why based on the config: > > *.err;kern.warning;auth.notice;mail.crit;local7.none > /dev/console > *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err;local7.no > ne > /var/log/messages > security.* /var/log/security > auth.info;authpriv.info /var/log/auth.log > mail.info /var/log/maillog > lpr.info /var/log/lpd-errs > ftp.info /var/log/xferlog > cron.* /var/log/cron > *.=debug;local7.none > /var/log/debug.log > *.emerg * > local7.* :ommysql:127.0.0.1,Syslog,syslog,lisboa08 > !startslip > *.* /var/log/slip.log > !ppp > *.* /var/log/ppp.log > > The messages in the messages file dont include a facility or > criticality value so makes it tough to work out. For example: > > 2009-10-28T12:21:36.782834+00:00 ourmailserver tpop3d[90983]: > connections_post_select: client [3]dave at somedomain.co.uk(1.8.1.1): > disconnected; 93/602717 bytes read/written > > > Can anyone give me a clue?? > > thanks, Andy. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jbondc at openmv.com Wed Oct 28 14:04:24 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Wed, 28 Oct 2009 09:04:24 -0400 Subject: [rsyslog] rsyslog and snare In-Reply-To: References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> Message-ID: <002301ca57cf$2cece590$86c6b0b0$@com> On Tue Oct 27 04:32 PM, david at lang.hm wrote: > On Tue, 27 Oct 2009, Martin Mielke wrote: > > > Hi, > > > > We recently started to receive logs from a Snare client. > > I applied the configuration changes proposed here: > > http://wiki.rsyslog.com/index.php/Using_Snare_as_a_client_on_Windows > > I just got a chance to read this, and I disagree with it's > recommendation > > the #011 is a tab. > > snare uses it to seperate fields from the windows event log. > > if you just replace it with a space you loose the ability to do things > like look for field 10 because field 8 may contain spaces. > > In addition there are two formats of logs that snare can output to > syslog > > 1. the snare format > 2. 'syslog' format > > the difference seems to be that in snare format it does > > Jan 1 01:01:01 > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > > while in syslog format it does > > Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri > > Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric character will terminate the TAG field and will be assumed to be the starting character of the CONTENT field.) The result would be: TAG: '' MSG: '#011MSWinEventLog#0111#011Security#0114169#011Fri...' At least I think, any thoughts Rainer? The following works for me: // If first character not alpha-numeric, skip tag parsing if( !isalphanum((int)*p2parse) ) bTAGCharDetected = 1; Is this bad for other loggers? From rgerhards at hq.adiscon.com Wed Oct 28 14:12:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 14:12:53 +0100 Subject: [rsyslog] rsyslog and snare References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> <002301ca57cf$2cece590$86c6b0b0$@com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> > > Jan 1 01:01:01 > > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > > > > while in syslog format it does > > > > Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri > > > > First things first: I really don't like to dig into snare, as I do not think it is a really good solution (but how could the designer of http://www.eventreporter.com say otherwise? ;)). so I assume you are talking about this message: mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric > character > will terminate the TAG field and will be assumed to be the starting > character of the CONTENT field.) Another thing to put straight: RFC3164 is NOT a standard, it is an informational document that describes things seen on the wire. And it does not even describe all *popular* cases. If it all, it gains some standard function via RFC3195, but this is questionable. Just for the records ;) So the key point here is that we have nothing to "respect" or "obey to", but rather something that describes things seen in practice. For any decent syslog receiver, this means it must try to work equally well with everything that comes in via legacy syslog. But, granted, RFC3164 is useful if we have nothing else to look at. So let's do that. We'll immediately see that snare is broken, because no control characters are allowed in the hostname. So rsyslog does the right thing and escapes these characters. So everthing belongs to the hostname and thus you have problems with tag. Damn... I didn't want to say somethign about snare. Now it has happened again. My conclusion: fix snare or use something that works ;) Rainer > > The result would be: > TAG: '' > MSG: '#011MSWinEventLog#0111#011Security#0114169#011Fri...' > > At least I think, any thoughts Rainer? > > The following works for me: > // If first character not alpha-numeric, skip tag parsing > if( !isalphanum((int)*p2parse) ) > bTAGCharDetected = 1; > > Is this bad for other loggers? > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From a.smith at ukgrid.net Wed Oct 28 14:17:41 2009 From: a.smith at ukgrid.net (Andy Smith) Date: Wed, 28 Oct 2009 13:17:41 +0000 Subject: [rsyslog] config prob with mail.info messages In-Reply-To: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> Message-ID: <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> Hi Rainer, ok, the question is because based on my config I do not expect or want routine mail messages logged to my messages file, it is over run by messages from courier IMAP and TPOP3 daemons. When using the built in FreeBSD syslog these types of messages are written to the maillog log file, and I dont understand why the same isnt true of rsyslog given the config file I am using, thanks Andy. Quoting Rainer Gerhards: > I have to admit I do not really understand the question From rgerhards at hq.adiscon.com Wed Oct 28 14:43:43 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 14:43:43 +0100 Subject: [rsyslog] config prob with mail.info messages References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032E5@GRFEXC.intern.adiscon.com> Ah, I see. It would probably be useful to see the priority values, as specified in the other link I posted. Maybe the problem is rooted there. Ultimately, a debug log could bring the information how selectors are processed. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Andy Smith > Sent: Wednesday, October 28, 2009 2:18 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] config prob with mail.info messages > > Hi Rainer, > > ok, the question is because based on my config I do not expect or > want routine mail messages logged to my messages file, it is over run > by messages from courier IMAP and TPOP3 daemons. When using the built > in FreeBSD syslog these types of messages are written to the maillog > log file, and I dont understand why the same isnt true of rsyslog > given the config file I am using, > > thanks Andy. > > Quoting Rainer Gerhards: > > > I have to admit I do not really understand the question > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Oct 28 15:50:29 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 07:50:29 -0700 (PDT) Subject: [rsyslog] Property-Based Filters, not working? In-Reply-To: <001c01ca57c9$c3870810$4a951830$@com> References: <002101ca573d$5f417f70$1dc47e50$@com> <001b01ca57c2$593248a0$0b96d9e0$@com> <9B6E2A8877C38245BFB15CC491A11DA71032DD@GRFEXC.intern.adiscon.com> <001c01ca57c9$c3870810$4a951830$@com> Message-ID: On Wed, 28 Oct 2009, Jonathan Bond-Caron wrote: > On Wed Oct 28 07:35 AM, Rainer Gerhards wrote: >>> Thanks a lot, that works well. >>> >>> The docs are bit misleading: >>> http://www.rsyslog.com/doc-rsyslog_conf_filter.html >>> >>> It could show something like this: >>> :msg, contains, "error" /var/log/error.log >>> :msg, !contains, "error" ~ >>> >>> All messages that contain the word "error" are logged to >>> /var/log/error.log All messages that do not contain the word "error" >>> are thrown away. >> >> The doc in general could be much improved (any volunteers?), but here >> I think it is right. The link you quote defines what filters are, but >> what you post is not only a filter but a full selector line, >> consisting from a filter and the associated action. >> >> I agree that it would be useful to have more scenario-based cases >> which contain all the pieces put together (again, any volunteers?). > > Noted ;) > > I'll prepare a patch for 4.4.2 this week, will also fix an issue with the > syslog 'tag' parsing. It should terminate when it sees a tab or escape > control character. > > Makes it a little more RFC3164 compliant. it should also terminate parsing a hostname when it hits a tab (the RFC says it should be a space, but I've got one case (snare) that sometimes uses a tab. David Lang From david at lang.hm Wed Oct 28 15:53:32 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 07:53:32 -0700 (PDT) Subject: [rsyslog] rsyslog and snare In-Reply-To: <002301ca57cf$2cece590$86c6b0b0$@com> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> <002301ca57cf$2cece590$86c6b0b0$@com> Message-ID: On Wed, 28 Oct 2009, Jonathan Bond-Caron wrote: > On Tue Oct 27 04:32 PM, david at lang.hm wrote: >> On Tue, 27 Oct 2009, Martin Mielke wrote: >> >>> Hi, >>> >>> We recently started to receive logs from a Snare client. >>> I applied the configuration changes proposed here: >>> http://wiki.rsyslog.com/index.php/Using_Snare_as_a_client_on_Windows >> >> I just got a chance to read this, and I disagree with it's >> recommendation >> >> the #011 is a tab. >> >> snare uses it to seperate fields from the windows event log. >> >> if you just replace it with a space you loose the ability to do things >> like look for field 10 because field 8 may contain spaces. >> >> In addition there are two formats of logs that snare can output to >> syslog >> >> 1. the snare format >> 2. 'syslog' format >> >> the difference seems to be that in snare format it does >> >> Jan 1 01:01:01 >> mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri >> >> while in syslog format it does >> >> Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri >> >> > > Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric character > will terminate the TAG field and will be assumed to be the starting > character of the CONTENT field.) > > The result would be: > TAG: '' > MSG: '#011MSWinEventLog#0111#011Security#0114169#011Fri...' > > At least I think, any thoughts Rainer? > > The following works for me: > // If first character not alpha-numeric, skip tag parsing > if( !isalphanum((int)*p2parse) ) > bTAGCharDetected = 1; > > Is this bad for other loggers? if it stops on the tab it should end up hostname =vmail.abc.com then it hits a tab, so stops putting things as part of the hostname tag =vMSWinEventLog then it hits a tab, so stops putting things as part of the tag message = 1#011Security#0114169#011Fri... David Lang From david at lang.hm Wed Oct 28 15:56:48 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 07:56:48 -0700 (PDT) Subject: [rsyslog] rsyslog and snare In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> <002301ca57cf$2cece590$86c6b0b0$@com> <9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 28 Oct 2009, Rainer Gerhards wrote: >>> Jan 1 01:01:01 >>> mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri >>> >>> while in syslog format it does >>> >>> Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri >>> >>> > > First things first: I really don't like to dig into snare, as I do not think > it is a really good solution (but how could the designer of > http://www.eventreporter.com say otherwise? ;)). > > so I assume you are talking about this message: > > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > >> Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric >> character >> will terminate the TAG field and will be assumed to be the starting >> character of the CONTENT field.) > > Another thing to put straight: RFC3164 is NOT a standard, it is an > informational document that describes things seen on the wire. And it does > not even describe all *popular* cases. If it all, it gains some standard > function via RFC3195, but this is questionable. Just for the records ;) > > So the key point here is that we have nothing to "respect" or "obey to", but > rather something that describes things seen in practice. For any decent > syslog receiver, this means it must try to work equally well with everything > that comes in via legacy syslog. But, granted, RFC3164 is useful if we have > nothing else to look at. > > So let's do that. We'll immediately see that snare is broken, because no > control characters are allowed in the hostname. So rsyslog does the right > thing and escapes these characters. So everthing belongs to the hostname and > thus you have problems with tag. two other valid behaviors 1. it has a control character in the first text field, so that cannot be a hostname or a tag, so it must be part of the message (after doing the escaping) 2. treat running into a tab like running into a space I think #1 would be better than what we do today, but #2 would be the best for users. David Lang > Damn... I didn't want to say somethign about snare. Now it has happened > again. My conclusion: fix snare or use something that works ;) > > Rainer > >> >> The result would be: >> TAG: '' >> MSG: '#011MSWinEventLog#0111#011Security#0114169#011Fri...' >> >> At least I think, any thoughts Rainer? >> >> The following works for me: >> // If first character not alpha-numeric, skip tag parsing >> if( !isalphanum((int)*p2parse) ) >> bTAGCharDetected = 1; >> >> Is this bad for other loggers? >> >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 28 16:00:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 16:00:12 +0100 Subject: [rsyslog] rsyslog and snare References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local><002301ca57cf$2cece590$86c6b0b0$@com><9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032E8@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, October 28, 2009 3:57 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog and snare > > On Wed, 28 Oct 2009, Rainer Gerhards wrote: > > >>> Jan 1 01:01:01 > >>> mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > >>> > >>> while in syslog format it does > >>> > >>> Jan 1 01:01:01 mail.abc.com > MSWinEventLog#011Security#0114169#011Fri > >>> > >>> > > > > First things first: I really don't like to dig into snare, as I do > not think > > it is a really good solution (but how could the designer of > > http://www.eventreporter.com say otherwise? ;)). > > > > so I assume you are talking about this message: > > > > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > > > >> Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric > >> character > >> will terminate the TAG field and will be assumed to be the starting > >> character of the CONTENT field.) > > > > Another thing to put straight: RFC3164 is NOT a standard, it is an > > informational document that describes things seen on the wire. And it > does > > not even describe all *popular* cases. If it all, it gains some > standard > > function via RFC3195, but this is questionable. Just for the records > ;) > > > > So the key point here is that we have nothing to "respect" or "obey > to", but > > rather something that describes things seen in practice. For any > decent > > syslog receiver, this means it must try to work equally well with > everything > > that comes in via legacy syslog. But, granted, RFC3164 is useful if > we have > > nothing else to look at. > > > > So let's do that. We'll immediately see that snare is broken, because > no > > control characters are allowed in the hostname. So rsyslog does the > right > > thing and escapes these characters. So everthing belongs to the > hostname and > > thus you have problems with tag. > > two other valid behaviors > > 1. it has a control character in the first text field, so that cannot > be a > hostname or a tag, so it must be part of the message (after doing the > escaping) > > 2. treat running into a tab like running into a space > > I think #1 would be better than what we do today, but #2 would be the > best > for users. The problem with all that is that it breaks message sanitation. When a message hits rsyslog, it first is sanitized (which is important for security reasons, e.g. to prevent NUL characters to make parts of the messages unreadable and, later, to prevent the myriad of Unicode-based Vulnerabilites). So when it hits the parser, the is no such thing like a HT present anymore. What we could do, however, is add an option that tells the sanitizer to replace HT by SP in all cases. Rainer From david at lang.hm Wed Oct 28 16:08:22 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 08:08:22 -0700 (PDT) Subject: [rsyslog] rsyslog and snare In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032E8@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local><002301ca57cf$2cece590$86c6b0b0$@com><9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71032E8@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 28 Oct 2009, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >> two other valid behaviors >> >> 1. it has a control character in the first text field, so that cannot >> be a >> hostname or a tag, so it must be part of the message (after doing the >> escaping) >> >> 2. treat running into a tab like running into a space >> >> I think #1 would be better than what we do today, but #2 would be the >> best >> for users. > > The problem with all that is that it breaks message sanitation. When a > message hits rsyslog, it first is sanitized (which is important for security > reasons, e.g. to prevent NUL characters to make parts of the messages > unreadable and, later, to prevent the myriad of Unicode-based > Vulnerabilites). > > So when it hits the parser, the is no such thing like a HT present anymore. > What we could do, however, is add an option that tells the sanitizer to > replace HT by SP in all cases. this is not as good because the tabs are useful in the message itself, it acts as a field seperator for the different fields from the windows logs, and the fields themselves can contain spaces, so if you replace them with tabs you no longer have any way to identify individual fields. the parser could look for # (which is not a valid character in a hostname, and I don't think it's valid in a tag), then if it's #011 treat is as a space for seperating the hoatname/tag/message and if it's anything else, make that the start of the message. David Lang From david at lang.hm Wed Oct 28 16:08:55 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 08:08:55 -0700 (PDT) Subject: [rsyslog] control character escaping change? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032D8@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71032D8@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 27 Oct 2009, Rainer Gerhards wrote: > This smells like a mistake. At least I don't remember changing that, but I'll > look at the changelog tomorrow... did you get a chance to look at this? David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Tuesday, October 27, 2009 9:06 PM >> To: rsyslog-users >> Subject: [rsyslog] control character escaping change? >> >> it appears that the 5.3 development series recently changed >> the default >> for escaping control characters from # to \ so a tab changed >> from #011 to >> \011 >> >> was this a deliberate change going forward, or a mistake? >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From epiphani at gmail.com Wed Oct 28 16:14:03 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Wed, 28 Oct 2009 11:14:03 -0400 Subject: [rsyslog] Queue sizing Message-ID: Greetings, I'm trying to get a good handle on queue sizing and memory usage. In a higher-traffic environment, I have a dedicated rsyslog machine with a large amount of ram. I want to try and find the right balance between large queues for burst traffic, but I also want to make sure that the queue sizes are small enough that we don't run out of memory and crash. I'm using 4.5.5 currently. I'm currently using disk-backed memory queues, configured in fixedarray, like so: $MainMsgQueueSize 2000000 $MainMsgQueueMaxFileSize 50g $MainMsgQueueSaveOnShutdown on $MainMsgQueueHighWaterMark 15000000 $MainMsgQueueLowWaterMark 10000000 $MainMsgQueueType FixedArray $MainMsgQueueWorkerThreads 2 $ActionQueueSize 200000 $ActionQueueMaxFileSize 50g $ActionQueueSaveOnShutdown on $ActionQueueHighWaterMark 1500000 $ActionQueueLowWaterMark 1000000 $ActionQueueType FixedArray $ActionQueueWorkerThreads 1 I usually have upwards of a dozen action queue definitions, and my omfile outputs are based on hostname of the incoming connection. Is there some way I can calculate maximum memory use by the queuing system, or force the system to a set memory maximum where I -know- that it will simply start pushing back on the incoming connection instead of adding more data to the system? Thoughts? -Aaron From rgerhards at hq.adiscon.com Wed Oct 28 16:25:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 16:25:27 +0100 Subject: [rsyslog] control character escaping change? References: <9B6E2A8877C38245BFB15CC491A11DA71032D8@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032EB@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, October 28, 2009 4:09 PM > To: rsyslog-users > Subject: Re: [rsyslog] control character escaping change? > > On Tue, 27 Oct 2009, Rainer Gerhards wrote: > > > This smells like a mistake. At least I don't remember changing that, > but I'll > > look at the changelog tomorrow... > > did you get a chance to look at this? yes, sorry, this is a mistake. I'll fix it soon, but today (and maybe up to the end of the week), I am working on a journal article whom's deadline *is* end of the week ;) Rainer From rgerhards at hq.adiscon.com Wed Oct 28 16:28:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 16:28:53 +0100 Subject: [rsyslog] rsyslog and snare References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local><002301ca57cf$2cece590$86c6b0b0$@com><9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71032E8@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032EC@GRFEXC.intern.adiscon.com> > > So when it hits the parser, the is no such thing like a HT present > anymore. > > What we could do, however, is add an option that tells the sanitizer > to > > replace HT by SP in all cases. > > this is not as good because the tabs are useful in the message itself, > it > acts as a field seperator for the different fields from the windows > logs, > and the fields themselves can contain spaces, so if you replace them > with > tabs you no longer have any way to identify individual fields. > > the parser could look for # (which is not a valid character in a > hostname, > and I don't think it's valid in a tag), then if it's #011 treat is as a > space for seperating the hoatname/tag/message and if it's anything > else, > make that the start of the message. Yeah, but (not only because it is snare ;)) wouldn't it be the right thing to do to fix the broken sender? Of course, I can add another exception AND "#" may validly be inside a tag. I am all for trying to guess right, but keep in mind that all this also has a performance toll. One of my plans is to add a custom parser interface. With it, custom parsers could be added and they could take care of all the anomalies of the real world (and there *are* many!). But that is up for another increment, after the queue work. I don't like to change the parsers that have proven to work well in practice just because one application misbehaves... Rainer From david at lang.hm Wed Oct 28 17:15:17 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 09:15:17 -0700 (PDT) Subject: [rsyslog] rsyslog and snare In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032EC@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local><002301ca57cf$2cece590$86c6b0b0$@com><9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71032E8@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71032EC@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 28 Oct 2009, Rainer Gerhards wrote: >>> So when it hits the parser, the is no such thing like a HT present >> anymore. >>> What we could do, however, is add an option that tells the sanitizer >> to >>> replace HT by SP in all cases. >> >> this is not as good because the tabs are useful in the message itself, >> it >> acts as a field seperator for the different fields from the windows >> logs, >> and the fields themselves can contain spaces, so if you replace them >> with >> tabs you no longer have any way to identify individual fields. >> >> the parser could look for # (which is not a valid character in a >> hostname, >> and I don't think it's valid in a tag), then if it's #011 treat is as a >> space for seperating the hoatname/tag/message and if it's anything >> else, >> make that the start of the message. > > Yeah, but (not only because it is snare ;)) wouldn't it be the right thing to > do to fix the broken sender? yes it would, but doing so is much harder than it should be :-( David Lang > Of course, I can add another exception AND "#" > may validly be inside a tag. I am all for trying to guess right, but keep in > mind that all this also has a performance toll. > > One of my plans is to add a custom parser interface. With it, custom parsers > could be added and they could take care of all the anomalies of the real > world (and there *are* many!). But that is up for another increment, after > the queue work. > > I don't like to change the parsers that have proven to work well in practice > just because one application misbehaves... > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From a.smith at ukgrid.net Wed Oct 28 19:30:59 2009 From: a.smith at ukgrid.net (Andy Smith) Date: Wed, 28 Oct 2009 18:30:59 +0000 Subject: [rsyslog] config prob with mail.info messages In-Reply-To: <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> Message-ID: <20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> Hi Rainer, ok thanks, I turned on the custom template as described in the link you posted and could immediately see that the events were being logged as mail.notice, which strangely isnt specifically configured in the FreeBSD built in syslog conf but these are none the less logged to maillog instead of messages. Anyway, thanks to knowing this I have been easily able to add an extra line in the config to direct these to the maillog as I want :) thanks again! Andy. Quoting Rainer Gerhards: > Ah, I see. It would probably be useful to see the priority values, as > specified in the other link I posted. From anichols at trumped.org Wed Oct 28 20:00:22 2009 From: anichols at trumped.org (Aaron Nichols) Date: Wed, 28 Oct 2009 13:00:22 -0600 Subject: [rsyslog] config prob with mail.info messages In-Reply-To: <20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> <20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> Message-ID: On Wed, Oct 28, 2009 at 12:30 PM, Andy Smith wrote: > ? ok thanks, I turned on the custom template as described in the link > you posted and could immediately see that the events were being logged > as mail.notice, which strangely isnt specifically configured in the > FreeBSD built in syslog conf but these are none the less logged to > maillog instead of messages. Your original configuration contained this: *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err;local7.none /var/log/messages If you have messages being delivered to mail.notice they will match the "*.notice" selector at the beginning of that line. It's easy enough to filter them out as you've found, but I figured I'd point out that the configuration did actually specify that these should be sent to /var/log/messages. Aaron From rgerhards at hq.adiscon.com Wed Oct 28 21:02:45 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 21:02:45 +0100 Subject: [rsyslog] config prob with mail.info messages References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net><20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net><20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032EF@GRFEXC.intern.adiscon.com> Aaron, Good catch - I overlooked that :) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Wednesday, October 28, 2009 8:00 PM > To: rsyslog-users > Subject: Re: [rsyslog] config prob with mail.info messages > > On Wed, Oct 28, 2009 at 12:30 PM, Andy Smith > wrote: > > ? ok thanks, I turned on the custom template as described > in the link > > you posted and could immediately see that the events were > being logged > > as mail.notice, which strangely isnt specifically configured in the > > FreeBSD built in syslog conf but these are none the less logged to > > maillog instead of messages. > > Your original configuration contained this: > *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err; > local7.none > /var/log/messages > > If you have messages being delivered to mail.notice they will match > the "*.notice" selector at the beginning of that line. It's easy > enough to filter them out as you've found, but I figured I'd point out > that the configuration did actually specify that these should be sent > to /var/log/messages. > > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From a.smith at ukgrid.net Wed Oct 28 21:51:44 2009 From: a.smith at ukgrid.net (Andy Smith) Date: Wed, 28 Oct 2009 20:51:44 +0000 Subject: [rsyslog] config prob with mail.info messages In-Reply-To: <20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> <20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> Message-ID: <20091028205144.17377b2djm644r8c@horde.ukgrid.net> Ah yep all is clear now, thanks Aaron, well spotted as Rainer said :P From martinmie at PartyGaming.com Sat Oct 31 02:28:22 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Sat, 31 Oct 2009 02:28:22 +0100 Subject: [rsyslog] rsyslog and snare In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local><002301ca57cf$2cece590$86c6b0b0$@com> <9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> Message-ID: <0B1B9163138571439EAF804646F3F6AA0892F6F9@GIBSVWIN004X.partygaming.local> Semi off-topic: release EventReporter under the same license as rsyslog and you'll win a lot of happy friends ;-) Cheers > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: 28 October 2009 14:13 > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog and snare > > > > Jan 1 01:01:01 > > > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > > > > > > while in syslog format it does > > > > > > Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri > > > > > > > > First things first: I really don't like to dig into snare, as I do not > think > it is a really good solution (but how could the designer of > http://www.eventreporter.com say otherwise? ;)). > > so I assume you are talking about this message: > > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > > > Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric > > character > > will terminate the TAG field and will be assumed to be the starting > > character of the CONTENT field.) > > Another thing to put straight: RFC3164 is NOT a standard, it is an > informational document that describes things seen on the wire. And it does > not even describe all *popular* cases. If it all, it gains some standard > function via RFC3195, but this is questionable. Just for the records ;) > > So the key point here is that we have nothing to "respect" or "obey to", > but > rather something that describes things seen in practice. For any decent > syslog receiver, this means it must try to work equally well with > everything > that comes in via legacy syslog. But, granted, RFC3164 is useful if we > have > nothing else to look at. > > So let's do that. We'll immediately see that snare is broken, because no > control characters are allowed in the hostname. So rsyslog does the right > thing and escapes these characters. So everthing belongs to the hostname > and > thus you have problems with tag. > > Damn... I didn't want to say somethign about snare. Now it has happened > again. My conclusion: fix snare or use something that works ;) > > Rainer > > > > > The result would be: > > TAG: '' > > MSG: '#011MSWinEventLog#0111#011Security#0114169#011Fri...' > > > > At least I think, any thoughts Rainer? > > > > The following works for me: > > // If first character not alpha-numeric, skip tag parsing > > if( !isalphanum((int)*p2parse) ) > > bTAGCharDetected = 1; > > > > Is this bad for other loggers? > > > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > /rsyslog > http://www.rsyslog.com This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From igalvarez at gmail.com Thu Oct 1 05:42:56 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Wed, 30 Sep 2009 22:42:56 -0500 Subject: [rsyslog] sending logs outside /etc/rsyslog.conf to remote server Message-ID: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> I've some daemons with their logs not define in /etc/rsyslog.conf and files outside /var/log folder. I have a central rsyslog receiving logs from severral servers. My question is: How can setup rsyslog on clients to send these logs outside /etc/rsyslog.conf to the central rsyslog server? Is this possible? -- Regards; Israel Garcia From david at lang.hm Thu Oct 1 06:17:19 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 21:17:19 -0700 (PDT) Subject: [rsyslog] sending logs outside /etc/rsyslog.conf to remote server In-Reply-To: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> References: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> Message-ID: On Wed, 30 Sep 2009, Israel Garcia wrote: > I've some daemons with their logs not define in /etc/rsyslog.conf and > files outside /var/log folder. I have a central rsyslog receiving logs > from severral servers. > > My question is: > > How can setup rsyslog on clients to send these logs outside > /etc/rsyslog.conf to the central rsyslog server? Is this possible? the first problem is to get the messages to rsyslog. if the applications don't have an option to log to syslog you can use the imfile input module in rsyslog to read lines from the file and process them. David Lang From rgerhards at hq.adiscon.com Thu Oct 1 11:01:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 1 Oct 2009 11:01:17 +0200 Subject: [rsyslog] DNS cache and expiration References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103110@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710311A@GRFEXC.intern.adiscon.com> I have now reviewed the discussion in depth. Many thanks for all the good thoughts. I will now first begin to move the name resolution calls to the "backend" threads (away from the inputs). However, I will create an option that the name resolution may be done before enqueing the message, which may be used by some folks for whatever reason (plus, it is trivially to do). I also think I will leave in the capability that inputs do the name resolution themselfs, at least if they prefer. For example, imuxsock always has the local host as input, so it makes limited sense to do name resolution calls for messages received from that input. Similarly, imtcp uses the name that was (once) obtained when the session was created (btw, this is also some expiry-less "name resolution", but I think here it is even harder to argue against it ;)). When I am done with this, I have a nice central place where the name resolution is done. In that place, I will then build a central name cache, at first with only two options - cache disabled - cache without expiration (requires shutdown) After that, I'll probably look into HUP plus probably provide a maximum TTL to rsyslog that says e.g. "check entry after n seconds". Even if n is low (maybe 10 seconds), the extra effort for doing these lookups is minimal: If the system has low throughput, the lookup cost is irrelevant in any case. If it is a high performance system, with more than e.g. 10,000 msgs/s, then n=10 means we do one lookup for at most 1/100,000th message, which doesn't matter either. Most importantly, I do not like the idea to use a third-party resolver library, as this creates another set of dependencies, and this for a core module. Booting rsyslog would get even more complicated. So I think the "expire ever n seconds" approach is a very practical one. You will see move these changes gradually into the master branch. One word on the performance: an avl-tree based internal cache can resolve names *very* fast. It has a depth of almost log n where n being the number of names cached. Building and maintaining the tree is almost for free, because the structure is rather static. Expiration, as I intend to implement it, is also a very simple operation, the node does not need to be removed and re-inserted but rather I need to update the resolution info by means of a new lookup (what costs the majority of time). So for a typical search, I would assume that the name can be resolved with less than 500 machine instructions (what I consider a conservative choice). The way I intend to implement it probably does not include any mutex calls, so there is no context switch involved. I consider the cost of a single context switch to be much higher than such an avl-tree lookup. Now consider any external solution. You need to do at least four context switches (rsyslog -> OS -> resolver -> OS -> rsyslog), probably more. Also, at least the OS scheduler is involved to activate the threads that are synchronized, plus probably a bit more of OS logic. Then, the external resolver can at best perform at the same rate rsyslog itself can (using a similar algorithm), but probably slower because it must be more generic. Looking at this picture, I think, clarifies why an (proper) internal implementation is so much more efficient than any external solution. Also note that I assume that in almost all cases, due to the structure of syslog traffic, no real lookup needs to be done, everything sits already in the resolver caches. I hope this is useful information. If someone has any additional comments, I would love to hear them. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Wednesday, September 30, 2009 9:57 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Wednesday, September 30, 2009 9:53 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] DNS cache and expiration > > > > On Wed, 30 Sep 2009, Rainer Gerhards wrote: > > > > > Thanks for the good discussion. I am lacking somewhat > > behind, but will review > > > it in depth tomorrow morning. > > > > > > I just wanted to stress the point that an external cache > > does not really > > > help, much for the reason David mentioned: if you process > > messages at very > > > high data rates, the context switch overhead involved with > > any external > > > solution is extremely costly. Also, in the usual cases, I > > may do several > > > million queries within a few seconds for just a handful of > > hosts. With an > > > internal cache, the overhead in doing so is very minimal. > > With an external > > > solution, the overhead in calling the external cache causes a lot > of > > > performance degredation, what in the case of UDP also > > implies (heavy!) > > > message loss. > > > > the message loss problem with UDP will not be solved completely by an > > internal cache. when the source is not in the cache and you > > have to go out > > to find it the lookup can take several seconds. > > > > moving the lookup out of the input module and into the output > > module would > > address this, anything else would leave you with losses as > > the cache gets > > populated. > > That's right and that's one reason why I intend to move this > (optionally) > over to the "backend" processing. However, even that does not > completely > solve the message loss problem, as we, in extreme cases, may loose > messages > when the queue is full - and for a myriad of other reasons, like > routers > discarding frames and such. Of course, you know that, but I'd like to > mention > if for those folks that at some time find our conversation via Google > ;) > > Rainer > > > > David Lang > > > > > Rainer > > > > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com > > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > david at lang.hm > > >> Sent: Wednesday, September 30, 2009 8:54 PM > > >> To: rsyslog-users > > >> Subject: Re: [rsyslog] DNS cache and expiration > > >> > > >> On Wed, 30 Sep 2009, Kenneth Marshall wrote: > > >> > > >>>>> mechanism to map IP addresses to names for the purposes > > of logging > > >>>>> error messages. As such, pretty much the only piece > > that needs to > > >>>>> be tracked within rsyslog is the TTL for the entry and the ip - > > > > >>>>> name mapping. A thread would be responsible for expiring > entries > > >>>>> from the cache (or refreshing the timeout) after validating the > > >>>>> correctness of the mapping. I think the DNS lookups should be > > >>>>> handled by a good resolver like pdns-recursor, djbdns,... The > > >>>>> goal here is to allow names in the log entries and not just IP > > >>>>> addresses and in a very high performance logging environment. > > >>>> > > >>>> the trouble is that doing _proper_ TTL expiration isn't as > > >> simple as it > > >>>> sounds. > > >>>> > > >>>> and if you are willing to back away from 'proper' > > >> expiration to something > > >>>> that will work in practice, why not go much further (as I > > >> have detailed in > > >>>> the other messages) > > >>>> > > >>>> David Lang > > >>> > > >>> I agree. I only mention TTL values as a reasonable > > upperbound of the > > >>> refresh check. The advantage is to remove the large pause > > in logging > > >>> due to a DNS refresh after a HUP to rsyslog, if that were the > only > > >>> method to flush/refresh. Like you mention, the IPs/names > > of systems > > >>> being logged change rarely so we should tune this for speed and > > >>> not worry about expiration correctness. I do not agree so other > > >>> statements that an external cache will perform as well as > > >> an internal > > >>> cache. Too many software products that I work with have > > >> needed exactly > > >>> that functionality to support very high levels of performance. > > >> > > >> actually, you could have the cache be configurable in three modes > > >> > > >> 1. no caching > > >> > > >> 2. blank the cache on HUP > > >> > > >> 3. never blank the cache (i.e. require a full restart to clear it) > > >> > > >> David Lang > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From igalvarez at gmail.com Thu Oct 1 17:09:04 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Thu, 1 Oct 2009 10:09:04 -0500 Subject: [rsyslog] sending logs outside /etc/rsyslog.conf to remote server In-Reply-To: References: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> Message-ID: <194a2c240910010809k99cd7e7o46d40e428923912f@mail.gmail.com> On 9/30/09, david at lang.hm wrote: > On Wed, 30 Sep 2009, Israel Garcia wrote: > >> I've some daemons with their logs not define in /etc/rsyslog.conf and >> files outside /var/log folder. I have a central rsyslog receiving logs >> from severral servers. >> >> My question is: >> >> How can setup rsyslog on clients to send these logs outside >> /etc/rsyslog.conf to the central rsyslog server? Is this possible? > > the first problem is to get the messages to rsyslog. > > if the applications don't have an option to log to syslog you can use the > imfile input module in rsyslog to read lines from the file and process > them. Hi David, Has anybody on the list setup apache, proftpd, momgrel or any other daemon logs to forward to a centralized syslog server? I'd appreciate your experiences. regards, Israel. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From rgerhards at hq.adiscon.com Thu Oct 1 17:25:24 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 1 Oct 2009 17:25:24 +0200 Subject: [rsyslog] sending logs outside /etc/rsyslog.conf to remoteserver References: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> <194a2c240910010809k99cd7e7o46d40e428923912f@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710312D@GRFEXC.intern.adiscon.com> no experience on my side, but I wanted to make you aware of the wiki, which (I think) contains a number of samples: http://wiki.rsyslog.com Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Israel Garcia > Sent: Thursday, October 01, 2009 5:09 PM > To: rsyslog-users > Subject: Re: [rsyslog] sending logs outside /etc/rsyslog.conf to > remoteserver > > On 9/30/09, david at lang.hm wrote: > > On Wed, 30 Sep 2009, Israel Garcia wrote: > > > >> I've some daemons with their logs not define in /etc/rsyslog.conf > and > >> files outside /var/log folder. I have a central rsyslog receiving > logs > >> from severral servers. > >> > >> My question is: > >> > >> How can setup rsyslog on clients to send these logs outside > >> /etc/rsyslog.conf to the central rsyslog server? Is this possible? > > > > the first problem is to get the messages to rsyslog. > > > > if the applications don't have an option to log to syslog you can use > the > > imfile input module in rsyslog to read lines from the file and > process > > them. > > Hi David, > > Has anybody on the list setup apache, proftpd, momgrel or any other > daemon logs to forward to a centralized syslog server? I'd appreciate > your experiences. > > regards, > Israel. > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > -- > Regards; > Israel Garcia > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Thu Oct 1 19:06:56 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 1 Oct 2009 10:06:56 -0700 (PDT) Subject: [rsyslog] sending logs outside /etc/rsyslog.conf to remote server In-Reply-To: <194a2c240910010809k99cd7e7o46d40e428923912f@mail.gmail.com> References: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> <194a2c240910010809k99cd7e7o46d40e428923912f@mail.gmail.com> Message-ID: On Thu, 1 Oct 2009, Israel Garcia wrote: > On 9/30/09, david at lang.hm wrote: >> On Wed, 30 Sep 2009, Israel Garcia wrote: >> >>> I've some daemons with their logs not define in /etc/rsyslog.conf and >>> files outside /var/log folder. I have a central rsyslog receiving logs >>> from severral servers. >>> >>> My question is: >>> >>> How can setup rsyslog on clients to send these logs outside >>> /etc/rsyslog.conf to the central rsyslog server? Is this possible? >> >> the first problem is to get the messages to rsyslog. >> >> if the applications don't have an option to log to syslog you can use the >> imfile input module in rsyslog to read lines from the file and process >> them. > > Hi David, > > Has anybody on the list setup apache, proftpd, momgrel or any other > daemon logs to forward to a centralized syslog server? I'd appreciate > your experiences. I haven't done it yet, but this page makes it seem pretty easy to setup apache http://www.oreillynet.com/pub/a/sysadmin/2006/10/12/httpd-syslog.html I've had proftpd send things to syslog in the past (IIRC it was a trivial config option) no expriance with mongrel David Lang From david at lang.hm Thu Oct 1 21:26:23 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 1 Oct 2009 12:26:23 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710311A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103110@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710311A@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 1 Oct 2009, Rainer Gerhards wrote: > I have now reviewed the discussion in depth. Many thanks for all the good > thoughts. > > I will now first begin to move the name resolution calls to the "backend" > threads (away from the inputs). However, I will create an option that the > name resolution may be done before enqueing the message, which may be used by > some folks for whatever reason (plus, it is trivially to do). I also think I > will leave in the capability that inputs do the name resolution themselfs, at > least if they prefer. For example, imuxsock always has the local host as > input, so it makes limited sense to do name resolution calls for messages > received from that input. Similarly, imtcp uses the name that was (once) > obtained when the session was created (btw, this is also some expiry-less > "name resolution", but I think here it is even harder to argue against it > ;)). on note on this, if you can delay the resolution to the point where something actually accesses the property that uses the name you may be able to skip doing name resolution entirely in many cases. David Lang > When I am done with this, I have a nice central place where the name > resolution is done. In that place, I will then build a central name cache, at > first with only two options > > - cache disabled > - cache without expiration (requires shutdown) > > After that, I'll probably look into HUP plus probably provide a maximum TTL > to rsyslog that says e.g. "check entry after n seconds". Even if n is low > (maybe 10 seconds), the extra effort for doing these lookups is minimal: > > If the system has low throughput, the lookup cost is irrelevant in any case. > If it is a high performance system, with more than e.g. 10,000 msgs/s, then > n=10 means we do one lookup for at most 1/100,000th message, which doesn't > matter either. > > Most importantly, I do not like the idea to use a third-party resolver > library, as this creates another set of dependencies, and this for a core > module. Booting rsyslog would get even more complicated. So I think the > "expire ever n seconds" approach is a very practical one. > > You will see move these changes gradually into the master branch. > > One word on the performance: an avl-tree based internal cache can resolve > names *very* fast. It has a depth of almost log n where n being the number of > names cached. Building and maintaining the tree is almost for free, because > the structure is rather static. Expiration, as I intend to implement it, is > also a very simple operation, the node does not need to be removed and > re-inserted but rather I need to update the resolution info by means of a new > lookup (what costs the majority of time). So for a typical search, I would > assume that the name can be resolved with less than 500 machine instructions > (what I consider a conservative choice). The way I intend to implement it > probably does not include any mutex calls, so there is no context switch > involved. > > I consider the cost of a single context switch to be much higher than such an > avl-tree lookup. Now consider any external solution. You need to do at least > four context switches (rsyslog -> OS -> resolver -> OS -> rsyslog), probably > more. Also, at least the OS scheduler is involved to activate the threads > that are synchronized, plus probably a bit more of OS logic. Then, the > external resolver can at best perform at the same rate rsyslog itself can > (using a similar algorithm), but probably slower because it must be more > generic. Looking at this picture, I think, clarifies why an (proper) internal > implementation is so much more efficient than any external solution. Also > note that I assume that in almost all cases, due to the structure of syslog > traffic, no real lookup needs to be done, everything sits already in the > resolver caches. > > I hope this is useful information. If someone has any additional comments, I > would love to hear them. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Wednesday, September 30, 2009 9:57 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] DNS cache and expiration >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com >>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Wednesday, September 30, 2009 9:53 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] DNS cache and expiration >>> >>> On Wed, 30 Sep 2009, Rainer Gerhards wrote: >>> >>>> Thanks for the good discussion. I am lacking somewhat >>> behind, but will review >>>> it in depth tomorrow morning. >>>> >>>> I just wanted to stress the point that an external cache >>> does not really >>>> help, much for the reason David mentioned: if you process >>> messages at very >>>> high data rates, the context switch overhead involved with >>> any external >>>> solution is extremely costly. Also, in the usual cases, I >>> may do several >>>> million queries within a few seconds for just a handful of >>> hosts. With an >>>> internal cache, the overhead in doing so is very minimal. >>> With an external >>>> solution, the overhead in calling the external cache causes a lot >> of >>>> performance degredation, what in the case of UDP also >>> implies (heavy!) >>>> message loss. >>> >>> the message loss problem with UDP will not be solved completely by an >>> internal cache. when the source is not in the cache and you >>> have to go out >>> to find it the lookup can take several seconds. >>> >>> moving the lookup out of the input module and into the output >>> module would >>> address this, anything else would leave you with losses as >>> the cache gets >>> populated. >> >> That's right and that's one reason why I intend to move this >> (optionally) >> over to the "backend" processing. However, even that does not >> completely >> solve the message loss problem, as we, in extreme cases, may loose >> messages >> when the queue is full - and for a myriad of other reasons, like >> routers >> discarding frames and such. Of course, you know that, but I'd like to >> mention >> if for those folks that at some time find our conversation via Google >> ;) >> >> Rainer >>> >>> David Lang >>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com >>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >>> david at lang.hm >>>>> Sent: Wednesday, September 30, 2009 8:54 PM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] DNS cache and expiration >>>>> >>>>> On Wed, 30 Sep 2009, Kenneth Marshall wrote: >>>>> >>>>>>>> mechanism to map IP addresses to names for the purposes >>> of logging >>>>>>>> error messages. As such, pretty much the only piece >>> that needs to >>>>>>>> be tracked within rsyslog is the TTL for the entry and the ip - >>> >>>>>>>> name mapping. A thread would be responsible for expiring >> entries >>>>>>>> from the cache (or refreshing the timeout) after validating the >>>>>>>> correctness of the mapping. I think the DNS lookups should be >>>>>>>> handled by a good resolver like pdns-recursor, djbdns,... The >>>>>>>> goal here is to allow names in the log entries and not just IP >>>>>>>> addresses and in a very high performance logging environment. >>>>>>> >>>>>>> the trouble is that doing _proper_ TTL expiration isn't as >>>>> simple as it >>>>>>> sounds. >>>>>>> >>>>>>> and if you are willing to back away from 'proper' >>>>> expiration to something >>>>>>> that will work in practice, why not go much further (as I >>>>> have detailed in >>>>>>> the other messages) >>>>>>> >>>>>>> David Lang >>>>>> >>>>>> I agree. I only mention TTL values as a reasonable >>> upperbound of the >>>>>> refresh check. The advantage is to remove the large pause >>> in logging >>>>>> due to a DNS refresh after a HUP to rsyslog, if that were the >> only >>>>>> method to flush/refresh. Like you mention, the IPs/names >>> of systems >>>>>> being logged change rarely so we should tune this for speed and >>>>>> not worry about expiration correctness. I do not agree so other >>>>>> statements that an external cache will perform as well as >>>>> an internal >>>>>> cache. Too many software products that I work with have >>>>> needed exactly >>>>>> that functionality to support very high levels of performance. >>>>> >>>>> actually, you could have the cache be configurable in three modes >>>>> >>>>> 1. no caching >>>>> >>>>> 2. blank the cache on HUP >>>>> >>>>> 3. never blank the cache (i.e. require a full restart to clear it) >>>>> >>>>> David Lang >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu Oct 1 21:27:40 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 1 Oct 2009 12:27:40 -0700 (PDT) Subject: [rsyslog] fromhost-ip not working with -x Message-ID: I am running all my servers with -x to disable name resolution. I have found that fromhost exists, but fromhost-ip does not this is either a bug in the documentation, or fromhost-ip doesn't get created if no name resolution is done. david Lang From rgerhards at hq.adiscon.com Thu Oct 1 21:45:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 1 Oct 2009 21:45:56 +0200 Subject: [rsyslog] DNS cache and expiration References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103110@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710311A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103133@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, October 01, 2009 9:26 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > On Thu, 1 Oct 2009, Rainer Gerhards wrote: > > > I have now reviewed the discussion in depth. Many thanks > for all the good > > thoughts. > > > > I will now first begin to move the name resolution calls to > the "backend" > > threads (away from the inputs). However, I will create an > option that the > > name resolution may be done before enqueing the message, > which may be used by > > some folks for whatever reason (plus, it is trivially to > do). I also think I > > will leave in the capability that inputs do the name > resolution themselfs, at > > least if they prefer. For example, imuxsock always has the > local host as > > input, so it makes limited sense to do name resolution > calls for messages > > received from that input. Similarly, imtcp uses the name > that was (once) > > obtained when the session was created (btw, this is also > some expiry-less > > "name resolution", but I think here it is even harder to > argue against it > > ;)). > > on note on this, if you can delay the resolution to the point where > something actually accesses the property that uses the name > you may be > able to skip doing name resolution entirely in many cases. This I plan to do, but in the TCP case I think it probably is better to do a name resolution once per *session* even if the name may never be used. Typically, there are many messages in a session and the overhead if the one name lookup does not really matter. At least this is my current thinking. Rainer From david at lang.hm Thu Oct 1 22:07:31 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 1 Oct 2009 13:07:31 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103133@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103110@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710311A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103133@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 1 Oct 2009, Rainer Gerhards wrote: >> -----Original Message----- >> On Thu, 1 Oct 2009, Rainer Gerhards wrote: >> >>> I have now reviewed the discussion in depth. Many thanks >> for all the good >>> thoughts. >>> >>> I will now first begin to move the name resolution calls to >> the "backend" >>> threads (away from the inputs). However, I will create an >> option that the >>> name resolution may be done before enqueing the message, >> which may be used by >>> some folks for whatever reason (plus, it is trivially to >> do). I also think I >>> will leave in the capability that inputs do the name >> resolution themselfs, at >>> least if they prefer. For example, imuxsock always has the >> local host as >>> input, so it makes limited sense to do name resolution >> calls for messages >>> received from that input. Similarly, imtcp uses the name >> that was (once) >>> obtained when the session was created (btw, this is also >> some expiry-less >>> "name resolution", but I think here it is even harder to >> argue against it >>> ;)). >> >> on note on this, if you can delay the resolution to the point where >> something actually accesses the property that uses the name >> you may be >> able to skip doing name resolution entirely in many cases. > > This I plan to do, but in the TCP case I think it probably is better to do a > name resolution once per *session* even if the name may never be used. > Typically, there are many messages in a session and the overhead if the one > name lookup does not really matter. At least this is my current thinking. no problem with filling it out earlier if the info happens to be available (or in the case of thinks like TCP, RELP, etc where it really is static). I'm thinking of things like my central server that receives the aggregated feed from several relays. everything that it does is based on the name in the log, and since the logs come from other rsyslog relay boxes, those boxes will have cleaned things up, so there is never (outside of debugging) a need for that central box to do _any_ lookups, no matter how fast they end up being. David Lang From anichols at trumped.org Fri Oct 2 01:52:22 2009 From: anichols at trumped.org (Aaron Nichols) Date: Thu, 1 Oct 2009 17:52:22 -0600 Subject: [rsyslog] "too many tcp sessions - dropping incoming request" - tried increasing max sessions to no avail Message-ID: Hi all, I am seeing the following in my logs running rsyslog 4.4.1: Oct 1 23:29:24 host rsyslogd-2079: too many tcp sessions - dropping incoming request [try http://www.rsyslog.com/e/2079 ] Oct 1 23:29:29 host rsyslogd-2079: too many tcp sessions - dropping incoming request [try http://www.rsyslog.com/e/2079 ] Oct 1 23:29:38 host rsyslogd-2079: too many tcp sessions - dropping incoming request [try http://www.rsyslog.com/e/2079 ] I went to the URL in the message and the answer makes sense so I added the following to my configuration: $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) I added it before the imtcp module is loaded so the config looks like: # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) $ModLoad imtcp.so # load module $InputTCPServerRun 514 # start up TCP listener at port 514 I am still seeing the "too many tcp sessions" in the logs. When I take a look at netstat (assuming this is the relevant data) I see less than 1000 *connections* - but I'm not sure if each of these is considered a "session" $ netstat -an | grep :514 | wc -l 269 If I look for only ESTABLISHED connections I see less: [anichols at inlwr01p1 inlwr01p1]$ netstat -an | grep :514 | grep ESTABLISHED | wc -l 231 Many of the connections show a SYN_RECV state on the server side - I assume these are cases where we aren't completing the tcp handshake due to too many connections. Are there other factors which may be impacting the max connections or is my configuration incorrect? Thanks, Aaron From anichols at trumped.org Fri Oct 2 04:44:50 2009 From: anichols at trumped.org (Aaron Nichols) Date: Thu, 1 Oct 2009 20:44:50 -0600 Subject: [rsyslog] "too many tcp sessions - dropping incoming request" - tried increasing max sessions to no avail In-Reply-To: References: Message-ID: I found the cause of my problem I believe - the $InputTCPMaxSessions value needs to come after the imtcp module is loaded. This makes sense in hindsight. Of course, the evidence was right in front of me had I looked in the right place: Oct 2 02:34:54 host rsyslogd-3003: invalid or yet-unknown config file command - have you forgotten to load a module? [try http://www.rsyslog.com/e/3003 ] Oct 2 02:34:54 host rsyslogd: the last error occured in /etc/rsyslog.d/rsyslog-server.conf, line 4 I have to thank you guys for making an effort to provide informative errors with URL references. Thanks, Aaron On Thu, Oct 1, 2009 at 5:52 PM, Aaron Nichols wrote: > Hi all, > > I am seeing the following in my logs running rsyslog 4.4.1: > > Oct 1 23:29:24 host rsyslogd-2079: too many tcp sessions - dropping > incoming request [try http://www.rsyslog.com/e/2079 ] > Oct 1 23:29:29 host rsyslogd-2079: too many tcp sessions - dropping > incoming request [try http://www.rsyslog.com/e/2079 ] > Oct 1 23:29:38 host rsyslogd-2079: too many tcp sessions - dropping > incoming request [try http://www.rsyslog.com/e/2079 ] > > I went to the URL in the message and the answer makes sense so I added the > following to my configuration: > > $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) > > I added it before the imtcp module is loaded so the config looks like: > > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) > $ModLoad imtcp.so # load module > $InputTCPServerRun 514 # start up TCP listener at port 514 > > I am still seeing the "too many tcp sessions" in the logs. When I take a > look at netstat (assuming this is the relevant data) I see less than 1000 > *connections* - but I'm not sure if each of these is considered a "session" > > $ netstat -an | grep :514 | wc -l > 269 > > If I look for only ESTABLISHED connections I see less: > > [anichols at inlwr01p1 inlwr01p1]$ netstat -an | grep :514 | grep ESTABLISHED > | wc -l > 231 > > Many of the connections show a SYN_RECV state on the server side - I assume > these are cases where we aren't completing the tcp handshake due to too many > connections. > > Are there other factors which may be impacting the max connections or is my > configuration incorrect? > > Thanks, > Aaron > From tbergfeld at hq.adiscon.com Mon Oct 5 15:13:52 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Mon, 5 Oct 2009 15:13:52 +0200 Subject: [rsyslog] rsyslog 5.3.1 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710315D@GRFEXC.intern.adiscon.com> Hi all, We have just released 5.3.1, a member of the devel branch. This is a bug-fixing release, providing some important fixes from the v4-stable and the current v4-beta, like a bugfix for a potential segfault in stream writer on destruction. This is a recommended update for all users of the devel branch. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article407.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-178.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From salim.engin at gmail.com Wed Oct 7 15:23:21 2009 From: salim.engin at gmail.com (Salim Engin) Date: Wed, 7 Oct 2009 16:23:21 +0300 Subject: [rsyslog] Sender IP or Hostname Message-ID: <43660480910070623s6eeb20b3gf200de18cff7d553@mail.gmail.com> Hello all, I have just installed Debain 5.0.3 with its default syslog server which is rsyslog 3.18.6 . and I configured it to receive external syslog messages from our network devices. What the problem is that, the sender IP or hostname of external devices which send syslog messages is not displayed in the log . Please see the example below; 2009-10-07T16:09:39.407770+03:00 debian dhcpd: DHCPACK on 172.10.10.101 to 00:24:7e:11:eb:49 (stajerstac) via vlan110 --------> "debian" hostname is OK 2009-09-06T14:04:53+03:00 - rCT0[184]: craft(8) : RTRV-HDR:::P6099; -----------------------------------------------------------------------------------> Neither IP nor hostname is printed 2009-09-06T14:04:56+03:00 - clij[258]: isadmin at 10.5.5.23(21) : info configure system sntp ----------------------------------------------------> Neither IP nor hostname is printed. Instead there is dash after the timestamp. I tried many things but was not able to make it work. Can someone help me to fix this ? Thanks Salim Engin From rgerhards at hq.adiscon.com Wed Oct 7 15:33:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 7 Oct 2009 15:33:17 +0200 Subject: [rsyslog] Sender IP or Hostname References: <43660480910070623s6eeb20b3gf200de18cff7d553@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103191@GRFEXC.intern.adiscon.com> Hi Salim, sounds like a malformed message is causing this: http://www.rsyslog.com/doc-syslog_parsing.html Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Salim Engin > Sent: Wednesday, October 07, 2009 3:23 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Sender IP or Hostname > > Hello all, > > I have just installed Debain 5.0.3 with its default syslog server which > is > rsyslog 3.18.6 . and I configured it to receive external syslog > messages > from our network devices. > > What the problem is that, the sender IP or hostname of external devices > which send syslog messages is not displayed in the log . Please see the > example below; > > > 2009-10-07T16:09:39.407770+03:00 debian dhcpd: DHCPACK on 172.10.10.101 > to > 00:24:7e:11:eb:49 (stajerstac) via vlan110 --------> "debian" hostname > is > OK > 2009-09-06T14:04:53+03:00 - rCT0[184]: craft(8) : RTRV-HDR:::P6099; > ----------------------------------------------------------------------- > ------------> > Neither IP nor hostname is printed > 2009-09-06T14:04:56+03:00 - clij[258]: isadmin at 10.5.5.23(21) : info > configure system sntp ------------------------------------------------- > ---> > Neither IP nor hostname is printed. Instead there is dash after the > timestamp. > > > I tried many things but was not able to make it work. > > Can someone help me to fix this ? > > Thanks > > Salim Engin > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From iamsayan at gmail.com Wed Oct 7 16:35:55 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Wed, 7 Oct 2009 20:05:55 +0530 Subject: [rsyslog] limiting message size while forwarding messages Message-ID: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> Hello All,RFC 3164 says that max rsyslog message length is 1024 bytes. Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 and has a default max message size of 2K. Now, I don't want to restrict the size while receiving messages.However when I forward message out of the box, I want to truncate it to 1K, as I am not sure what implementations I will have to interop with. I could not find an easy way to do this. $MaxMessageSize seems to control both sending and receiving. Is there a way in which I can achieve this? Regards, Sayan From rgerhards at hq.adiscon.com Wed Oct 7 16:43:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 7 Oct 2009 16:43:56 +0200 Subject: [rsyslog] limiting message size while forwarding messages References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> you need to look at the property replacer, and do something like %msg:1:950% in your template. I am not sure, but I think the default template already does this. Did you experience oversize messages on a receiver? > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Wednesday, October 07, 2009 4:36 PM > To: rsyslog-users > Subject: [rsyslog] limiting message size while forwarding messages > > Hello All,RFC 3164 says that max rsyslog message length is 1024 bytes. > Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 and > has a > default max message size of 2K. > Now, I don't want to restrict the size while receiving messages.However > when > I forward message out of the box, I want to truncate it > to 1K, as I am not sure what implementations I will have to interop > with. > I could not find an easy way to do this. $MaxMessageSize seems to > control > both sending and receiving. > Is there a way in which I can achieve this? > Regards, > Sayan > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From dirk.schulz at kinzesberg.de Wed Oct 7 16:31:34 2009 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Wed, 07 Oct 2009 16:31:34 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG Message-ID: <4ACCA646.8050209@kinzesberg.de> Hi folks, I am not sure if this is the right list to ask, but I hope that there is more of the related knowledge here. I have several Syslog-NG servers reporting to Rsyslog central servers (just because nobody could tell me how to run Rsyslog on NetBSD ...). Now I fould that the facility is not transferred to/evaluated on the Rsyslog servers; e.g. mail messages are written to syslog instead of maillog on the Rsyslog machines. Is there something special I have to configure on the Rsyslog servers or the Syslog-NG servers to fix that? Any hint or help is appreciated. Dirk From iamsayan at gmail.com Wed Oct 7 16:53:08 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Wed, 7 Oct 2009 20:23:08 +0530 Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> thanks Rainer, I will try that.So that means if I use the default template for forwarding, i think it should just work then. I did actually get a message from an application running on my box(uses standard syslog api) which was around 4K in size rsyslog did the right thing and truncated it to 2K.and logged into the file as per the configured rule. now I have requirement to support messages up to 5K while logging to the local file, but truncating it to 1K while sending it out. On Wed, Oct 7, 2009 at 8:13 PM, Rainer Gerhards wrote: > you need to look at the property replacer, and do something like > %msg:1:950% > in your template. I am not sure, but I think the default template already > does this. Did you experience oversize messages on a receiver? > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > Sent: Wednesday, October 07, 2009 4:36 PM > > To: rsyslog-users > > Subject: [rsyslog] limiting message size while forwarding messages > > > > Hello All,RFC 3164 says that max rsyslog message length is 1024 bytes. > > Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 and > > has a > > default max message size of 2K. > > Now, I don't want to restrict the size while receiving messages.However > > when > > I forward message out of the box, I want to truncate it > > to 1K, as I am not sure what implementations I will have to interop > > with. > > I could not find an easy way to do this. $MaxMessageSize seems to > > control > > both sending and receiving. > > Is there a way in which I can achieve this? > > Regards, > > Sayan > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 7 18:32:52 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 7 Oct 2009 18:32:52 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG References: <4ACCA646.8050209@kinzesberg.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dirk H. Schulz > Sent: Wednesday, October 07, 2009 4:32 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Getting Facility from Synlog-NG > > Hi folks, > > I am not sure if this is the right list to ask, but I hope that there > is > more of the related knowledge here. > > I have several Syslog-NG servers reporting to Rsyslog central servers > (just because nobody could tell me how to run Rsyslog on NetBSD ...). > Now I fould that the facility is not transferred to/evaluated on the > Rsyslog servers; e.g. mail messages are written to syslog instead of > maillog on the Rsyslog machines. This is strange. Facility is a fairly standard field. Could you tell me the content of %rawmwg%. This is done along these lines (if it does not work, check doc for exact syntax): $template raw,"%rawmsg%\n" *.* /path/to/file;raw Rainer > > Is there something special I have to configure on the Rsyslog servers > or > the Syslog-NG servers to fix that? > > Any hint or help is appreciated. > > Dirk > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 7 18:37:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 7 Oct 2009 18:37:17 +0200 Subject: [rsyslog] limiting message size while forwarding messages References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> I just checked, and the default is to send the full message. See: http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/syslogd.c;h=0f4f8a233c97 a44e72c97a555ff6f95eb367573a;hb=HEAD#l382 But all you need to do is use %msg:1:950...% in a template that otherwise is the same as the canned template. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Wednesday, October 07, 2009 4:53 PM > To: rsyslog-users > Subject: Re: [rsyslog] limiting message size while forwarding messages > > thanks Rainer, I will try that.So that means if I use the default > template > for forwarding, i think it should just work then. I did actually get a > message from an application running on my box(uses standard syslog api) > which was around 4K in size > rsyslog did the right thing and truncated it to 2K.and logged into the > file > as per the configured rule. > > now I have requirement to support messages up to 5K while logging to > the > local file, but truncating it to 1K while sending it out. > > > On Wed, Oct 7, 2009 at 8:13 PM, Rainer Gerhards > wrote: > > > you need to look at the property replacer, and do something like > > %msg:1:950% > > in your template. I am not sure, but I think the default template > already > > does this. Did you experience oversize messages on a receiver? > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > Sent: Wednesday, October 07, 2009 4:36 PM > > > To: rsyslog-users > > > Subject: [rsyslog] limiting message size while forwarding messages > > > > > > Hello All,RFC 3164 says that max rsyslog message length is 1024 > bytes. > > > Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 > and > > > has a > > > default max message size of 2K. > > > Now, I don't want to restrict the size while receiving > messages.However > > > when > > > I forward message out of the box, I want to truncate it > > > to 1K, as I am not sure what implementations I will have to interop > > > with. > > > I could not find an easy way to do this. $MaxMessageSize seems to > > > control > > > both sending and receiving. > > > Is there a way in which I can achieve this? > > > Regards, > > > Sayan > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From salim.engin at gmail.com Wed Oct 7 19:00:38 2009 From: salim.engin at gmail.com (Salim Engin) Date: Wed, 7 Oct 2009 20:00:38 +0300 Subject: [rsyslog] Sender IP or Hostname In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103191@GRFEXC.intern.adiscon.com> References: <43660480910070623s6eeb20b3gf200de18cff7d553@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103191@GRFEXC.intern.adiscon.com> Message-ID: <43660480910071000o6dbd760cn66ee567ccb844c28@mail.gmail.com> That is right. I configured an other networking device to use rsyslog and saw that the sender IP is displayed in the log file. But the WA mentioned in the document did not work ... Thanks On Wed, Oct 7, 2009 at 4:33 PM, Rainer Gerhards wrote: > Hi Salim, > > sounds like a malformed message is causing this: > > http://www.rsyslog.com/doc-syslog_parsing.html > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Salim Engin > > Sent: Wednesday, October 07, 2009 3:23 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] Sender IP or Hostname > > > > Hello all, > > > > I have just installed Debain 5.0.3 with its default syslog server which > > is > > rsyslog 3.18.6 . and I configured it to receive external syslog > > messages > > from our network devices. > > > > What the problem is that, the sender IP or hostname of external devices > > which send syslog messages is not displayed in the log . Please see the > > example below; > > > > > > 2009-10-07T16:09:39.407770+03:00 debian dhcpd: DHCPACK on 172.10.10.101 > > to > > 00:24:7e:11:eb:49 (stajerstac) via vlan110 --------> "debian" hostname > > is > > OK > > 2009-09-06T14:04:53+03:00 - rCT0[184]: craft(8) : RTRV-HDR:::P6099; > > ----------------------------------------------------------------------- > > ------------> > > Neither IP nor hostname is printed > > 2009-09-06T14:04:56+03:00 - clij[258]: isadmin at 10.5.5.23(21) : info > > configure system sntp ------------------------------------------------- > > ---> > > Neither IP nor hostname is printed. Instead there is dash after the > > timestamp. > > > > > > I tried many things but was not able to make it work. > > > > Can someone help me to fix this ? > > > > Thanks > > > > Salim Engin > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 7 19:05:28 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 7 Oct 2009 10:05:28 -0700 (PDT) Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> Message-ID: On Wed, 7 Oct 2009, Sayan Chowdhury wrote: > thanks Rainer, I will try that.So that means if I use the default template > for forwarding, i think it should just work then. I did actually get a > message from an application running on my box(uses standard syslog api) > which was around 4K in size > rsyslog did the right thing and truncated it to 2K.and logged into the file > as per the configured rule. > > now I have requirement to support messages up to 5K while logging to the > local file, but truncating it to 1K while sending it out. RFC 3164 says that a syslog implementation should truncate any oversized messages that it gets, so it should be safe to send a 2k message to a syslog daemon that truncates it to 1k note that in rsyslog the max message size is configrable. you may want to just set it to 1k until everything in your infrastructure handles larger messages (unless you want rsyslog to get the full message and send a portion of it, in which case you may want to set it larger) David Lang > > On Wed, Oct 7, 2009 at 8:13 PM, Rainer Gerhards wrote: > >> you need to look at the property replacer, and do something like >> %msg:1:950% >> in your template. I am not sure, but I think the default template already >> does this. Did you experience oversize messages on a receiver? >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury >>> Sent: Wednesday, October 07, 2009 4:36 PM >>> To: rsyslog-users >>> Subject: [rsyslog] limiting message size while forwarding messages >>> >>> Hello All,RFC 3164 says that max rsyslog message length is 1024 bytes. >>> Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 and >>> has a >>> default max message size of 2K. >>> Now, I don't want to restrict the size while receiving messages.However >>> when >>> I forward message out of the box, I want to truncate it >>> to 1K, as I am not sure what implementations I will have to interop >>> with. >>> I could not find an easy way to do this. $MaxMessageSize seems to >>> control >>> both sending and receiving. >>> Is there a way in which I can achieve this? >>> Regards, >>> Sayan >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iamsayan at gmail.com Wed Oct 7 20:11:49 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Wed, 7 Oct 2009 23:41:49 +0530 Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> Yes, I saw that too ... I made the template based out of Rsyslog traditional forward format and then used the property replacer as you mentioned. It works like a charm. Thanks a bunch! Regards, Sayan On Wed, Oct 7, 2009 at 10:07 PM, Rainer Gerhards wrote: > I just checked, and the default is to send the full message. See: > > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/syslogd.c;h=0f4f8a233c97 > a44e72c97a555ff6f95eb367573a;hb=HEAD#l382 > > But all you need to do is use %msg:1:950...% in a template that otherwise > is > the same as the canned template. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > Sent: Wednesday, October 07, 2009 4:53 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] limiting message size while forwarding messages > > > > thanks Rainer, I will try that.So that means if I use the default > > template > > for forwarding, i think it should just work then. I did actually get a > > message from an application running on my box(uses standard syslog api) > > which was around 4K in size > > rsyslog did the right thing and truncated it to 2K.and logged into the > > file > > as per the configured rule. > > > > now I have requirement to support messages up to 5K while logging to > > the > > local file, but truncating it to 1K while sending it out. > > > > > > On Wed, Oct 7, 2009 at 8:13 PM, Rainer Gerhards > > wrote: > > > > > you need to look at the property replacer, and do something like > > > %msg:1:950% > > > in your template. I am not sure, but I think the default template > > already > > > does this. Did you experience oversize messages on a receiver? > > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > > Sent: Wednesday, October 07, 2009 4:36 PM > > > > To: rsyslog-users > > > > Subject: [rsyslog] limiting message size while forwarding messages > > > > > > > > Hello All,RFC 3164 says that max rsyslog message length is 1024 > > bytes. > > > > Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 > > and > > > > has a > > > > default max message size of 2K. > > > > Now, I don't want to restrict the size while receiving > > messages.However > > > > when > > > > I forward message out of the box, I want to truncate it > > > > to 1K, as I am not sure what implementations I will have to interop > > > > with. > > > > I could not find an easy way to do this. $MaxMessageSize seems to > > > > control > > > > both sending and receiving. > > > > Is there a way in which I can achieve this? > > > > Regards, > > > > Sayan > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From jbondc at openmv.com Thu Oct 8 03:25:21 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Wed, 7 Oct 2009 21:25:21 -0400 Subject: [rsyslog] Queue System Message-ID: <007901ca47b6$34ac0870$9e041950$@com> Hi everyone, I'm been experimenting with rsyslogd to see if I could avoid losing messages/events when the central server is rebooted. I'll start with what works, then what doesn't L Both the server & client are running rsyslogd 3.22.1 The client has the following configuration: $ModLoad imklog.so # kernel logging $ModLoad imuxsock.so # provides support for local system logging $WorkDirectory /var/spool/rsyslog $ActionQueueType Disk $ActionQueueFileName srvrfwd $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on *.err;*.crit;*.emerg;*.warn;auth.*;user.notice @@server.domain.com # Stop the central server [root at server] $ service rsyslogd stop # Start client logger [root at client] $ service rsyslogd start ** (client) syslog(test a, NOTICE) ** (client) syslog(test b, NOTICE) Since the central server is down I expect that the client will keep messages in the disk queue and send them when the server is back online. # Start the central server [root at server] $ service rsyslogd start . 2009-10-08T00:44:21-04:00 client php: test a 2009-10-08T00:45:39-04:00 client php: test b . Great it works! ****** Now the problem, when client and server are both connected and exchanging messages # Stop the central server [root at server] $ service rsyslogd stop ** (client) syslog(test c - lost message, NOTICE) With some debugging: 4241.782860452:action 10 queue:Reg/w0: action 10 queue: entry deleted, state 0, size now 0 entries 4241.782860452:action 10 queue:Reg/w0: server 4241.782860452:action 10 queue:Reg/w0: server:10514/tcp 4241.782860452:action 10 queue:Reg/w0: TCP sent 36 bytes, requested 36 The client 'sends the message over tcp' and actually empties the queue! Is this expected behavior? rsyslogd was shutdown 'cleanly' on the server. Shouldn't it force the client to close the TCP connection? Using netstat, the two servers are : Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 client.10514 server..60726 FIN_WAIT_2 Proto Recv-Q Send-Q Local Address Foreign Address State tcp 1 36 server:60726 client:10514 CLOSE_WAIT Have I misunderstood something? I've read Rainer's blog and issues with TCP "unreliability" / case of a power failure but don't think this related From david at lang.hm Thu Oct 8 06:58:21 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 7 Oct 2009 21:58:21 -0700 (PDT) Subject: [rsyslog] Queue System In-Reply-To: <007901ca47b6$34ac0870$9e041950$@com> References: <007901ca47b6$34ac0870$9e041950$@com> Message-ID: On Wed, 7 Oct 2009, Jonathan Bond-Caron wrote: > The client 'sends the message over tcp' and actually empties the queue! > > Is this expected behavior? rsyslogd was shutdown 'cleanly' on the server. > Shouldn't it force the client to close the TCP connection? > > > > Using netstat, the two servers are : > > > > Proto Recv-Q Send-Q Local Address Foreign Address (state) > > tcp4 0 0 client.10514 server..60726 FIN_WAIT_2 > > > > Proto Recv-Q Send-Q Local Address Foreign Address > State > > tcp 1 36 server:60726 client:10514 CLOSE_WAIT > > > > Have I misunderstood something? I've read Rainer's blog and issues with TCP > "unreliability" / case of a power failure but don't think this related I think it's the same problem, one side thinks the connection is closed, the other doesn't and so the one that doesn't happily sends the data out. try reconfiguring to use relp and things should work as expected. now, I don't know why the close isn't getting through from the one system to the other rapidly. is there any sort of firewall in between them? David Lang From rgerhards at hq.adiscon.com Thu Oct 8 09:27:30 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 09:27:30 +0200 Subject: [rsyslog] limiting message size while forwarding messages References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com><87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Wednesday, October 07, 2009 8:12 PM > To: rsyslog-users > Subject: Re: [rsyslog] limiting message size while forwarding messages > > Yes, I saw that too ... I made the template based out of Rsyslog > traditional > forward format and then used the property replacer as you mentioned. > It works like a charm. Thanks a bunch! I am glad it works, but I also tend to think that the default template is probably not correct. As it claims to be "traditional" format, I think it should really limit itself to 1K message size, so that another template must be picked if messages of "non-traditional large size" are to be transmitted. However, changing that default would potentially break a number of existing deployments. Does anybody else have an opinion on that? Thanks, Rainer > Regards, > Sayan > > > On Wed, Oct 7, 2009 at 10:07 PM, Rainer Gerhards > wrote: > > > I just checked, and the default is to send the full message. See: > > > > > > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/syslogd.c;h=0f4f8a > 233c97 > > a44e72c97a555ff6f95eb367573a;hb=HEAD#l382 > > > > But all you need to do is use %msg:1:950...% in a template that > otherwise > > is > > the same as the canned template. > > > > Rainer > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > Sent: Wednesday, October 07, 2009 4:53 PM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] limiting message size while forwarding > messages > > > > > > thanks Rainer, I will try that.So that means if I use the default > > > template > > > for forwarding, i think it should just work then. I did actually > get a > > > message from an application running on my box(uses standard syslog > api) > > > which was around 4K in size > > > rsyslog did the right thing and truncated it to 2K.and logged into > the > > > file > > > as per the configured rule. > > > > > > now I have requirement to support messages up to 5K while logging > to > > > the > > > local file, but truncating it to 1K while sending it out. > > > > > > > > > On Wed, Oct 7, 2009 at 8:13 PM, Rainer Gerhards > > > wrote: > > > > > > > you need to look at the property replacer, and do something like > > > > %msg:1:950% > > > > in your template. I am not sure, but I think the default template > > > already > > > > does this. Did you experience oversize messages on a receiver? > > > > > > > > > -----Original Message----- > > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > > > Sent: Wednesday, October 07, 2009 4:36 PM > > > > > To: rsyslog-users > > > > > Subject: [rsyslog] limiting message size while forwarding > messages > > > > > > > > > > Hello All,RFC 3164 says that max rsyslog message length is 1024 > > > bytes. > > > > > Rsyslog version 4.2.0 which I am using, is compliant to rfc > 5424 > > > and > > > > > has a > > > > > default max message size of 2K. > > > > > Now, I don't want to restrict the size while receiving > > > messages.However > > > > > when > > > > > I forward message out of the box, I want to truncate it > > > > > to 1K, as I am not sure what implementations I will have to > interop > > > > > with. > > > > > I could not find an easy way to do this. $MaxMessageSize seems > to > > > > > control > > > > > both sending and receiving. > > > > > Is there a way in which I can achieve this? > > > > > Regards, > > > > > Sayan > > > > > _______________________________________________ > > > > > rsyslog mailing list > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > http://www.rsyslog.com > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Thu Oct 8 09:41:16 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 8 Oct 2009 00:41:16 -0700 (PDT) Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com><87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 8 Oct 2009, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury >> >> Yes, I saw that too ... I made the template based out of Rsyslog >> traditional >> forward format and then used the property replacer as you mentioned. >> It works like a charm. Thanks a bunch! > > I am glad it works, but I also tend to think that the default template is > probably not correct. As it claims to be "traditional" format, I think it > should really limit itself to 1K message size, so that another template must > be picked if messages of "non-traditional large size" are to be transmitted. > > However, changing that default would potentially break a number of existing > deployments. > > Does anybody else have an opinion on that? existing syslog implementations already need to be able to deal with oversized messages (almost nothing checks the size before it gets sent), so I don't see a big benifit in changing rsyslog to limit what it outputs. David Lang From rgerhards at hq.adiscon.com Thu Oct 8 09:43:47 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 09:43:47 +0200 Subject: [rsyslog] Queue System References: <007901ca47b6$34ac0870$9e041950$@com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710319B@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, October 08, 2009 6:58 AM > To: rsyslog-users > Subject: Re: [rsyslog] Queue System > > On Wed, 7 Oct 2009, Jonathan Bond-Caron wrote: > > > The client 'sends the message over tcp' and actually empties the > queue! > > > > Is this expected behavior? rsyslogd was shutdown 'cleanly' on the > server. > > Shouldn't it force the client to close the TCP connection? > > > > > > > > Using netstat, the two servers are : > > > > > > > > Proto Recv-Q Send-Q Local Address Foreign Address > (state) > > > > tcp4 0 0 client.10514 server..60726 FIN_WAIT_2 > > > > > > > > Proto Recv-Q Send-Q Local Address Foreign Address > > State > > > > tcp 1 36 server:60726 client:10514 > CLOSE_WAIT > > > > > > > > Have I misunderstood something? I've read Rainer's blog and issues > with TCP > > "unreliability" / case of a power failure but don't think this > related > > I think it's the same problem, one side thinks the connection is > closed, > the other doesn't and so the one that doesn't happily sends the data > out. > > try reconfiguring to use relp and things should work as expected. > > > now, I don't know why the close isn't getting through from the one > system > to the other rapidly. is there any sort of firewall in between them? The close may get through. It is a (kind of) race condition, inside the tcp stack. Assume the following happens (System S being the server, system C being the client): S : closes connection, sends close info C1: rsyslog writes data (not yet sent, but acknowledged by tcp stack) C2: receives close request C3: discards data The race is in C1/C2. In this order, data is lost. If C2 happens before C1, no data is lost. No chance to solve that with plain tcp without app-level acks. Rainer From david at lang.hm Thu Oct 8 09:48:08 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 8 Oct 2009 00:48:08 -0700 (PDT) Subject: [rsyslog] Queue System In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710319B@GRFEXC.intern.adiscon.com> References: <007901ca47b6$34ac0870$9e041950$@com> <9B6E2A8877C38245BFB15CC491A11DA710319B@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 8 Oct 2009, Rainer Gerhards wrote: >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >> On Wed, 7 Oct 2009, Jonathan Bond-Caron wrote: >> >>> The client 'sends the message over tcp' and actually empties the >> queue! >>> >>> Is this expected behavior? rsyslogd was shutdown 'cleanly' on the >> server. >>> Shouldn't it force the client to close the TCP connection? >>> >>> >>> >>> Using netstat, the two servers are : >>> >>> >>> >>> Proto Recv-Q Send-Q Local Address Foreign Address >> (state) >>> >>> tcp4 0 0 client.10514 server..60726 FIN_WAIT_2 >>> >>> >>> >>> Proto Recv-Q Send-Q Local Address Foreign Address >>> State >>> >>> tcp 1 36 server:60726 client:10514 >> CLOSE_WAIT >>> >>> >>> >>> Have I misunderstood something? I've read Rainer's blog and issues >> with TCP >>> "unreliability" / case of a power failure but don't think this >> related >> >> I think it's the same problem, one side thinks the connection is >> closed, >> the other doesn't and so the one that doesn't happily sends the data >> out. >> >> try reconfiguring to use relp and things should work as expected. >> >> >> now, I don't know why the close isn't getting through from the one >> system >> to the other rapidly. is there any sort of firewall in between them? > > The close may get through. It is a (kind of) race condition, inside the tcp > stack. Assume the following happens (System S being the server, system C > being the client): > > S : closes connection, sends close info > C1: rsyslog writes data (not yet sent, but acknowledged by tcp stack) > C2: receives close request > C3: discards data > > The race is in C1/C2. In this order, data is lost. If C2 happens before C1, > no data is lost. No chance to solve that with plain tcp without app-level > acks. but that window should be very short, for it to last long enough to show up in the netstat commands sounds odd to me. David Lang From rgerhards at hq.adiscon.com Thu Oct 8 09:51:39 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 09:51:39 +0200 Subject: [rsyslog] Queue System References: <007901ca47b6$34ac0870$9e041950$@com><9B6E2A8877C38245BFB15CC491A11DA710319B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710319D@GRFEXC.intern.adiscon.com> > -----Original Message----- > > The close may get through. It is a (kind of) race condition, inside > the tcp > > stack. Assume the following happens (System S being the server, > system C > > being the client): > > > > S : closes connection, sends close info > > C1: rsyslog writes data (not yet sent, but acknowledged by tcp stack) > > C2: receives close request > > C3: discards data > > > > The race is in C1/C2. In this order, data is lost. If C2 happens > before C1, > > no data is lost. No chance to solve that with plain tcp without app- > level > > acks. > > but that window should be very short, for it to last long enough to > show > up in the netstat commands sounds odd to me. Ah, I didn't pay attention to the netstat. You are right, in *this* case the close looks like it does not go through. HOWEVER, for the argument I have given, this race exists in general. The window is extremely short (at least on a local LAN), but I have learned if there is a potential for a race, it will happen sooner or later. Chances go up very soon as quickly as you have millions and millions of cases... Just yesterday I was able to find a race with a much lower probability in v5-beta during shutdown... And it really happened (thankfully only in my lab, where I set parameters to make such races more probable). Rainer From martinmie at PartyGaming.com Thu Oct 8 10:50:13 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 8 Oct 2009 10:50:13 +0200 Subject: [rsyslog] URGENT - rsyslog stops sshd Message-ID: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Hi all, odd behavior here... Today one of our core systems stopped accepting ssh connections. -- $ telnet sshserver 22 Trying 1.2.3.4... Connected to sshserver. Escape character is '^]'. Connection closed. -- Using a pre-established session rsyslog was shutdown: -- # /etc/init.d/rsyslog stop Shutting down system logger: [ OK ][/quote] -- ..and the system started accepting connections again: -- $ ssh sshserver Last login: Thu Oct 8 01:12:59 2009 from foo.bar.tld -- We are just one step before dropping rsyslog and use syslog-ng again (which was dismissed some time ago to favor rsyslog) but I would like to know what's the reason for this and, most important, what's the solution :-) Some additional info: -- # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 32767 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 32767 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited -- Thanks! Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From david at lang.hm Thu Oct 8 11:16:44 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 8 Oct 2009 02:16:44 -0700 (PDT) Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Message-ID: On Thu, 8 Oct 2009, Martin Mielke wrote: > Hi all, > > odd behavior here... > > Today one of our core systems stopped accepting ssh connections. > -- > $ telnet sshserver 22 > Trying 1.2.3.4... > Connected to sshserver. > Escape character is '^]'. > Connection closed. > -- > > Using a pre-established session rsyslog was shutdown: > -- > # /etc/init.d/rsyslog stop > Shutting down system logger: [ OK > ][/quote] > -- > > ..and the system started accepting connections again: > -- > $ ssh sshserver > Last login: Thu Oct 8 01:12:59 2009 from foo.bar.tld > -- > > We are just one step before dropping rsyslog and use syslog-ng again > (which was dismissed some time ago to favor rsyslog) but I would like to > know what's the reason for this and, most important, what's the solution > :-) ssh logs connections, if the syslog process cannot process the message, ssh is designed to stop and wait until it does (it deems the log so important that it refuses to do anything until the log is written) conole logins do the same thing. so something causes rsyslog to stop accepting messages. the same thing can happen to syslog-ng or to plain sysklog, just under different conditions. so the question is why rsyslog stopped. what is rsyslog configured to do with log messages? is there any chance that it was unable to do something with a message and so would have had to keep it in it's queue until the queue filled up? (logging via TCP to a remote server that stops responding will do this, so will writing to a full filesystem) rsyslog can be configured to accept and discard log entries when the queue is full, doing this can avoid this sort of situation. David Lang > Some additional info: > -- > # ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 32767 > max locked memory (kbytes, -l) 32 > max memory size (kbytes, -m) unlimited > open files (-n) 1024 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 10240 > cpu time (seconds, -t) unlimited > max user processes (-u) 32767 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > -- > > > Thanks! > > Martin > > > This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. > > Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iamsayan at gmail.com Thu Oct 8 11:16:14 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Thu, 8 Oct 2009 05:16:14 -0400 Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540910080216y1caef6begd013ff6b722e76ef@mail.gmail.com> I agree ... As long as rsyslog has a mechanism to do the truncation if somebody wants on the egress(which it has), it is fine. I don't think the default template should change. BTW, there are implementations out there which behaves badly when they receive large messages/control characters etc. I want to be defensive about what send out and hence my requirement. On Thu, Oct 8, 2009 at 3:41 AM, wrote: > On Thu, 8 Oct 2009, Rainer Gerhards wrote: > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > >> > >> Yes, I saw that too ... I made the template based out of Rsyslog > >> traditional > >> forward format and then used the property replacer as you mentioned. > >> It works like a charm. Thanks a bunch! > > > > I am glad it works, but I also tend to think that the default template is > > probably not correct. As it claims to be "traditional" format, I think it > > should really limit itself to 1K message size, so that another template > must > > be picked if messages of "non-traditional large size" are to be > transmitted. > > > > However, changing that default would potentially break a number of > existing > > deployments. > > > > Does anybody else have an opinion on that? > > existing syslog implementations already need to be able to deal with > oversized messages (almost nothing checks the size before it gets sent), > so I don't see a big benifit in changing rsyslog to limit what it outputs. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu Oct 8 11:26:15 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 8 Oct 2009 02:26:15 -0700 (PDT) Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <87a1ae540910080216y1caef6begd013ff6b722e76ef@mail.gmail.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> <87a1ae540910080216y1caef6begd013ff6b722e76ef@mail.gmail.com> Message-ID: On Thu, 8 Oct 2009, Sayan Chowdhury wrote: > I agree ... As long as rsyslog has a mechanism to do the truncation if > somebody wants on the egress(which it has), it is fine. I don't think the > default template should change. > BTW, there are implementations out there which behaves badly when they > receive large messages/control characters etc. I want to be defensive about > what send out and hence my requirement. which implementation caused you grief? (so I know what to avoid) David Lang > > > > On Thu, Oct 8, 2009 at 3:41 AM, wrote: > >> On Thu, 8 Oct 2009, Rainer Gerhards wrote: >> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury >>>> >>>> Yes, I saw that too ... I made the template based out of Rsyslog >>>> traditional >>>> forward format and then used the property replacer as you mentioned. >>>> It works like a charm. Thanks a bunch! >>> >>> I am glad it works, but I also tend to think that the default template is >>> probably not correct. As it claims to be "traditional" format, I think it >>> should really limit itself to 1K message size, so that another template >> must >>> be picked if messages of "non-traditional large size" are to be >> transmitted. >>> >>> However, changing that default would potentially break a number of >> existing >>> deployments. >>> >>> Does anybody else have an opinion on that? >> >> existing syslog implementations already need to be able to deal with >> oversized messages (almost nothing checks the size before it gets sent), >> so I don't see a big benifit in changing rsyslog to limit what it outputs. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Oct 8 11:28:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 11:28:56 +0200 Subject: [rsyslog] limiting message size while forwarding messages References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com><87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com><87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com><87a1ae540910080216y1caef6begd013ff6b722e76ef@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031A2@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, October 08, 2009 11:26 AM > To: rsyslog-users > Subject: Re: [rsyslog] limiting message size while forwarding messages > > On Thu, 8 Oct 2009, Sayan Chowdhury wrote: > > > I agree ... As long as rsyslog has a mechanism to do the truncation > if > > somebody wants on the egress(which it has), it is fine. I don't think > the > > default template should change. > > BTW, there are implementations out there which behaves badly when > they > > receive large messages/control characters etc. I want to be defensive > about > > what send out and hence my requirement. > > which implementation caused you grief? (so I know what to avoid) > The one I know about is darn old Solaris syslogd, which reliably died with messages of 1025 bytes and above. However, I think this is fixed for 10 years or so now ;) I've seen no other problems with implementations, but I definitely have not seen everything ;) Rainer > David Lang > > > > > > > > > On Thu, Oct 8, 2009 at 3:41 AM, wrote: > > > >> On Thu, 8 Oct 2009, Rainer Gerhards wrote: > >> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > >>>> > >>>> Yes, I saw that too ... I made the template based out of Rsyslog > >>>> traditional > >>>> forward format and then used the property replacer as you > mentioned. > >>>> It works like a charm. Thanks a bunch! > >>> > >>> I am glad it works, but I also tend to think that the default > template is > >>> probably not correct. As it claims to be "traditional" format, I > think it > >>> should really limit itself to 1K message size, so that another > template > >> must > >>> be picked if messages of "non-traditional large size" are to be > >> transmitted. > >>> > >>> However, changing that default would potentially break a number of > >> existing > >>> deployments. > >>> > >>> Does anybody else have an opinion on that? > >> > >> existing syslog implementations already need to be able to deal with > >> oversized messages (almost nothing checks the size before it gets > sent), > >> so I don't see a big benifit in changing rsyslog to limit what it > outputs. > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From dag at wieers.com Thu Oct 8 12:36:38 2009 From: dag at wieers.com (Dag Wieers) Date: Thu, 8 Oct 2009 12:36:38 +0200 (CEST) Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Message-ID: On Thu, 8 Oct 2009, david at lang.hm wrote: > rsyslog can be configured to accept and discard log entries when the queue > is full, doing this can avoid this sort of situation. Hi Martin, We were in a similar situation recently (because of a limit of open file descriptors) and I am very interested to learn how you can do the above. Thanks in advance, -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] From martinmie at PartyGaming.com Thu Oct 8 13:28:36 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 8 Oct 2009 13:28:36 +0200 Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Message-ID: <0B1B9163138571439EAF804646F3F6AA08771181@GIBSVWIN004X.partygaming.local> Hi Dag, The number of open file descriptors wasn't the issue here... and I'm also interested in how to configure rsyslog as described by David ;-) Cheers, Martin > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dag Wieers > Sent: 08 October 2009 12:37 > To: rsyslog-users > Subject: Re: [rsyslog] URGENT - rsyslog stops sshd > > On Thu, 8 Oct 2009, david at lang.hm wrote: > > > rsyslog can be configured to accept and discard log entries when the > queue > > is full, doing this can avoid this sort of situation. > > Hi Martin, > > We were in a similar situation recently (because of a limit of open file > descriptors) and I am very interested to learn how you can do the above. > > Thanks in advance, > -- > -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- > [Any errors in spelling, tact or fact are transmission errors] > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > nfo/rsyslog > http://www.rsyslog.com This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From martinmie at PartyGaming.com Thu Oct 8 13:32:23 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 8 Oct 2009 13:32:23 +0200 Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Message-ID: <0B1B9163138571439EAF804646F3F6AA08771183@GIBSVWIN004X.partygaming.local> Hi, > > ssh logs connections, if the syslog process cannot process the message, > ssh is designed to stop and wait until it does (it deems the log so > important that it refuses to do anything until the log is written) > > conole logins do the same thing. Is there any way I can test this? Strace? Ltrace? Sshd configuration directive?... > > so something causes rsyslog to stop accepting messages. the same thing can > happen to syslog-ng or to plain sysklog, just under different conditions. > > so the question is why rsyslog stopped. > > what is rsyslog configured to do with log messages? > > is there any chance that it was unable to do something with a message and > so would have had to keep it in it's queue until the queue filled up? > (logging via TCP to a remote server that stops responding will do this, so > will writing to a full filesystem) The system sends its logs over TCP to a remote logserver. Initial analysis yield a routing problem which should have been fixed so I hope not to see such a problem due to this. But, my question now is: shouldn't logfiles be spooled under $WorkDirectory /var/spool/rsyslog and sent over to the logserver once it becomes available again? There's plenty of space on /var ... > > rsyslog can be configured to accept and discard log entries when the queue > is full, doing this can avoid this sort of situation. Could you please tell us how? :-) Regards, Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From rgerhards at hq.adiscon.com Thu Oct 8 14:11:45 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 14:11:45 +0200 Subject: [rsyslog] URGENT - rsyslog stops sshd References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031A5@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dag Wieers > Sent: Thursday, October 08, 2009 12:37 PM > To: rsyslog-users > Subject: Re: [rsyslog] URGENT - rsyslog stops sshd > > On Thu, 8 Oct 2009, david at lang.hm wrote: > > > rsyslog can be configured to accept and discard log entries when the > queue > > is full, doing this can avoid this sort of situation. > > Hi Martin, > > We were in a similar situation recently (because of a limit of open > file > descriptors) and I am very interested to learn how you can do the > above. Well, it depends on your configuration. If you use a plain default config, this situation is expected to never happen, because we assume that log files can be written and discard messages when this is not the case. However, if you begin to configure the system to be reliable, you need to think about the implications. In almost all cases where I have seen such a problem (assuming it was not a real bug, of course), the configuration demanded reliable delivery, but it was not thought about the fact that if the message could not be delivered, the system would stall. Some organizations actually prefer this mode, so it may be useful. If you post your config, I could check if there is something in it that demands such reliability and prevents the engine from discarding messages when needed. Rainer From martinmie at PartyGaming.com Thu Oct 8 14:24:06 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 8 Oct 2009 14:24:06 +0200 Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031A5@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> <9B6E2A8877C38245BFB15CC491A11DA71031A5@GRFEXC.intern.adiscon.com> Message-ID: <0B1B9163138571439EAF804646F3F6AA087711B7@GIBSVWIN004X.partygaming.local> Hi Rainer, This is the rsyslog.conf used - quite simple as you can see: -- $ cat /etc/rsyslog.conf # rsyslog v3: load input modules # If you do not load inputs, nothing happens! # You may need to set the module load path if modules are not found. $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # provides kernel logging support (previously done by rklogd) # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none -/var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* -/var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit -/var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. $WorkDirectory /var/spool/rsyslog # where to place spool files $ActionQueueFileName uniqName # unique name prefix for spool files $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) $ActionQueueSaveOnShutdown on # save messages to disk on shutdown $ActionQueueType LinkedList # run asynchronously $ActionResumeRetryCount 5 # five retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional *.* @@logserver -- Do you see anything suspicious? Thanks, Martin > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: 08 October 2009 14:12 > To: rsyslog-users > Subject: Re: [rsyslog] URGENT - rsyslog stops sshd > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Dag Wieers > > Sent: Thursday, October 08, 2009 12:37 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] URGENT - rsyslog stops sshd > > > > On Thu, 8 Oct 2009, david at lang.hm wrote: > > > > > rsyslog can be configured to accept and discard log entries when the > > queue > > > is full, doing this can avoid this sort of situation. > > > > Hi Martin, > > > > We were in a similar situation recently (because of a limit of open > > file > > descriptors) and I am very interested to learn how you can do the > > above. > > Well, it depends on your configuration. If you use a plain default config, > this situation is expected to never happen, because we assume that log > files > can be written and discard messages when this is not the case. However, if > you begin to configure the system to be reliable, you need to think about > the > implications. In almost all cases where I have seen such a problem > (assuming > it was not a real bug, of course), the configuration demanded reliable > delivery, but it was not thought about the fact that if the message could > not > be delivered, the system would stall. Some organizations actually prefer > this > mode, so it may be useful. > > If you post your config, I could check if there is something in it that > demands such reliability and prevents the engine from discarding messages > when needed. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > /rsyslog > http://www.rsyslog.com This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From jbondc at openmv.com Thu Oct 8 15:17:44 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Thu, 8 Oct 2009 09:17:44 -0400 Subject: [rsyslog] Queue System In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710319D@GRFEXC.intern.adiscon.com> References: <007901ca47b6$34ac0870$9e041950$@com><9B6E2A8877C38245BFB15CC491A11DA710319B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710319D@GRFEXC.intern.adiscon.com> Message-ID: <000e01ca4819$b942b760$2bc82620$@com> On Thu Oct 8 03:51 AM, Rainer Gerhards wrote: > > -----Original Message----- > > > The close may get through. It is a (kind of) race condition, > > > inside > > the tcp > > > stack. Assume the following happens (System S being the server, > > system C > > > being the client): > > Ah, I didn't pay attention to the netstat. You are right, in *this* > case the close looks like it does not go through. HOWEVER, for the > argument I have given, this race exists in general. The window is > extremely short (at least on a local LAN), but I have learned if there > is a potential for a race, it will happen sooner or later. Chances go > up very soon as quickly as you have millions and millions of cases... > Just yesterday I was able to find a race with a much lower probability > in v5-beta during shutdown... And it really happened (thankfully only > in my lab, where I set parameters to make such races more probable). > I decided to look at the code and I must say, wow documented open source code! I haven't written c in a while but easily found my way, so big congrats on keeping the code so clean. After some googling, I found that whenever TCP is in status CLOSE_WAIT, the host is expected to close the socket sometime soon. It might not want to close it right away for example to keep-alive the connection... So I searched for all close() calls on the socket. I found sockClose() then static void CheckConnection(nsd_t *pNsd) -- runtime/nsd_ptcp.c http://blog.gerhards.net/2008/06/getting-bit-more-reliability-from-plain.htm l With debug mode I found: 5543.432507331:action 10 queue:Reg/w0: server:10514/tcp 5543.432507331:action 10 queue:Reg/w0: TCP sent 36 bytes, requested 36 It turns out I was pointing to the SSL config which had: $DefaultNetstreamDriver gtls For this, I found: void CheckConnection(nsd_t *pNsd) -- runtime/nsd_gtls.c /* This function checks if the connection is still alive - well, kind of... * This is a dummy here. For details, check function common in ptcp driver. * rgerhards, 2008-06-09 */ static void CheckConnection(nsd_t *pNsd) { dbgprintf("CheckConnection SSL - do something\n"); nsd_gtls_t *pThis = (nsd_gtls_t*) pNsd; ISOBJ_TYPE_assert(pThis, nsd_gtls); nsd_ptcp.CheckConnection(pThis->pTcp); } 5649.998580185:action 10 queue:Reg/w0: server:10514/tcp 5649.998580185:action 10 queue:Reg/w0: CheckConnection SSL - do something 5649.998580185:action 10 queue:Reg/w0: CheckConnection detected broken connection - closing it Wonderful! It works as I would expect. But, is there any reason why the socket check wasn't added for SSL? I'm currently testing this 'patch' with a couple of 'live' servers and randomly stopping the central logging server. From dirk.schulz at kinzesberg.de Thu Oct 8 17:58:51 2009 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Thu, 08 Oct 2009 17:58:51 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> References: <4ACCA646.8050209@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> Message-ID: <4ACE0C3B.7090207@kinzesberg.de> Rainer Gerhards schrieb: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Dirk H. Schulz >> Sent: Wednesday, October 07, 2009 4:32 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] Getting Facility from Synlog-NG >> >> Hi folks, >> >> I am not sure if this is the right list to ask, but I hope that there >> is >> more of the related knowledge here. >> >> I have several Syslog-NG servers reporting to Rsyslog central servers >> (just because nobody could tell me how to run Rsyslog on NetBSD ...). >> Now I fould that the facility is not transferred to/evaluated on the >> Rsyslog servers; e.g. mail messages are written to syslog instead of >> maillog on the Rsyslog machines. >> > > This is strange. Facility is a fairly standard field. Could you tell me the > content of %rawmwg%. This is done along these lines (if it does not work, > check doc for exact syntax): > > $template raw,"%rawmsg%\n" > *.* /path/to/file;raw > I did not find these terms, neither in rsyslog.conf on the central logserver servers nor in syslog-ng.conf on the log client servers. Rsyslog is 3.18.6-4 on Debian 5. Dirk > > Rainer > >> Is there something special I have to configure on the Rsyslog servers >> or >> the Syslog-NG servers to fix that? >> >> Any hint or help is appreciated. >> >> Dirk >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Oct 8 18:03:54 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 18:03:54 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG References: <4ACCA646.8050209@kinzesberg.de><9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> <4ACE0C3B.7090207@kinzesberg.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031AC@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dirk H. Schulz > Sent: Thursday, October 08, 2009 5:59 PM > To: rsyslog-users > Subject: Re: [rsyslog] Getting Facility from Synlog-NG > > Rainer Gerhards schrieb: > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Dirk H. Schulz > >> Sent: Wednesday, October 07, 2009 4:32 PM > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] Getting Facility from Synlog-NG > >> > >> Hi folks, > >> > >> I am not sure if this is the right list to ask, but I hope that > there > >> is > >> more of the related knowledge here. > >> > >> I have several Syslog-NG servers reporting to Rsyslog central > servers > >> (just because nobody could tell me how to run Rsyslog on NetBSD > ...). > >> Now I fould that the facility is not transferred to/evaluated on the > >> Rsyslog servers; e.g. mail messages are written to syslog instead of > >> maillog on the Rsyslog machines. > >> > > > > This is strange. Facility is a fairly standard field. Could you tell > me the > > content of %rawmwg%. This is done along these lines (if it does not > work, > > check doc for exact syntax): > > > > $template raw,"%rawmsg%\n" > > *.* /path/to/file;raw > > > > I did not find these terms, neither in rsyslog.conf on the central > logserver servers nor in syslog-ng.conf on the log client servers. > Rsyslog is 3.18.6-4 on Debian 5. Well, these are the config statements for the central rsyslog instance... Rainer From dirk.schulz at kinzesberg.de Thu Oct 8 18:22:49 2009 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Thu, 08 Oct 2009 18:22:49 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031AC@GRFEXC.intern.adiscon.com> References: <4ACCA646.8050209@kinzesberg.de><9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> <4ACE0C3B.7090207@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA71031AC@GRFEXC.intern.adiscon.com> Message-ID: <4ACE11D9.1020806@kinzesberg.de> Rainer Gerhards schrieb: >>> check doc for exact syntax): >>> >>> $template raw,"%rawmsg%\n" >>> *.* /path/to/file;raw >>> >>> >> I did not find these terms, neither in rsyslog.conf on the central >> logserver servers nor in syslog-ng.conf on the log client servers. >> Rsyslog is 3.18.6-4 on Debian 5. >> > > Well, these are the config statements for the central rsyslog instance... > Does that mean that I am lacking a template statement in the central rsyslog config? Dirk > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iamsayan at gmail.com Thu Oct 8 18:48:33 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Thu, 8 Oct 2009 22:18:33 +0530 Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031A2@GRFEXC.intern.adiscon.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> <87a1ae540910080216y1caef6begd013ff6b722e76ef@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71031A2@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540910080948t1859f0dav90e3cd15c1f951b7@mail.gmail.com> it was an in house properietary "data mining" implementation. I don't know why people try to reinvent the wheel :) On Thu, Oct 8, 2009 at 2:58 PM, Rainer Gerhards wrote: > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Thursday, October 08, 2009 11:26 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] limiting message size while forwarding messages > > > > On Thu, 8 Oct 2009, Sayan Chowdhury wrote: > > > > > I agree ... As long as rsyslog has a mechanism to do the truncation > > if > > > somebody wants on the egress(which it has), it is fine. I don't think > > the > > > default template should change. > > > BTW, there are implementations out there which behaves badly when > > they > > > receive large messages/control characters etc. I want to be defensive > > about > > > what send out and hence my requirement. > > > > which implementation caused you grief? (so I know what to avoid) > > > > The one I know about is darn old Solaris syslogd, which reliably died with > messages of 1025 bytes and above. However, I think this is fixed for 10 > years > or so now ;) I've seen no other problems with implementations, but I > definitely have not seen everything ;) > > Rainer > > > David Lang > > > > > > > > > > > > > > On Thu, Oct 8, 2009 at 3:41 AM, wrote: > > > > > >> On Thu, 8 Oct 2009, Rainer Gerhards wrote: > > >> > > >>>> -----Original Message----- > > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >>>> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > >>>> > > >>>> Yes, I saw that too ... I made the template based out of Rsyslog > > >>>> traditional > > >>>> forward format and then used the property replacer as you > > mentioned. > > >>>> It works like a charm. Thanks a bunch! > > >>> > > >>> I am glad it works, but I also tend to think that the default > > template is > > >>> probably not correct. As it claims to be "traditional" format, I > > think it > > >>> should really limit itself to 1K message size, so that another > > template > > >> must > > >>> be picked if messages of "non-traditional large size" are to be > > >> transmitted. > > >>> > > >>> However, changing that default would potentially break a number of > > >> existing > > >>> deployments. > > >>> > > >>> Does anybody else have an opinion on that? > > >> > > >> existing syslog implementations already need to be able to deal with > > >> oversized messages (almost nothing checks the size before it gets > > sent), > > >> so I don't see a big benifit in changing rsyslog to limit what it > > outputs. > > >> > > >> David Lang > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iamsayan at gmail.com Thu Oct 8 18:51:45 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Thu, 8 Oct 2009 22:21:45 +0530 Subject: [rsyslog] Getting Facility from Synlog-NG In-Reply-To: <4ACE11D9.1020806@kinzesberg.de> References: <4ACCA646.8050209@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> <4ACE0C3B.7090207@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA71031AC@GRFEXC.intern.adiscon.com> <4ACE11D9.1020806@kinzesberg.de> Message-ID: <87a1ae540910080951g6c7106fby53979808642cb537@mail.gmail.com> Hello Dirk, I think Rainer wants you to add this to your rsyslog.conf file, and send him the file that gets saved because of adding this statement. So as to debug the issue. You will need to restart rsyslog to pick up the config change Hope this helps. Sayan On Thu, Oct 8, 2009 at 9:52 PM, Dirk H. Schulz wrote: > Rainer Gerhards schrieb: > >>> check doc for exact syntax): > >>> > >>> $template raw,"%rawmsg%\n" > >>> *.* /path/to/file;raw > >>> > >>> > >> I did not find these terms, neither in rsyslog.conf on the central > >> logserver servers nor in syslog-ng.conf on the log client servers. > >> Rsyslog is 3.18.6-4 on Debian 5. > >> > > > > Well, these are the config statements for the central rsyslog instance... > > > Does that mean that I am lacking a template statement in the central > rsyslog config? > > Dirk > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu Oct 8 20:31:09 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 8 Oct 2009 11:31:09 -0700 (PDT) Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: <0B1B9163138571439EAF804646F3F6AA087711B7@GIBSVWIN004X.partygaming.local> References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> <9B6E2A8877C38245BFB15CC491A11DA71031A5@GRFEXC.intern.adiscon.com> <0B1B9163138571439EAF804646F3F6AA087711B7@GIBSVWIN004X.partygaming.local> Message-ID: On Thu, 8 Oct 2009, Martin Mielke wrote: > Hi Rainer, > > This is the rsyslog.conf used - quite simple as you can see: > -- > $ cat /etc/rsyslog.conf > # rsyslog v3: load input modules > # If you do not load inputs, nothing happens! > # You may need to set the module load path if modules are not found. > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # provides kernel logging support (previously done by > rklogd) > > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none > -/var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* > -/var/log/maillog > > > # Log cron stuff > cron.* -/var/log/cron > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit > -/var/log/spooler > > # Save boot messages also to boot.log > local7.* > /var/log/boot.log > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/spool/rsyslog # where to place spool files > $ActionQueueFileName uniqName # unique name prefix for spool files > $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > possible) > $ActionQueueSaveOnShutdown on # save messages to disk on shutdown > $ActionQueueType LinkedList # run asynchronously > $ActionResumeRetryCount 5 # five retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > *.* @@logserver > -- > > Do you see anything suspicious? this TCP logging would cause it. you have it limited to 1g of disk space, how large did it get? also, disk queues are pretty slow, it may be that you had messages arriving faster than they could be put into the queue also, you still have it using the default main queue size (something tiny like 1000 messages IIRC) I would bump the main queue size up significantly (assuming you can afford the ram) also, look at the high and low watermark config variables, they (along with one other variable I don't remember offhand) tell the system what to do in crisis mode and how to define that mode. unfortunantly I don't have time until late today to go through the config page and dig up the exact details. David Lang From dag at wieers.com Thu Oct 8 21:10:22 2009 From: dag at wieers.com (Dag Wieers) Date: Thu, 8 Oct 2009 21:10:22 +0200 (CEST) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole Message-ID: Hi, We have been looking at why we don't see messages from netconsole ending up in our logfiles. netconsole is a kernel module that sends information of kernel panics to a syslog server. The documentation indicates it should work with syslog as an endpoint but that is not true with rsyslog, apparently :) https://bugzilla.redhat.com/show_bug.cgi?id=432160 Also, I have noticed that our central rsyslog-server (more than 400 systems log to it) have directories with names like: Detected/, exiting/, ext3_abort/, EXT3-fs/, journal/, last/, martian/, program/, Remounting/, Restarting/, ... So it is obvious that something is not working as expected coming from the kernel. >From information I found online, a message formatted like: <4>Testing through netcat should work for testing the syslog connectivity, but it is clear that rsyslog only accepts: <4>Oct 7 12:34:56 hostname Testing but of course I cannot influence our production kernels to do the right thing. What can I do to have rsyslog accept the "wrong" thing ? :) Advice is appreciated :) -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] From aoz.syn at gmail.com Thu Oct 8 21:28:38 2009 From: aoz.syn at gmail.com (RB) Date: Thu, 8 Oct 2009 13:28:38 -0600 Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: References: Message-ID: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> On Thu, Oct 8, 2009 at 13:10, Dag Wieers wrote: > We have been looking at why we don't see messages from netconsole ending > up in our logfiles. netconsole is a kernel module that sends information > of kernel panics to a syslog server. The documentation indicates it should > work with syslog as an endpoint but that is not true with rsyslog, > apparently :) Mostly because said syslog daemons are not only very loose in their interpretation of a valid syslog entry, but in their default configurations they also tend to silently interpolate data sloppy senders leave out. > Also, I have noticed that our central rsyslog-server (more than 400 > systems log to it) have directories with names like: Detected/, exiting/, > ext3_abort/, EXT3-fs/, journal/, last/, martian/, program/, Remounting/, > Restarting/, ... So it is obvious that something is not working as > expected coming from the kernel. More than likely because you're trying to make directories based on the %hostname% property, which rsyslog assumes is a specific field. With the sloppier daemons (FreeBSD in particular), I've had far more luck using the %fromhost-ip% property (as well as the $ system properties for timestamps). Of course, that breaks down if you're doing relaying, but relying on values the other end sends you to create filesystem artifacts is dangerous at best anyway > but of course I cannot influence our production kernels to do the right > thing. What can I do to have rsyslog accept the "wrong" thing ? :) Use %fromhost% or %fromhost-ip% to make the directory structures/filenames, and make a custom format if you need to handle the remaning lack of data (again, timestamp & host). I'm sure there are many other ways to approach it, but that's the way I've solved it. From mrdemeanour at jackpot.uk.net Fri Oct 9 13:59:52 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 09 Oct 2009 12:59:52 +0100 Subject: [rsyslog] Deviant PRI values causing dropped messages Message-ID: <4ACF25B8.3040002@jackpot.uk.net> Hi, I have a Speedtouch router that produces firewall events whose PRI value is <84> (auth.notice, according to RFC3164). I have a script that retrieves events from the router via Telnet, and passes them on to rsyslog via UDP. The firewall events with PRI=<84> are dropped. If I modify the script so that they are submitted to rsyslog with a PRI of <36> (facility changed from 10 to 4), they are logged. They are also logged if the facility is 13 ("audit"?). According to RFC3614, facilities numbers 4 and 10 are both "auth". I haven't inspected the source, but I suspect that rsyslog doesn't like facility number 10, and is ignoring it. Is this a correct interpretation? Why should it object to this facility in particular? Regards, -- Jack. From mrdemeanour at jackpot.uk.net Fri Oct 9 14:08:38 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 09 Oct 2009 13:08:38 +0100 Subject: [rsyslog] Deviant PRI values causing dropped messages In-Reply-To: <4ACF25B8.3040002@jackpot.uk.net> References: <4ACF25B8.3040002@jackpot.uk.net> Message-ID: <4ACF27C6.5080705@jackpot.uk.net> Mr. Demeanour wrote: > Hi, > > I have a Speedtouch router that produces firewall events whose PRI value > is <84> (auth.notice, according to RFC3164). Sorry; that should be "auth.warn". -- Jack. From rgerhards at hq.adiscon.com Fri Oct 9 14:10:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 9 Oct 2009 14:10:27 +0200 Subject: [rsyslog] Deviant PRI values causing dropped messages References: <4ACF25B8.3040002@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031BB@GRFEXC.intern.adiscon.com> Hi Jack, rsyslog does not do any processing per se on the message. It only does what you instruct it to do. So there is probably something in your rsyslog.conf that does either not process messages with this facility or drops them. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Friday, October 09, 2009 2:00 PM > To: rsyslog-users > Subject: [rsyslog] Deviant PRI values causing dropped messages > > Hi, > > I have a Speedtouch router that produces firewall events whose PRI > value > is <84> (auth.notice, according to RFC3164). I have a script that > retrieves events from the router via Telnet, and passes them on to > rsyslog via UDP. > > The firewall events with PRI=<84> are dropped. If I modify the script > so > that they are submitted to rsyslog with a PRI of <36> (facility changed > from 10 to 4), they are logged. They are also logged if the facility is > 13 ("audit"?). > > According to RFC3614, facilities numbers 4 and 10 are both "auth". I > haven't inspected the source, but I suspect that rsyslog doesn't like > facility number 10, and is ignoring it. Is this a correct > interpretation? Why should it object to this facility in particular? From mrdemeanour at jackpot.uk.net Fri Oct 9 14:15:21 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 09 Oct 2009 13:15:21 +0100 Subject: [rsyslog] Deviant PRI values causing dropped messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031BB@GRFEXC.intern.adiscon.com> References: <4ACF25B8.3040002@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71031BB@GRFEXC.intern.adiscon.com> Message-ID: <4ACF2959.2020707@jackpot.uk.net> Rainer Gerhards wrote: > Hi Jack, > > rsyslog does not do any processing per se on the message. It only > does what you instruct it to do. So there is probably something in > your rsyslog.conf that does either not process messages with this > facility or drops them. Oh, thanks, Rainer. I had this entry in the config: authpriv.info ~ Commenting it out fixed the problem. -- Jack. From tbergfeld at hq.adiscon.com Fri Oct 9 14:34:49 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 9 Oct 2009 14:34:49 +0200 Subject: [rsyslog] rsyslog 4.4.2 (v4-stable) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031BE@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.4.2, a member of the v4-stable branch. This is a bug-fixing release containing some important fixes. It is a recommended update for all users of the v4-stabled. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article409.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-179.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From dag at wieers.com Fri Oct 9 14:38:01 2009 From: dag at wieers.com (Dag Wieers) Date: Fri, 9 Oct 2009 14:38:01 +0200 (CEST) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> Message-ID: On Thu, 8 Oct 2009, RB wrote: > On Thu, Oct 8, 2009 at 13:10, Dag Wieers wrote: > >> Also, I have noticed that our central rsyslog-server (more than 400 >> systems log to it) have directories with names like: Detected/, exiting/, >> ext3_abort/, EXT3-fs/, journal/, last/, martian/, program/, Remounting/, >> Restarting/, ... So it is obvious that something is not working as >> expected coming from the kernel. > > More than likely because you're trying to make directories based on > the %hostname% property, which rsyslog assumes is a specific field. > With the sloppier daemons (FreeBSD in particular), I've had far more > luck using the %fromhost-ip% property (as well as the $ system > properties for timestamps). Of course, that breaks down if you're > doing relaying, but relying on values the other end sends you to > create filesystem artifacts is dangerous at best anyway You are correct, that is exactly what we do. However with rsyslog v2.0.6 it seems there is no %FROMHOST-IP% and the %FROMHOST% property only contains the IP address. Maybe there is something else I need to do to get the short hostname from DNS, rather than an IP on rsyslog v2 ? >> but of course I cannot influence our production kernels to do the right >> thing. What can I do to have rsyslog accept the "wrong" thing ? :) > > Use %fromhost% or %fromhost-ip% to make the directory > structures/filenames, and make a custom format if you need to handle > the remaning lack of data (again, timestamp & host). I'm sure there > are many other ways to approach it, but that's the way I've solved it. Thanks for the feedback. I hope more people can chime into this. -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] From rgerhards at hq.adiscon.com Fri Oct 9 14:47:39 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 9 Oct 2009 14:47:39 +0200 Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031BF@GRFEXC.intern.adiscon.com> Unfortunately, 2.0.6 is a very old and deprecated version. I really don't dig any longer into that code base (except, of course, for folks with support contracts). It really is not worth the hassle. You'll run into a couple of issues in the long term. I suggest upgrading at least to v3-stable. Project status is here: http://www.rsyslog.com/doc-status.html Sorry I have no better reply, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dag Wieers > Sent: Friday, October 09, 2009 2:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] Wrongly formatted messages from > kernel/netconsole > > On Thu, 8 Oct 2009, RB wrote: > > > On Thu, Oct 8, 2009 at 13:10, Dag Wieers wrote: > > > >> Also, I have noticed that our central rsyslog-server (more than 400 > >> systems log to it) have directories with names like: Detected/, > exiting/, > >> ext3_abort/, EXT3-fs/, journal/, last/, martian/, program/, > Remounting/, > >> Restarting/, ... So it is obvious that something is not working as > >> expected coming from the kernel. > > > > More than likely because you're trying to make directories based on > > the %hostname% property, which rsyslog assumes is a specific field. > > With the sloppier daemons (FreeBSD in particular), I've had far more > > luck using the %fromhost-ip% property (as well as the $ system > > properties for timestamps). Of course, that breaks down if you're > > doing relaying, but relying on values the other end sends you to > > create filesystem artifacts is dangerous at best anyway > > You are correct, that is exactly what we do. However with rsyslog > v2.0.6 > it seems there is no %FROMHOST-IP% and the %FROMHOST% property only > contains the IP address. Maybe there is something else I need to do to > get > the short hostname from DNS, rather than an IP on rsyslog v2 ? > > > >> but of course I cannot influence our production kernels to do the > right > >> thing. What can I do to have rsyslog accept the "wrong" thing ? :) > > > > Use %fromhost% or %fromhost-ip% to make the directory > > structures/filenames, and make a custom format if you need to handle > > the remaning lack of data (again, timestamp & host). I'm sure there > > are many other ways to approach it, but that's the way I've solved > it. > > Thanks for the feedback. I hope more people can chime into this. > > -- > -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- > [Any errors in spelling, tact or fact are transmission errors] > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From dag at wieers.com Fri Oct 9 16:17:27 2009 From: dag at wieers.com (Dag Wieers) Date: Fri, 9 Oct 2009 16:17:27 +0200 (CEST) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031BF@GRFEXC.intern.adiscon.com> References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71031BF@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 9 Oct 2009, Rainer Gerhards wrote: > Unfortunately, 2.0.6 is a very old and deprecated version. I really don't dig > any longer into that code base (except, of course, for folks with support > contracts). It really is not worth the hassle. You'll run into a couple of > issues in the long term. I suggest upgrading at least to v3-stable. Project > status is here: > > http://www.rsyslog.com/doc-status.html > > Sorry I have no better reply, Don't feel sorry, I understand. We are looking with Red Hat to rebase rsyslog to a newer version asap. We expect by RHEL5.5 to have either v3 or v4. No word yet from Red Hat what their target will be for RHEL5.5. (Hopefully the same as RHEL6) This is related to the bugzilla reports I send to the list recently. But it would have been nice to know whether there was a solution for this on v2 as well, even when I accept that it is old and unsupported here :-) Kind regards, -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] From aoz.syn at gmail.com Fri Oct 9 16:19:30 2009 From: aoz.syn at gmail.com (RB) Date: Fri, 9 Oct 2009 08:19:30 -0600 Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> Message-ID: <4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> On Fri, Oct 9, 2009 at 06:38, Dag Wieers wrote: > You are correct, that is exactly what we do. However with rsyslog v2.0.6 > it seems there is no %FROMHOST-IP% and the %FROMHOST% property only > contains the IP address. Maybe there is something else I need to do to get > the short hostname from DNS, rather than an IP on rsyslog v2 ? This usually has more to do with the host's DNS configuration than the syslog daemon - I've not looked at the 2.x code (ever), but if your host is configured with a resolver that will perform reverse lookups of the IPs and with an appropriate domain search list, you should be able to get the short hostnames. That can be shortcut by placing entries in /etc/hosts, but that obviously doesn't scale very well. From dag at wieers.com Fri Oct 9 16:36:23 2009 From: dag at wieers.com (Dag Wieers) Date: Fri, 9 Oct 2009 16:36:23 +0200 (CEST) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: <4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> <4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> Message-ID: On Fri, 9 Oct 2009, RB wrote: > On Fri, Oct 9, 2009 at 06:38, Dag Wieers wrote: >> You are correct, that is exactly what we do. However with rsyslog v2.0.6 >> it seems there is no %FROMHOST-IP% and the %FROMHOST% property only >> contains the IP address. Maybe there is something else I need to do to get >> the short hostname from DNS, rather than an IP on rsyslog v2 ? > > This usually has more to do with the host's DNS configuration than the > syslog daemon - I've not looked at the 2.x code (ever), but if your > host is configured with a resolver that will perform reverse lookups > of the IPs and with an appropriate domain search list, you should be > able to get the short hostnames. That can be shortcut by placing > entries in /etc/hosts, but that obviously doesn't scale very well. We do have a working DNS resolver and reverse lookups work fine. I assume that the problem is that with rsyslog v2 %FROMHOST% simply is not resolved. To be sure, I will have to dig into the sources. -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] From david at lang.hm Sat Oct 10 08:06:17 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 9 Oct 2009 23:06:17 -0700 (PDT) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: <4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> <4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> Message-ID: On Fri, 9 Oct 2009, RB wrote: > On Fri, Oct 9, 2009 at 06:38, Dag Wieers wrote: >> You are correct, that is exactly what we do. However with rsyslog v2.0.6 >> it seems there is no %FROMHOST-IP% and the %FROMHOST% property only >> contains the IP address. Maybe there is something else I need to do to get >> the short hostname from DNS, rather than an IP on rsyslog v2 ? > > This usually has more to do with the host's DNS configuration than the > syslog daemon - I've not looked at the 2.x code (ever), but if your > host is configured with a resolver that will perform reverse lookups > of the IPs and with an appropriate domain search list, you should be > able to get the short hostnames. That can be shortcut by placing > entries in /etc/hosts, but that obviously doesn't scale very well. even the 4.x and 5.x do not populate the fromhost-ip if you have -x on the command line. I think this is a bug, but there are bigger bugs that are being worked on. David Lang From rgerhards at hq.adiscon.com Mon Oct 12 16:01:23 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 12 Oct 2009 16:01:23 +0200 Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com><4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031DA@GRFEXC.intern.adiscon.com> David, All, > even the 4.x and 5.x do not populate the fromhost-ip if you have -x on > the > command line. > > I think this is a bug, but there are bigger bugs that are being worked > on. Does anybody have a debug log where this happens? I do not see it in my lab (but I did not try hard). For the DNS cache, I am shuffling code in v5, and that would be a good time to address the bug. Raienr From marc.schiffbauer at mightycare.de Tue Oct 13 13:20:45 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 13 Oct 2009 13:20:45 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? Message-ID: <200910131320.45629.marc.schiffbauer@mightycare.de> Hello all, I am running a rsyslog server version 4.5.2 I tested the message queuing capabilities recently and noticed that queue files do not get cleaned up. Evertime rsyslog has to spool something (e.g. when the database server ist stopped) it creates new spoolfiles so the spool area will grow over time. Is this a bug or how can I handle those old queue files? This is the relevant part from my config: # enable message spooling $WorkDirectory /data/syslog-spool # default location for work (spool) files $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName dbq # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure TIA -Marc Schiffbauer -- Senior Consultant :: Solution Architect IT-Security :: Free Software :: GNU/Linux MightyCare Solutions GmbH http://www.mightycare.de Firmenangaben unter http://www.mightycare.de/#/de/impressum/ From rgerhards at hq.adiscon.com Tue Oct 13 14:16:47 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 13 Oct 2009 14:16:47 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com> This looks like a dirty shutdown. In general, rsyslog cleans up when processing completes. Do you have a .qi file inside that directory? Absence of a .qi file - when the instance is not running - is indication of an issue during shutdown. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, October 13, 2009 1:21 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Hello all, > > I am running a rsyslog server version 4.5.2 > > I tested the message queuing capabilities recently and noticed that > queue > files do not get cleaned up. Evertime rsyslog has to spool something > (e.g. > when the database server ist stopped) it creates new spoolfiles so the > spool > area will grow over time. > > Is this a bug or how can I handle those old queue files? > > This is the relevant part from my config: > > # enable message spooling > $WorkDirectory /data/syslog-spool # default location for work (spool) > files > $ActionQueueType LinkedList # use asynchronous processing > $ActionQueueFileName dbq # set file name, also enables disk > mode > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > TIA > -Marc Schiffbauer > -- > Senior Consultant :: Solution Architect > IT-Security :: Free Software :: GNU/Linux > MightyCare Solutions GmbH http://www.mightycare.de > Firmenangaben unter http://www.mightycare.de/#/de/impressum/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Oct 13 15:31:44 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 13 Oct 2009 15:31:44 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com> Message-ID: <200910131531.44817.marc.schiffbauer@mightycare.de> Hello Rainer, Am Dienstag, 13. Oktober 2009 14:16:47 schrieb Rainer Gerhards: > This looks like a dirty shutdown. I am not aware of any... > In general, rsyslog cleans up when > processing completes. OK. This is good to know. So something went wrong here: I produced a lot of logging traffic using the logger shell command. Then I stopped that postgres server to test that spooling works. After a while I started it again. After a while rsyslog reconnected to the db and logging to the DB was working again. > Do you have a .qi file inside that directory? Absence > of a .qi file - when the instance is not running - is indication of an > issue during shutdown. There is a .qi file when rsyslog is NOT running and ist being removed when I start it again. I will remove stop rsyslog now, then remove *everything* from the spool- directory and start it up again. Then I will repeat the spool-test. Thanks for helping -Marc > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > > Sent: Tuesday, October 13, 2009 1:21 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > > Hello all, > > > > I am running a rsyslog server version 4.5.2 > > > > I tested the message queuing capabilities recently and noticed that > > queue > > files do not get cleaned up. Evertime rsyslog has to spool something > > (e.g. > > when the database server ist stopped) it creates new spoolfiles so the > > spool > > area will grow over time. > > > > Is this a bug or how can I handle those old queue files? > > > > This is the relevant part from my config: > > > > # enable message spooling > > $WorkDirectory /data/syslog-spool # default location for work (spool) > > files > > $ActionQueueType LinkedList # use asynchronous processing > > $ActionQueueFileName dbq # set file name, also enables disk > > mode > > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > > > > TIA > > -Marc Schiffbauer -- Senior Consultant :: Solution Architect IT-Security :: Free Software :: GNU/Linux Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 MightyCare Solutions GmbH http://www.mightycare.de Firmenangaben unter http://www.mightycare.de/#/de/impressum/ From marc.schiffbauer at mightycare.de Tue Oct 13 15:37:52 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 13 Oct 2009 15:37:52 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <200910131531.44817.marc.schiffbauer@mightycare.de> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com> <200910131531.44817.marc.schiffbauer@mightycare.de> Message-ID: <200910131537.52917.marc.schiffbauer@mightycare.de> Am Dienstag, 13. Oktober 2009 15:31:44 schrieb Marc Schiffbauer: > > Do you have a .qi file inside that directory? Absence > > of a .qi file - when the instance is not running - is indication of an > > issue during shutdown. > > There is a .qi file when rsyslog is NOT running and ist being removed when > I start it again. > Now this is strange: To prepare the new test I stopped rsyslog again, and now it did NOT create the .qi file. Another restart: same result. This rsyslog is idle ATM BTW. -Marc -- Senior Consultant :: Solution Architect IT-Security :: Free Software :: GNU/Linux Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 MightyCare Solutions GmbH http://www.mightycare.de Firmenangaben unter http://www.mightycare.de/#/de/impressum/ From rgerhards at hq.adiscon.com Tue Oct 13 15:40:07 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 13 Oct 2009 15:40:07 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com><200910131531.44817.marc.schiffbauer@mightycare.de> <200910131537.52917.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031E6@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, October 13, 2009 3:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Am Dienstag, 13. Oktober 2009 15:31:44 schrieb Marc Schiffbauer: > > > Do you have a .qi file inside that directory? Absence > > > of a .qi file - when the instance is not running - is indication of > an > > > issue during shutdown. > > > > There is a .qi file when rsyslog is NOT running and ist being removed > when > > I start it again. > > > > Now this is strange: To prepare the new test I stopped rsyslog again, > and now > it did NOT create the .qi file. Another restart: same result. > > This rsyslog is idle ATM BTW. Both sounds like there simply was nothing in the queue... > > -Marc > -- > Senior Consultant :: Solution Architect > IT-Security :: Free Software :: GNU/Linux > Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 > MightyCare Solutions GmbH http://www.mightycare.de > Firmenangaben unter http://www.mightycare.de/#/de/impressum/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Oct 13 15:41:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 13 Oct 2009 15:41:17 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com> <200910131531.44817.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031E7@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, October 13, 2009 3:32 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > Am Dienstag, 13. Oktober 2009 14:16:47 schrieb Rainer Gerhards: > > This looks like a dirty shutdown. > > I am not aware of any... > > > In general, rsyslog cleans up when > > processing completes. > > OK. This is good to know. So something went wrong here: I produced a > lot of > logging traffic using the logger shell command. Then I stopped that > postgres > server to test that spooling works. After a while I started it again. > After a while rsyslog reconnected to the db and logging to the DB was > working > again. > > > > Do you have a .qi file inside that directory? Absence > > of a .qi file - when the instance is not running - is indication of > an > > issue during shutdown. > > There is a .qi file when rsyslog is NOT running and ist being removed > when I > start it again. That is OK, because then rsyslog keeps the information in memory once it runs, to safe i/o time. You can modify that by specifying a different queue configuration, but you need to understand how the queue system works in order to do so. A good read is the queue doc. Rainer > > I will remove stop rsyslog now, then remove *everything* from the > spool- > directory and start it up again. > > Then I will repeat the spool-test. > > Thanks for helping > -Marc > > > > > Rainer > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > > > Sent: Tuesday, October 13, 2009 1:21 PM > > > To: rsyslog at lists.adiscon.com > > > Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > > > > Hello all, > > > > > > I am running a rsyslog server version 4.5.2 > > > > > > I tested the message queuing capabilities recently and noticed that > > > queue > > > files do not get cleaned up. Evertime rsyslog has to spool > something > > > (e.g. > > > when the database server ist stopped) it creates new spoolfiles so > the > > > spool > > > area will grow over time. > > > > > > Is this a bug or how can I handle those old queue files? > > > > > > This is the relevant part from my config: > > > > > > # enable message spooling > > > $WorkDirectory /data/syslog-spool # default location for work > (spool) > > > files > > > $ActionQueueType LinkedList # use asynchronous processing > > > $ActionQueueFileName dbq # set file name, also enables > disk > > > mode > > > $ActionResumeRetryCount -1 # infinite retries on insert > failure > > > > > > > > > TIA > > > -Marc Schiffbauer > > -- > Senior Consultant :: Solution Architect > IT-Security :: Free Software :: GNU/Linux > Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 > MightyCare Solutions GmbH http://www.mightycare.de > Firmenangaben unter http://www.mightycare.de/#/de/impressum/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Oct 13 16:27:44 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 13 Oct 2009 16:27:44 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031E7@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <200910131531.44817.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA71031E7@GRFEXC.intern.adiscon.com> Message-ID: <200910131627.44897.marc.schiffbauer@mightycare.de> Am Dienstag, 13. Oktober 2009 15:41:17 schrieb Rainer Gerhards: > > > > There is a .qi file when rsyslog is NOT running and ist being removed > > when I > > start it again. > > That is OK, because then rsyslog keeps the information in memory once it > runs, to safe i/o time. You can modify that by specifying a different queue > configuration, but you need to understand how the queue system works in > order to do so. A good read is the queue doc. > > Rainer > > > I will remove stop rsyslog now, then remove *everything* from the > > spool- > > directory and start it up again. > > > > Then I will repeat the spool-test. > > As it seems, rsyslog will not write a .qi file in all cases. New tests were not all successful (with rsyslog under load): OK => spooling while DB is offline OK => reconnect to DB OK => despooling while still under load and spooling to disk Now the following produced "stale" queue files and a loss of messages I guess: NOT OK => despooling while under load and while spooling to disk, then stopping rsyslogd (stopped via /etc/init.d/syslog stop) -> no .qi file has been created! after making sure there are no more rsyslog processes I started it again. The spool files will not be cleared (no load anymore and DB started of course) bug? TIA -Marc -- Senior Consultant :: Solution Architect IT-Security :: Free Software :: GNU/Linux Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 MightyCare Solutions GmbH http://www.mightycare.de Firmenangaben unter http://www.mightycare.de/#/de/impressum/ From rgerhards at hq.adiscon.com Tue Oct 13 16:35:49 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 13 Oct 2009 16:35:49 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><200910131531.44817.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA71031E7@GRFEXC.intern.adiscon.com> <200910131627.44897.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031EA@GRFEXC.intern.adiscon.com> > As it seems, rsyslog will not write a .qi file in all cases. not always, but it should always write them when necessary ;) > > New tests were not all successful (with rsyslog under load): > OK => spooling while DB is offline > OK => reconnect to DB > OK => despooling while still under load and spooling to disk > > Now the following produced "stale" queue files and a loss of messages I > guess: > NOT OK => despooling while under load and while spooling to disk, > then stopping rsyslogd > (stopped via /etc/init.d/syslog stop) > -> no .qi file has been created! > after making sure there are no more rsyslog processes I started it > again. > The spool files will not be cleared (no load anymore and DB started of > course) > > bug? Smells like one. I re-checked your config, I think it does not include a directive to tell the engine to persist messages on shutdown. Even if it does not do that, it should clean up the files. A debug log would be most useful. Note that the v4 engine, and more so the v5 engine, has had a number of important changes, and people only gradually begin to utilize it in practice. The past couple of month, I had comparatively few bug reports, but the past three weeks or so people tend to adopt the new features and consequently the "bug rate" goes up. This is good, as it helps iron out things, but it is also somewhat bad, because I need to prioritize work and for some bugs that means I have not to touch too many things at once. Looking forward for additional information. Rainer PS: please all keep contributing bug reports! It is really useful to have them (even better if they are timely ;)) as my lab can not cover everything practice does ;) > > TIA > -Marc > -- > Senior Consultant :: Solution Architect > IT-Security :: Free Software :: GNU/Linux > Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 > MightyCare Solutions GmbH http://www.mightycare.de > Firmenangaben unter http://www.mightycare.de/#/de/impressum/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Oct 14 00:40:14 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 13 Oct 2009 15:40:14 -0700 (PDT) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031DA@GRFEXC.intern.adiscon.com> References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com><4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71031DA@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 12 Oct 2009, Rainer Gerhards wrote: > David, All, > >> even the 4.x and 5.x do not populate the fromhost-ip if you have -x on >> the >> command line. >> >> I think this is a bug, but there are bigger bugs that are being worked >> on. > > Does anybody have a debug log where this happens? I do not see it in my lab > (but I did not try hard). For the DNS cache, I am shuffling code in v5, and > that would be a good time to address the bug. you should have it in the debug logs I've made available for the segfault problems. Those instances are running with -x David Lang From rgerhards at hq.adiscon.com Wed Oct 14 07:14:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 14 Oct 2009 07:14:44 +0200 Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com><4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71031DA@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031F1@GRFEXC.intern.adiscon.com> thanks, then I know where to look at :) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, October 14, 2009 12:40 AM > To: rsyslog-users > Subject: Re: [rsyslog] Wrongly formatted messages from > kernel/netconsole > > On Mon, 12 Oct 2009, Rainer Gerhards wrote: > > > David, All, > > > >> even the 4.x and 5.x do not populate the fromhost-ip if you have -x > on > >> the > >> command line. > >> > >> I think this is a bug, but there are bigger bugs that are being > worked > >> on. > > > > Does anybody have a debug log where this happens? I do not see it in > my lab > > (but I did not try hard). For the DNS cache, I am shuffling code in > v5, and > > that would be a good time to address the bug. > > you should have it in the debug logs I've made available for the > segfault > problems. Those instances are running with -x > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From tbergfeld at hq.adiscon.com Thu Oct 15 10:59:17 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Thu, 15 Oct 2009 10:59:17 +0200 Subject: [rsyslog] rsyslog 5.1.6 (v5-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103210@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.1.6, a member of the v5-beta branch. This is a bug-fixing release containing some important fixes. It is a recommended update for all users of the v5-beta. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article413.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-180.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From marc.schiffbauer at mightycare.de Thu Oct 15 14:14:33 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Thu, 15 Oct 2009 14:14:33 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? Message-ID: <200910151414.33860.marc.schiffbauer@mightycare.de> [ sorry, attachement in first try was too big, next try with shortened debug log ] Am Dienstag, 13. Oktober 2009 16:35:49 schrieb Rainer Gerhards: > > As it seems, rsyslog will not write a .qi file in all cases. > > not always, but it should always write them when necessary ;) > > > New tests were not all successful (with rsyslog under load): > > OK => spooling while DB is offline > > OK => reconnect to DB > > OK => despooling while still under load and spooling to disk > > > > Now the following produced "stale" queue files and a loss of messages I > > guess: > > NOT OK => despooling while under load and while spooling to disk, > > then stopping rsyslogd > > (stopped via /etc/init.d/syslog stop) > > -> no .qi file has been created! > > after making sure there are no more rsyslog processes I started it > > again. > > The spool files will not be cleared (no load anymore and DB started of > > course) > > > > bug? > > Smells like one. I re-checked your config, I think it does not include a > directive to tell the engine to persist messages on shutdown. Oh, I did not know that there is a directive for it.. good to know. > Even if it > does not do that, it should clean up the files. A debug log would be most > useful. Trying to produce one it seems a difficult task to me because rsyslogd seems to behave completely different when in debug mode... In one console I started: /sbin/rsyslogd -c 4 -f /etc/rsyslog.conf -d &> rsyslog-debug.log Then I stopped postgres: /etc/init.d/postgresql stop Now I started a logger loop: while true; do logger -t spool-test "no real message here... (PID=$$)"; done After the first spool file is being created I stop the loop again. Now I want to stop rsyslogd by calling "killall rsyslogd" I had to call this several times before rsyslogd really did exit. After the first attemots it seemed that it did try to reach the DB in a loop and did not attempt to prepare for exit. See the log attached. Is this helpful? -Marc > > Note that the v4 engine, and more so the v5 engine, has had a number of > important changes, and people only gradually begin to utilize it in > practice. The past couple of month, I had comparatively few bug reports, > but the past three weeks or so people tend to adopt the new features and > consequently the "bug rate" goes up. This is good, as it helps iron out > things, but it is also somewhat bad, because I need to prioritize work and > for some bugs that means I have not to touch too many things at once. > > Looking forward for additional information. > > Rainer > PS: please all keep contributing bug reports! It is really useful to have > them (even better if they are timely ;)) as my lab can not cover everything > practice does ;) > -- Senior Consultant :: Solution Architect IT-Security :: Free Software :: GNU/Linux From rgerhards at hq.adiscon.com Thu Oct 15 14:21:42 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 15 Oct 2009 14:21:42 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910151414.33860.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710321F@GRFEXC.intern.adiscon.com> Hi Marc, the mailing list is set to be very restrictive in what it accepts. So the attachment did not went through ;) You can mail it to me at rgerhards at gmail.com - but please let me know when you have done, I do not regularly check that mailbox. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Thursday, October 15, 2009 2:15 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > [ sorry, attachement in first try was too big, next try with shortened > debug > log ] > > Am Dienstag, 13. Oktober 2009 16:35:49 schrieb Rainer Gerhards: > > > As it seems, rsyslog will not write a .qi file in all cases. > > > > not always, but it should always write them when necessary ;) > > > > > New tests were not all successful (with rsyslog under load): > > > OK => spooling while DB is offline > > > OK => reconnect to DB > > > OK => despooling while still under load and spooling to disk > > > > > > Now the following produced "stale" queue files and a loss of > messages I > > > guess: > > > NOT OK => despooling while under load and while spooling to disk, > > > then stopping rsyslogd > > > (stopped via /etc/init.d/syslog stop) > > > -> no .qi file has been created! > > > after making sure there are no more rsyslog processes I started it > > > again. > > > The spool files will not be cleared (no load anymore and DB started > of > > > course) > > > > > > bug? > > > > Smells like one. I re-checked your config, I think it does not > include a > > directive to tell the engine to persist messages on shutdown. > > Oh, I did not know that there is a directive for it.. good to know. > > > Even if it > > does not do that, it should clean up the files. A debug log would be > most > > useful. > > Trying to produce one it seems a difficult task to me because rsyslogd > seems > to behave completely different when in debug mode... > > In one console I started: > /sbin/rsyslogd -c 4 -f /etc/rsyslog.conf -d &> rsyslog-debug.log > > Then I stopped postgres: > /etc/init.d/postgresql stop > > Now I started a logger loop: > while true; do logger -t spool-test "no real message here... > (PID=$$)"; > done > > After the first spool file is being created I stop the loop again. > > Now I want to stop rsyslogd by calling "killall rsyslogd" > > I had to call this several times before rsyslogd really did exit. After > the > first attemots it seemed that it did try to reach the DB in a loop and > did not > attempt to prepare for exit. > > See the log attached. Is this helpful? > > -Marc > > > > > Note that the v4 engine, and more so the v5 engine, has had a number > of > > important changes, and people only gradually begin to utilize it in > > practice. The past couple of month, I had comparatively few bug > reports, > > but the past three weeks or so people tend to adopt the new features > and > > consequently the "bug rate" goes up. This is good, as it helps iron > out > > things, but it is also somewhat bad, because I need to prioritize > work and > > for some bugs that means I have not to touch too many things at > once. > > > > Looking forward for additional information. > > > > Rainer > > PS: please all keep contributing bug reports! It is really useful to > have > > them (even better if they are timely ;)) as my lab can not cover > everything > > practice does ;) > > > -- > Senior Consultant :: Solution Architect > IT-Security :: Free Software :: GNU/Linux From marc.schiffbauer at mightycare.de Thu Oct 15 19:33:11 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Thu, 15 Oct 2009 19:33:11 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710321F@GRFEXC.intern.adiscon.com> References: <200910151414.33860.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA710321F@GRFEXC.intern.adiscon.com> Message-ID: <200910151933.11296.marc.schiffbauer@mightycare.de> Am Donnerstag, 15. Oktober 2009 14:21:42 schrieb Rainer Gerhards: > Hi Marc, > > the mailing list is set to be very restrictive in what it accepts. So the > attachment did not went through ;) You can mail it to me at > rgerhards at gmail.com - but please let me know when you have done, I do not > regularly check that mailbox. Hi Rainer, mail sent! TIA -Marc From marc.schiffbauer at mightycare.de Fri Oct 16 12:52:26 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Fri, 16 Oct 2009 12:52:26 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031EA@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <200910131627.44897.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA71031EA@GRFEXC.intern.adiscon.com> Message-ID: <200910161252.26765.marc.schiffbauer@mightycare.de> Hi Rainer, Am Dienstag, 13. Oktober 2009 16:35:49 schrieb Rainer Gerhards: > > > > bug? > > Smells like one. I re-checked your config, I think it does not include a > directive to tell the engine to persist messages on shutdown. Even if it > does not do that, it should clean up the files. A debug log would be most > useful. I now added "$ActionQueueSaveOnShutdown on" to the configuration. But I still get queuefiles without a .qi file after rsyslog has quit by SIGTERM -Marc From rgerhards at hq.adiscon.com Fri Oct 16 14:07:23 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 16 Oct 2009 14:07:23 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><200910131627.44897.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA71031EA@GRFEXC.intern.adiscon.com> <200910161252.26765.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103231@GRFEXC.intern.adiscon.com> busy day today... I had a quick look at the debug log, it looks like there indeed is a shutdown issue. Out of curiosity, would you be willing to test a current (v5) devel version? I am currently working on an enhanced queue engine, which has been greatly simplified and speeded up. During that work, I notice that there exist some potential issue in v4, that I had thought to not surface in practice. Queue shutdown is very, very complicated. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Friday, October 16, 2009 12:52 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > Hi Rainer, > > Am Dienstag, 13. Oktober 2009 16:35:49 schrieb Rainer Gerhards: > > > > > > bug? > > > > Smells like one. I re-checked your config, I think it does not > include a > > directive to tell the engine to persist messages on shutdown. Even if > it > > does not do that, it should clean up the files. A debug log would be > most > > useful. > > I now added "$ActionQueueSaveOnShutdown on" to the configuration. But I > still > get queuefiles without a .qi file after rsyslog has quit by SIGTERM > > -Marc > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Fri Oct 16 15:55:40 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Fri, 16 Oct 2009 15:55:40 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103231@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <200910161252.26765.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103231@GRFEXC.intern.adiscon.com> Message-ID: <200910161555.40812.marc.schiffbauer@mightycare.de> Am Freitag, 16. Oktober 2009 14:07:23 schrieb Rainer Gerhards: > busy day today... I had a quick look at the debug log, it looks like there > indeed is a shutdown issue. > Do you think you can get it fixed in 4.5.4? I would be pleased to test patches... > Out of curiosity, would you be willing to test a current (v5) devel > version? As this is at a customers system I can do only a short test to see whether it works or not. > I am currently working on an enhanced queue engine, which has > been greatly simplified and speeded up. During that work, I notice that > there exist some potential issue in v4, that I had thought to not surface > in practice. Queue shutdown is very, very complicated. > I see. Sounds good for future versions.... Are these DA-queuing features are present in the 3.xx version already? Maybe I should consider using 3.xx the instead if its more robust. Thanks -Marc > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > > Sent: Friday, October 16, 2009 12:52 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > > > > Hi Rainer, > > > > Am Dienstag, 13. Oktober 2009 16:35:49 schrieb Rainer Gerhards: > > > > bug? > > > > > > Smells like one. I re-checked your config, I think it does not > > > > include a > > > > > directive to tell the engine to persist messages on shutdown. Even if > > > > it > > > > > does not do that, it should clean up the files. A debug log would be > > > > most > > > > > useful. > > > > I now added "$ActionQueueSaveOnShutdown on" to the configuration. But I > > still > > get queuefiles without a .qi file after rsyslog has quit by SIGTERM > > > > -Marc > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Oct 16 15:59:18 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 16 Oct 2009 15:59:18 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><200910161252.26765.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA7103231@GRFEXC.intern.adiscon.com> <200910161555.40812.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103234@GRFEXC.intern.adiscon.com> just quickly > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Friday, October 16, 2009 3:56 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Am Freitag, 16. Oktober 2009 14:07:23 schrieb Rainer Gerhards: > > busy day today... I had a quick look at the debug log, it looks like > there > > indeed is a shutdown issue. > > > > Do you think you can get it fixed in 4.5.4? I would be pleased to test > patches... > > > > Out of curiosity, would you be willing to test a current (v5) devel > > version? > > As this is at a customers system I can do only a short test to see > whether it > works or not. > > > > I am currently working on an enhanced queue engine, which has > > been greatly simplified and speeded up. During that work, I notice > that > > there exist some potential issue in v4, that I had thought to not > surface > > in practice. Queue shutdown is very, very complicated. > > > > I see. Sounds good for future versions.... > > > Are these DA-queuing features are present in the 3.xx version already? > Maybe I should consider using 3.xx the instead if its more robust. Indeed, they are. Raienr From marc.schiffbauer at mightycare.de Fri Oct 16 16:20:50 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Fri, 16 Oct 2009 16:20:50 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <200910161555.40812.marc.schiffbauer@mightycare.de> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103231@GRFEXC.intern.adiscon.com> <200910161555.40812.marc.schiffbauer@mightycare.de> Message-ID: <200910161620.50460.marc.schiffbauer@mightycare.de> Am Freitag, 16. Oktober 2009 15:55:40 schrieb Marc Schiffbauer: > Am Freitag, 16. Oktober 2009 14:07:23 schrieb Rainer Gerhards: > > busy day today... I had a quick look at the debug log, it looks like > > there indeed is a shutdown issue. > > Do you think you can get it fixed in 4.5.4? I would be pleased to test > patches... I meant 4.4.3 of course ;-) From marc.schiffbauer at mightycare.de Fri Oct 16 16:32:20 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Fri, 16 Oct 2009 16:32:20 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103234@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <200910161555.40812.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103234@GRFEXC.intern.adiscon.com> Message-ID: <200910161632.20392.marc.schiffbauer@mightycare.de> Am Freitag, 16. Oktober 2009 15:59:18 schrieb Rainer Gerhards: > just quickly > > > > > Are these DA-queuing features are present in the 3.xx version already? > > Maybe I should consider using 3.xx the instead if its more robust. > > Indeed, they are. But 3.22.1 seems to have the same issues with DA-spooling :-( I will do some more testing on monday. -Marc From rgerhards at hq.adiscon.com Fri Oct 16 16:39:16 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 16 Oct 2009 16:39:16 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><200910161555.40812.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA7103234@GRFEXC.intern.adiscon.com> <200910161632.20392.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103235@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Friday, October 16, 2009 4:32 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Am Freitag, 16. Oktober 2009 15:59:18 schrieb Rainer Gerhards: > > just quickly > > > > > > > > Are these DA-queuing features are present in the 3.xx version > already? > > > Maybe I should consider using 3.xx the instead if its more robust. > > > > Indeed, they are. > > But 3.22.1 seems to have the same issues with DA-spooling :-( > I will do some more testing on monday. One problem is that most people didn't use the enhanced queing modes until lately. So even while the versions were out for quite some while, and there were no bug reports, there seem to be some hidden issues. I have been receiving bug reports the past three month or so and I am working to get these things straight. I consider these reports to be a good sign, as it indicates the functionality gradually is adopted in practice. Right now, I am working on a much more streamlined queue engine in v5. Queue shutdown is a very complex operation, and I have been able to greatly simplify processing. Also, there has been a tremendous amount of work gone into performance optimization, where the simplified engine will also set new marks, I think. Of course, that doesn't help you immediately. I just thought I try to convey at least some important bits of "the big picture". Rainer From david at lang.hm Fri Oct 16 22:05:14 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 16 Oct 2009 13:05:14 -0700 (PDT) Subject: [rsyslog] disabling doc installation Message-ID: is there a configure option that will skip populating everything under the DATAROOT directory? while I see the value of having everything there by default, on my server builds I don't want to waste the space of having everything there, so I end up having to create the package, and then manually modify it. David Lang From sovrevage at gmail.com Mon Oct 19 12:46:17 2009 From: sovrevage at gmail.com (=?ISO-8859-1?Q?Stian_=D8vrev=E5ge?=) Date: Mon, 19 Oct 2009 12:46:17 +0200 Subject: [rsyslog] timegenerated and timereported Message-ID: timegenerated - timestamp when the message was RECEIVED. Always in high resolution timereported - timestamp from the message. Resolution depends on what was provided in the message (in most cases, only seconds) Neither of these fit into either a mysql datetime nor unixtime field type! They come in the format "Oct 19 12:32:54", exacly what is high resolution about that I don't know. So, does rsyslog provide a way to easily fit these into a time-aware mysql field type? My template looks like this: $template SyslogAllInsert,"INSERT INTO `Syslog`.`AllMessages` (`ID` ,`timeGenerated` ,`timeReported` ,`source` ,`programName` ,`syslogTag` ,`syslogFacility` ,`syslogSeverity` ,`message`)VALUES ( NULL , '%timegenerated%', '%timereported%', '%source%', '%programname%', '%syslogtag%', '%syslogfacility%', '%syslogseverity%', '%msg%');",sql TIA Cheers, Stian ?vrev?ge From dirk.schulz at kinzesberg.de Mon Oct 19 13:51:15 2009 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Mon, 19 Oct 2009 13:51:15 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG In-Reply-To: <87a1ae540910080951g6c7106fby53979808642cb537@mail.gmail.com> References: <4ACCA646.8050209@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> <4ACE0C3B.7090207@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA71031AC@GRFEXC.intern.adiscon.com> <4ACE11D9.1020806@kinzesberg.de> <87a1ae540910080951g6c7106fby53979808642cb537@mail.gmail.com> Message-ID: <4ADC52B3.4050806@kinzesberg.de> I fould the problem - I have to admit it was me. :-( Had a configuration error (cleverly combined with a research error). It is working fine now. Thanks a lot for you help. Dirk Sayan Chowdhury schrieb: > Hello Dirk, I think Rainer wants you to add this to your rsyslog.conf file, > and send him the file that gets saved because of adding this statement. So > as to debug the issue. > You will need to restart rsyslog to pick up the config change > Hope this helps. > Sayan > > On Thu, Oct 8, 2009 at 9:52 PM, Dirk H. Schulz wrote: > > >> Rainer Gerhards schrieb: >> >>>>> check doc for exact syntax): >>>>> >>>>> $template raw,"%rawmsg%\n" >>>>> *.* /path/to/file;raw >>>>> >>>>> >>>>> >>>> I did not find these terms, neither in rsyslog.conf on the central >>>> logserver servers nor in syslog-ng.conf on the log client servers. >>>> Rsyslog is 3.18.6-4 on Debian 5. >>>> >>>> >>> Well, these are the config statements for the central rsyslog instance... >>> >>> >> Does that mean that I am lacking a template statement in the central >> rsyslog config? >> >> Dirk >> >>> Rainer >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Oct 20 11:32:55 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 11:32:55 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><200910161555.40812.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA7103234@GRFEXC.intern.adiscon.com> <200910161632.20392.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103259@GRFEXC.intern.adiscon.com> Marc, I finally had some time to look into the issue. The problem actually is rooted in the shutdown sequence, which is rather complex in v3/v4. I am right now working on great simplification in v5, which will lead to more robust and even better performing code. Thanks to that work, I think I was able to quickly find the culprit and also to develop a patch. In my lab it works, and in theory it works as well, but practice is always another beast... So I would appreciate if you could give it a try. It is a pre-release of what once will become 4.4.3 and is available at: http://download.rsyslog.com/rsyslog/pre/rsyslog-4.4.3.tar.gz FYI, the actual patch is small: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=8e45fd5c8e7df45532c5d239 676cd92107c9e942 Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Friday, October 16, 2009 4:32 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Am Freitag, 16. Oktober 2009 15:59:18 schrieb Rainer Gerhards: > > just quickly > > > > > > > > Are these DA-queuing features are present in the 3.xx version > already? > > > Maybe I should consider using 3.xx the instead if its more robust. > > > > Indeed, they are. > > But 3.22.1 seems to have the same issues with DA-spooling :-( > I will do some more testing on monday. > > -Marc > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Oct 20 12:50:45 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 20 Oct 2009 12:50:45 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103259@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <200910161632.20392.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103259@GRFEXC.intern.adiscon.com> Message-ID: <200910201250.45820.marc.schiffbauer@mightycare.de> Am Dienstag, 20. Oktober 2009 11:32:55 schrieb Rainer Gerhards: > Marc, Hi Rainer, > > I finally had some time to look into the issue. The problem actually is > rooted in the shutdown sequence, which is rather complex in v3/v4. I am > right now working on great simplification in v5, which will lead to more > robust and even better performing code. > > Thanks to that work, I think I was able to quickly find the culprit and > also to develop a patch. In my lab it works, and in theory it works as > well, but practice is always another beast... > > So I would appreciate if you could give it a try. It is a pre-release of > what once will become 4.4.3 and is available at: > > http://download.rsyslog.com/rsyslog/pre/rsyslog-4.4.3.tar.gz Thanks very much! Now it works as expected with one exception: In my test that I did now I put an ID into each log-message to see if no messages get lost. That way I found that one message always gets lost when I do the following: * DB and rsyslogd started * Start logger loop with ID (slow message rate): X=1; while true; do logger -t spool-test "no real message here... (ID=$X)"; X=$((X+1)); sleep 3; done & * stop DB server, spooling to mem starts * start another logger loop to force DA spooling: while true; do logger -t spool-test "no real message here..."; done & * watch spool dir: as soon as a queue file is being created: stop both loops ("fg" followeg by ctrl-c, two times) * stop rsyslogd: qi is file being created now! now there should be no lost message, right? * start DB server * start rsyslogd * check the log: *one* of those ID=$X messages is lost. I did that test two times. -Marc From rgerhards at hq.adiscon.com Tue Oct 20 13:36:48 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 13:36:48 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? Message-ID: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> Can you mail me a debug log where the message loss can be seen? rainer ----- Urspr?ngliche Nachricht ----- Von: "Marc Schiffbauer" An: "rsyslog-users" Gesendet: 20.10.09 12:51 Betreff: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? Am Dienstag, 20. Oktober 2009 11:32:55 schrieb Rainer Gerhards: > Marc, Hi Rainer, > > I finally had some time to look into the issue. The problem actually is > rooted in the shutdown sequence, which is rather complex in v3/v4. I am > right now working on great simplification in v5, which will lead to more > robust and even better performing code. > > Thanks to that work, I think I was able to quickly find the culprit and > also to develop a patch. In my lab it works, and in theory it works as > well, but practice is always another beast... > > So I would appreciate if you could give it a try. It is a pre-release of > what once will become 4.4.3 and is available at: > > http://download.rsyslog.com/rsyslog/pre/rsyslog-4.4.3.tar.gz Thanks very much! Now it works as expected with one exception: In my test that I did now I put an ID into each log-message to see if no messages get lost. That way I found that one message always gets lost when I do the following: * DB and rsyslogd started * Start logger loop with ID (slow message rate): X=1; while true; do logger -t spool-test "no real message here... (ID=$X)"; X=$((X+1)); sleep 3; done & * stop DB server, spooling to mem starts * start another logger loop to force DA spooling: while true; do logger -t spool-test "no real message here..."; done & * watch spool dir: as soon as a queue file is being created: stop both loops ("fg" followeg by ctrl-c, two times) * stop rsyslogd: qi is file being created now! now there should be no lost message, right? * start DB server * start rsyslogd * check the log: *one* of those ID=$X messages is lost. I did that test two times. -Marc _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Oct 20 14:48:16 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 20 Oct 2009 14:48:16 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> Message-ID: <200910201448.16203.marc.schiffbauer@mightycare.de> Am Dienstag, 20. Oktober 2009 13:36:48 schrieb Rainer Gerhards: > Can you mail me a debug log where the message loss can be seen? Hi Rainer, I sent the logs to your gmail address. The lost message contains "ID=12". TIA -Marc From rgerhards at hq.adiscon.com Tue Oct 20 15:23:31 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 15:23:31 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200910201448.16203.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> interesting... the log looks like all works ok (of course, without the missing message ;)). Can you send me the contents of the .qi file after the shutdown? I am asking because it should contain the missing message (with that version of the queue engine). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, October 20, 2009 2:48 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Am Dienstag, 20. Oktober 2009 13:36:48 schrieb Rainer Gerhards: > > Can you mail me a debug log where the message loss can be seen? > > Hi Rainer, > > I sent the logs to your gmail address. > > The lost message contains "ID=12". > > > TIA > -Marc > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Oct 20 16:12:59 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 20 Oct 2009 16:12:59 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200910201448.16203.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> Message-ID: <200910201612.59316.marc.schiffbauer@mightycare.de> Am Dienstag, 20. Oktober 2009 15:23:31 schrieb Rainer Gerhards: > interesting... the log looks like all works ok (of course, without the > missing message ;)). Can you send me the contents of the .qi file after the > shutdown? I am asking because it should contain the missing message (with > that version of the queue engine). > > Rainer > Hi Rainer, I sent you another gmail. I repeated the test and this time I saved everything from the queue-Directory (one queue file + the dbq.qi file) before restarting rsyslogd The missing log entry in this try: ID=7 TIA -Marc From marc.schiffbauer at mightycare.de Tue Oct 20 16:39:42 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 20 Oct 2009 16:39:42 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200910201448.16203.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> Message-ID: <200910201639.43143.marc.schiffbauer@mightycare.de> Hi Rainer, new mail on gmail, now with attachment ;) TIA -Marc From anichols at trumped.org Tue Oct 20 17:32:07 2009 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 20 Oct 2009 09:32:07 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase in write volume (bytes/sec) to NAS - can someone help shed light? Message-ID: Hello, We have two rsyslog "writer" instances which are receiving syslog messages and writing them to an NFS mount on a NAS. The NAS is connected via 10gb ethernet but has a number of other applications writing to it as well. Typical write volume for our log writers is 2-3Mbps to the NAS. Yesterday I attempted to resolve some complaints with logs being "bursty" - events appear to be written in batches which sounded consistent with buffering on the rsyslog daemon or not syncing the file with every message. These bursts could come as infrequently as every 30 seconds however, which seems excessive. I took a number of steps to remedy this but ultimately had to reverse the changes because write volume (measured at the disks) to the NAS went from 2Mbps up to 9Mbps and iops went from 30 up to 900 at the peak. I'm sure there is an explanation for this - but there are some subtle configuration changes which had an impact and I don't understand exactly why. First, the main change was to set "$ActionFileEnableSync on" in each logging servers configuration. From what I can tell this globally enables the ability to turn on file sync and it defaults to "off". This change, combined with removing the "-" from all the actions in my configuration should have resulted in all files being synced all the time - probably the worst case scenario and I'm not all that suprised this caused problems. Second, before rolling the change back 100% - I removed the "$ActionFileEnableSync on" configuration from the servers but left the action configurations without the "-" - assuming that the global configuration option would completely disable file sync and using a "-" would have no effect. This did not seem to be the case - it did dramatically decrease the volume of traffic to the NAS but still the volume was about double (5Mbps) the normal volume. Third, I added "-" before all the action lines in the configuration and this brought volume back down to the 2-3Mbps we are used to. This leaves me with a few questions: 1) I would expect file sync to increase transactions with the NAS but not increase the volume of data being written to the NAS as measured at the disk. Can someone shed light on why this would so profoundly impact write volume? Load on the rsyslog servers also went down substantially after making this change - presumably because there were fewer queued transactions - not sure. 2) Is there some difference between what "$ActionFileEnableSync on" does vs. adding a "-" before an action other than one is global and one is per-action? I thought these were just two different levels of granularity for configuring the same thing but that doesn't appear to be the case. 3) Is it expected to have rsyslog take up to 30 seconds to flush messages to a logfile? These are BUSY logs - being written to multiple times per second, so it's not a delay on the client. I can observe these same clients logging messages to another logging server (the old server) running syslog-ng and messages are observable in near real-time. Ultimate - #3 is the problem I am trying to solve, but I'm very curious why these changes had the impact they did and why globally disabling file sync using "$ActionFileEnableSync" would not entire decrease log volume until I add "-" to all the actions. Let me know if I can add info that would help shed light on this. Thanks, Aaron From rgerhards at hq.adiscon.com Tue Oct 20 18:32:30 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 18:32:30 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase inwrite volume (bytes/sec) to NAS - can someone help shed light? References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> Hi Aaron, first and very important question: which version of rsyslog are you using? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Tuesday, October 20, 2009 5:32 PM > To: rsyslog-users > Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive > increase inwrite volume (bytes/sec) to NAS - can someone help shed > light? > > Hello, > We have two rsyslog "writer" instances which are receiving syslog > messages and writing them to an NFS mount on a NAS. The NAS is > connected via > 10gb ethernet but has a number of other applications writing to it as > well. > Typical write volume for our log writers is 2-3Mbps to the NAS. > Yesterday I > attempted to resolve some complaints with logs being "bursty" - events > appear to be written in batches which sounded consistent with buffering > on > the rsyslog daemon or not syncing the file with every message. These > bursts > could come as infrequently as every 30 seconds however, which seems > excessive. I took a number of steps to remedy this but ultimately had > to > reverse the changes because write volume (measured at the disks) to the > NAS > went from 2Mbps up to 9Mbps and iops went from 30 up to 900 at the > peak. I'm > sure there is an explanation for this - but there are some subtle > configuration changes which had an impact and I don't understand > exactly > why. > > First, the main change was to set "$ActionFileEnableSync on" in each > logging > servers configuration. From what I can tell this globally enables the > ability to turn on file sync and it defaults to "off". This change, > combined > with removing the "-" from all the actions in my configuration should > have > resulted in all files being synced all the time - probably the worst > case > scenario and I'm not all that suprised this caused problems. > > Second, before rolling the change back 100% - I removed the > "$ActionFileEnableSync on" configuration from the servers but left the > action configurations without the "-" - assuming that the global > configuration option would completely disable file sync and using a "-" > would have no effect. This did not seem to be the case - it did > dramatically > decrease the volume of traffic to the NAS but still the volume was > about > double (5Mbps) the normal volume. > > Third, I added "-" before all the action lines in the configuration and > this > brought volume back down to the 2-3Mbps we are used to. > > This leaves me with a few questions: > > 1) I would expect file sync to increase transactions with the NAS but > not > increase the volume of data being written to the NAS as measured at the > disk. Can someone shed light on why this would so profoundly impact > write > volume? Load on the rsyslog servers also went down substantially after > making this change - presumably because there were fewer queued > transactions > - not sure. > > 2) Is there some difference between what "$ActionFileEnableSync on" > does vs. > adding a "-" before an action other than one is global and one is > per-action? I thought these were just two different levels of > granularity > for configuring the same thing but that doesn't appear to be the case. > > 3) Is it expected to have rsyslog take up to 30 seconds to flush > messages to > a logfile? These are BUSY logs - being written to multiple times per > second, > so it's not a delay on the client. I can observe these same clients > logging > messages to another logging server (the old server) running syslog-ng > and > messages are observable in near real-time. > > Ultimate - #3 is the problem I am trying to solve, but I'm very curious > why > these changes had the impact they did and why globally disabling file > sync > using "$ActionFileEnableSync" would not entire decrease log volume > until I > add "-" to all the actions. > > Let me know if I can add info that would help shed light on this. > > Thanks, > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Oct 20 18:38:16 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 18:38:16 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com><200910201448.16203.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> <200910201639.43143.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> thanks! Interesting, I see that only one of the messages that should be in the .qi are actually present. I wonder if this is related to the fact that ompgsql emits an error message itself while being down (the others don't do this AFIK). Will try to dig down to this (but I have to do so as time permits). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, October 20, 2009 4:40 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > Hi Rainer, > > new mail on gmail, now with attachment ;) > > TIA > -Marc > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From anichols at trumped.org Tue Oct 20 18:54:02 2009 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 20 Oct 2009 10:54:02 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase inwrite volume (bytes/sec) to NAS - can someone help shed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> Message-ID: On Tue, Oct 20, 2009 at 10:32 AM, Rainer Gerhards wrote: > Hi Aaron, > > first and very important question: which version of rsyslog are you using? > > Of course - I forgot to include that, sorry - 4.4.1. Running on RHEL 5.3. From rgerhards at hq.adiscon.com Tue Oct 20 18:58:19 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 18:58:19 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincrease inwrite volume (bytes/sec) to NAS - can someone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Tuesday, October 20, 2009 6:54 PM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling $ActionFileEnableSync caused > massiveincrease inwrite volume (bytes/sec) to NAS - can someone > helpshed light? > > On Tue, Oct 20, 2009 at 10:32 AM, Rainer Gerhards > wrote: > > > Hi Aaron, > > > > first and very important question: which version of rsyslog are you > using? > > > > > Of course - I forgot to include that, sorry - 4.4.1. Running on RHEL > 5.3. Hmmm... OK, so this is a version without output batching support (v5+) and without background writer (current v4-beta). So I do not really see how rsyslog could request this writing (other, of course, than by using the sync calls). Could you create a debug log? I'd try to dig through it. But I'd actually think that it has something to do with the way NFS parameters are set. Rainer From anichols at trumped.org Tue Oct 20 20:41:44 2009 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 20 Oct 2009 12:41:44 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincrease inwrite volume (bytes/sec) to NAS - can someone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> Message-ID: On Tue, Oct 20, 2009 at 10:58 AM, Rainer Gerhards wrote: > Hmmm... OK, so this is a version without output batching support (v5+) and > without background writer (current v4-beta). So I do not really see how > rsyslog could request this writing (other, of course, than by using the > sync > calls). Could you create a debug log? I'd try to dig through it. But I'd > actually think that it has something to do with the way NFS parameters are > set. > Here are the NFS mount options for the nodes writing to the NFS volume. The NFS server is an EMC Celerra - I can get any specific parameters there are questions about but can't easily provide a dump of the configured options: hostname:/log/syslog /log/syslog nfs rw,vers=3,rsize=32768,wsize=32768,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=hostname 0 0 I will work on a debug log but as these are production systems it'll take a little time. Thanks, Aaron From anichols at trumped.org Tue Oct 20 22:37:43 2009 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 20 Oct 2009 14:37:43 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincrease inwrite volume (bytes/sec) to NAS - can someone helpshed light? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> Message-ID: Also - can someone just confirm that my assumption is correct about the expected behavior? Should the "$ActionFileEnableSync [on/off]" option enable the ABILITY to turn on file sync per action using "-" or does it do something different? If I have "$ActionFileEnableSync off" set - should I expect adding or removing "-" before an action to have any impact at all? The documentation just associates these two options with enabling/disabling "file sync" but it seems like there are some additional subtleties that aren't clear. Thanks, Aaron On Tue, Oct 20, 2009 at 12:41 PM, Aaron Nichols wrote: > > > On Tue, Oct 20, 2009 at 10:58 AM, Rainer Gerhards < > rgerhards at hq.adiscon.com> wrote: > >> Hmmm... OK, so this is a version without output batching support (v5+) >> and >> without background writer (current v4-beta). So I do not really see how >> rsyslog could request this writing (other, of course, than by using the >> sync >> calls). Could you create a debug log? I'd try to dig through it. But I'd >> actually think that it has something to do with the way NFS parameters are >> set. >> > > From david at lang.hm Wed Oct 21 07:12:39 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 20 Oct 2009 22:12:39 -0700 (PDT) Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase in write volume (bytes/sec) to NAS - can someone help shed light? In-Reply-To: References: Message-ID: hmm, how large are the log entries? I'm wondering if the fact that every log entry then updates the metadata (size, time modified, etc) could account for most, if not all, of your extra traffic. the other thing that you could be seeing is if the NFS server records write traffic based on # of blocks written * block size. when doing a sync after each write you will do a lot of blocks (but re-write the same block many times), if it counts each of these writes as full block that would greatly magnify the write totals. I think you will really like the message batching in v5. you configure it to output every X messages or Y ms, whichever comes first. doing a sync after every write will have _far_ less effect. David Lang On Tue, 20 Oct 2009, Aaron Nichols wrote: > Date: Tue, 20 Oct 2009 09:32:07 -0600 > From: Aaron Nichols > Reply-To: rsyslog-users > To: rsyslog-users > Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase in > write volume (bytes/sec) to NAS - can someone help shed light? > > Hello, > We have two rsyslog "writer" instances which are receiving syslog > messages and writing them to an NFS mount on a NAS. The NAS is connected via > 10gb ethernet but has a number of other applications writing to it as well. > Typical write volume for our log writers is 2-3Mbps to the NAS. Yesterday I > attempted to resolve some complaints with logs being "bursty" - events > appear to be written in batches which sounded consistent with buffering on > the rsyslog daemon or not syncing the file with every message. These bursts > could come as infrequently as every 30 seconds however, which seems > excessive. I took a number of steps to remedy this but ultimately had to > reverse the changes because write volume (measured at the disks) to the NAS > went from 2Mbps up to 9Mbps and iops went from 30 up to 900 at the peak. I'm > sure there is an explanation for this - but there are some subtle > configuration changes which had an impact and I don't understand exactly > why. > > First, the main change was to set "$ActionFileEnableSync on" in each logging > servers configuration. From what I can tell this globally enables the > ability to turn on file sync and it defaults to "off". This change, combined > with removing the "-" from all the actions in my configuration should have > resulted in all files being synced all the time - probably the worst case > scenario and I'm not all that suprised this caused problems. > > Second, before rolling the change back 100% - I removed the > "$ActionFileEnableSync on" configuration from the servers but left the > action configurations without the "-" - assuming that the global > configuration option would completely disable file sync and using a "-" > would have no effect. This did not seem to be the case - it did dramatically > decrease the volume of traffic to the NAS but still the volume was about > double (5Mbps) the normal volume. > > Third, I added "-" before all the action lines in the configuration and this > brought volume back down to the 2-3Mbps we are used to. > > This leaves me with a few questions: > > 1) I would expect file sync to increase transactions with the NAS but not > increase the volume of data being written to the NAS as measured at the > disk. Can someone shed light on why this would so profoundly impact write > volume? Load on the rsyslog servers also went down substantially after > making this change - presumably because there were fewer queued transactions > - not sure. > > 2) Is there some difference between what "$ActionFileEnableSync on" does vs. > adding a "-" before an action other than one is global and one is > per-action? I thought these were just two different levels of granularity > for configuring the same thing but that doesn't appear to be the case. > > 3) Is it expected to have rsyslog take up to 30 seconds to flush messages to > a logfile? These are BUSY logs - being written to multiple times per second, > so it's not a delay on the client. I can observe these same clients logging > messages to another logging server (the old server) running syslog-ng and > messages are observable in near real-time. > > Ultimate - #3 is the problem I am trying to solve, but I'm very curious why > these changes had the impact they did and why globally disabling file sync > using "$ActionFileEnableSync" would not entire decrease log volume until I > add "-" to all the actions. > > Let me know if I can add info that would help shed light on this. > > Thanks, > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From anichols at trumped.org Wed Oct 21 07:28:48 2009 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 20 Oct 2009 23:28:48 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase in write volume (bytes/sec) to NAS - can someone help shed light? In-Reply-To: References: Message-ID: On Tue, Oct 20, 2009 at 11:12 PM, wrote: > hmm, how large are the log entries? I'm wondering if the fact that every > log entry then updates the metadata (size, time modified, etc) could > account for most, if not all, of your extra traffic. > Most log entries are small (under 2k) but there is one log destination which produces very large log entries. We have had to increase the max message size to 64k to accommodate these messages. This is only one destination but comes from multiple sources (collated into a single file) - so with a better understanding of how the "$ActionFileEnableSync" option vs. using "-" on a particular action works, I may be able to enable sync for log destinations other than that one high volume, large message size destination. > the other thing that you could be seeing is if the NFS server records > write traffic based on # of blocks written * block size. > > when doing a sync after each write you will do a lot of blocks (but > re-write the same block many times), if it counts each of these writes as > full block that would greatly magnify the write totals. > > I think you will really like the message batching in v5. you configure it > to output every X messages or Y ms, whichever comes first. doing a sync > after every write will have _far_ less effect. > > I probably will enjoy v5 but am not yet ready to run it in production. I need to find a good balance between log update frequency and keeping NAS performance sane. I did pass this feedback along to our storage guys to see if there is any tuning we can do around this. I will also be gathering a debug log soon - awaiting approvals to run the test. I assume the debug output would show some indication of where this might be causing problems? Thanks again for the feedback. I'm pushing for a support contract as I get the feeling it'll come in handy as well and I want to support the project, but I appreciate the feedback in this forum, I know everyone is hard at work. Thanks, Aaron From rgerhards at hq.adiscon.com Wed Oct 21 08:59:48 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 21 Oct 2009 08:59:48 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase in write volume (bytes/sec) to NAS - can someone help shed light? References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103267@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, October 21, 2009 7:13 AM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling $ActionFileEnableSync caused massive > increase in write volume (bytes/sec) to NAS - can someone help shed > light? > > hmm, how large are the log entries? I'm wondering if the fact that > every > log entry then updates the metadata (size, time modified, etc) could > account for most, if not all, of your extra traffic. rsyslog does an fdatasync() to avoid this problem. However, that API is not available on all platforms... Rainer From rgerhards at hq.adiscon.com Wed Oct 21 09:33:22 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 21 Oct 2009 09:33:22 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincrease in write volume (bytes/sec) to NAS - can someonehelp shed light? References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710326A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Wednesday, October 21, 2009 7:29 AM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling $ActionFileEnableSync caused > massiveincrease in write volume (bytes/sec) to NAS - can someonehelp > shed light? > > On Tue, Oct 20, 2009 at 11:12 PM, wrote: > > > hmm, how large are the log entries? I'm wondering if the fact that > every > > log entry then updates the metadata (size, time modified, etc) could > > account for most, if not all, of your extra traffic. > > > > Most log entries are small (under 2k) but there is one log destination > which > produces very large log entries. We have had to increase the max > message > size to 64k to accommodate these messages. This is only one destination > but > comes from multiple sources (collated into a single file) - so with a > better > understanding of how the "$ActionFileEnableSync" option vs. using "-" > on a > particular action works, I may be able to enable sync for log > destinations > other than that one high volume, large message size destination. Sorry, I forgot to answer that in my posting yesterday. $ActionFileEnableSync is position dependent. So I assume that it is giving right at the top of rsyslog.conf, before any action. If set to on, it *enables* the sync facility. So you can use the dash in front of the file to specify whether or not a sync should happen. If it is set to "off" (the default), the "dash-rule" is always ignored and a sync never happens. This was done because many default configs have the sync option set just out of habit and the performance toll, as you have seen, is immense. > > > > the other thing that you could be seeing is if the NFS server records > > write traffic based on # of blocks written * block size. > > > > when doing a sync after each write you will do a lot of blocks (but > > re-write the same block many times), if it counts each of these > writes as > > full block that would greatly magnify the write totals. > > > > I think you will really like the message batching in v5. you > configure it > > to output every X messages or Y ms, whichever comes first. doing a > sync > > after every write will have _far_ less effect. > > > > > I probably will enjoy v5 but am not yet ready to run it in production. > I > need to find a good balance between log update frequency and keeping > NAS > performance sane. I did pass this feedback along to our storage guys to > see > if there is any tuning we can do around this. > > I will also be gathering a debug log soon - awaiting approvals to run > the > test. I assume the debug output would show some indication of where > this > might be causing problems? My primary interest in the debug log is to see if something works other than I would expect it. If that is not the case, you need to dig down into what NFS does to cause this behavior. Doing an strace may also be a good idea, depending on the output of the debug log. The core problem from my POV is that in 4.4.1 there is no code that could create the behavior you describe (at least not to the best of my knowledge, and this is what the debug log helps verify ;)). Rainer From rgerhards at hq.adiscon.com Wed Oct 21 09:38:19 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 21 Oct 2009 09:38:19 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincreaseinwrite volume (bytes/sec) to NAS - can someone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Tuesday, October 20, 2009 10:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling $ActionFileEnableSync caused > massiveincreaseinwrite volume (bytes/sec) to NAS - can someone helpshed > light? > > Also - can someone just confirm that my assumption is correct about the > expected behavior? Should the "$ActionFileEnableSync [on/off]" option > enable > the ABILITY to turn on file sync per action using "-" or does it do > something different? See my other note, but there may be a subtle misunderstanding: The presence of "-" turns OFF file syncing. Absence of "-" causes a sync! So in *.* /path/to/fileWithSync *.* -/path/to/fileWithoutSync fileWithSync is being synced and fileWithoutSync is not! This is traditional config file syntax and probably the main reason why *a lot* of real-world config files do syncs... Rainer From tbergfeld at hq.adiscon.com Wed Oct 21 11:57:33 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 21 Oct 2009 11:57:33 +0200 Subject: [rsyslog] rsyslog 4.5.5 (v4-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103272@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.5.5, a member of the v4-beta branch. This release offers a new config directive $InputTCPServerNotifyOnConnectionClose. Also there are also some bug fixes. See Changelog for more details. This is a recommended update for all users of the devel branch. ChangeLog: http://www.rsyslog.com/Article416.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-182.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com _______________________________________________ From tbergfeld at hq.adiscon.com Wed Oct 21 11:56:58 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 21 Oct 2009 11:56:58 +0200 Subject: [rsyslog] rsyslog 5.3.2 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103271@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.3.2, a member of the devel branch. This release offers a lot of enhancement like the enhanced omfile to support transactional interface. Also there are also some bug fixes. See Changelog for more details. This is a recommended update for all users of the devel branch. ChangeLog: http://www.rsyslog.com/Article415.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-181.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com _______________________________________________ From anichols at trumped.org Wed Oct 21 15:17:49 2009 From: anichols at trumped.org (Aaron Nichols) Date: Wed, 21 Oct 2009 07:17:49 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincreaseinwrite volume (bytes/sec) to NAS - can someone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> Message-ID: On Wed, Oct 21, 2009 at 1:38 AM, Rainer Gerhards wrote: > See my other note, but there may be a subtle misunderstanding: The presence > of "-" turns OFF file syncing. Absence of "-" causes a sync! So in > > *.* /path/to/fileWithSync > *.* -/path/to/fileWithoutSync > > fileWithSync is being synced and fileWithoutSync is not! > > This is traditional config file syntax and probably the main reason why *a > lot* of real-world config files do syncs... > > I understood correctly and now I'm certain I know how it should work. I need to do some more testing and I think I have enough information to do that and will come back here when I have something interesting to share. An observation I did make when this was happening was that if $ActionFileEnableSync was disabled (commented out) but the "-" was missing before the actions in the configuration, write volume to the NAS was higher than when we had $ActionFileEnableSync disabled and had the "-" added before the action (write volume was 100% higher if $ActionFileEnableSync was OFF and the "-" option was missing from all log actions than if $ActionFileEnableSync was OFF and "-" was included in all log actions.) This led to my original suspicion that the two options somehow change sync behavior independently. If they shouldn't, then I need to get the debug log and do some analysis. I appreciate the guidance - I'll gather some more details. Thanks, Aaron From rgerhards at hq.adiscon.com Wed Oct 21 15:22:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 21 Oct 2009 15:22:29 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync causedmassiveincreaseinwrite volume (bytes/sec) to NAS - cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Wednesday, October 21, 2009 3:18 PM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling $ActionFileEnableSync > causedmassiveincreaseinwrite volume (bytes/sec) to NAS - cansomeone > helpshed light? > > On Wed, Oct 21, 2009 at 1:38 AM, Rainer Gerhards > wrote: > > > See my other note, but there may be a subtle misunderstanding: The > presence > > of "-" turns OFF file syncing. Absence of "-" causes a sync! So in > > > > *.* /path/to/fileWithSync > > *.* -/path/to/fileWithoutSync > > > > fileWithSync is being synced and fileWithoutSync is not! > > > > This is traditional config file syntax and probably the main reason > why *a > > lot* of real-world config files do syncs... > > > > > I understood correctly and now I'm certain I know how it should work. I > need > to do some more testing and I think I have enough information to do > that and > will come back here when I have something interesting to share. An > observation I did make when this was happening was that if > $ActionFileEnableSync was disabled (commented out) but the "-" was > missing > before the actions in the configuration, write volume to the NAS was > higher > than when we had $ActionFileEnableSync disabled and had the "-" added > before > the action (write volume was 100% higher if $ActionFileEnableSync was > OFF > and the "-" option was missing from all log actions than if > $ActionFileEnableSync was OFF and "-" was included in all log actions.) That indeed sounds odd. Are you sure the $ActionFileEnableSync is in front of all write statements? On some systems, write statements may be included via files and if these are in front of the directive, those would be unaffected by it... Rainer From anichols at trumped.org Wed Oct 21 16:25:21 2009 From: anichols at trumped.org (Aaron Nichols) Date: Wed, 21 Oct 2009 08:25:21 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync causedmassiveincreaseinwrite volume (bytes/sec) to NAS - cansomeone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> Message-ID: On Wed, Oct 21, 2009 at 7:22 AM, Rainer Gerhards wrote: > That indeed sounds odd. Are you sure the $ActionFileEnableSync is in front > of > all write statements? On some systems, write statements may be included via > files and if these are in front of the directive, those would be unaffected > by it... > We do also use includes but in this case $ActionFileEnableSync was at the top of the included file with all write statements below it. I will also note that the write volume changed significantly with two different config changes. To uber-simplify my original email: "normal" volume - $ActionFileEnableSync off & all actions include "-" in front : 2-3mbps ~30 iops $ActionFileEnableSync off & all actions exclude "-" in front : 5mbps (~100% increase) (iops unknown) $ActionFileEnableSync on & all actions exclude "-" in front : 9mbps (~300-400% increase) ~900 iops I am using a mix of static and dynamic filename actions and some actions use format templates as well. I suppose there could be one of those combinations which is not honoring the global $ActionFileEnableSync configuration. Again, I need to do some testing because at this point I'm just making guesses without much information. Thanks, Aaron From rgerhards at hq.adiscon.com Wed Oct 21 17:32:59 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 21 Oct 2009 17:32:59 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Wednesday, October 21, 2009 4:25 PM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling > $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to > NAS -cansomeone helpshed light? > > On Wed, Oct 21, 2009 at 7:22 AM, Rainer Gerhards > wrote: > > > That indeed sounds odd. Are you sure the $ActionFileEnableSync is in > front > > of > > all write statements? On some systems, write statements may be > included via > > files and if these are in front of the directive, those would be > unaffected > > by it... > > > > We do also use includes but in this case $ActionFileEnableSync was at > the > top of the included file with all write statements below it. I will > also > note that the write volume changed significantly with two different > config > changes. To uber-simplify my original email: > > "normal" volume - $ActionFileEnableSync off & all actions include "-" > in > front : 2-3mbps ~30 iops > $ActionFileEnableSync off & all actions exclude "-" in front : 5mbps > (~100% > increase) (iops unknown) > $ActionFileEnableSync on & all actions exclude "-" in front : 9mbps > (~300-400% increase) ~900 iops > > I am using a mix of static and dynamic filename actions and some > actions use > format templates as well. I suppose there could be one of those > combinations > which is not honoring the global $ActionFileEnableSync configuration. I may be overlooking something here, but the check is very straightforward and I don't see why some should be affected but others not. This is the file (and spot) in question: http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/omfile.c;h=bb12b4b61dacb 84167b9b0d17e49a597342dddc0;hb=07b076ddcfed4ea9b447d0be574d1dcdb799bc2f#l623 Do a search for "bEnableSync" to see all occurences. But the debug log will show... Rainer > Again, > I need to do some testing because at this point I'm just making guesses > without much information. > > Thanks, > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From wirelessdreamer at gmail.com Wed Oct 21 18:09:36 2009 From: wirelessdreamer at gmail.com (Nathanael Anderson) Date: Wed, 21 Oct 2009 11:09:36 -0500 Subject: [rsyslog] Regex matching problems Message-ID: I've been trying to get a basic regex to work in the regex tester at: http://www.rsyslog.com/user-regex.php \.[0-9]{1,3} I'm using this regex to match one octet in this message: Killing attempted connection: tcp (192.168.129.13:48351 - 192.168.145.91:35500) This should return 6 matches that can be accessed by changing the position fied from 0 to the field I want. in both rsyslog and the testing link, this is not the case. %msg:R,ERE,0,DFLT:\.[0-9]{1,3}--end% Is this a bug, or am I misunderstanding how this should behave? Nathanael From david at lang.hm Wed Oct 21 20:04:27 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 21 Oct 2009 11:04:27 -0700 (PDT) Subject: [rsyslog] makefile tweak Message-ID: when doing 'make clean' it should not need to run configure first. David Lang From mbiebl at gmail.com Wed Oct 21 20:11:49 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 21 Oct 2009 20:11:49 +0200 Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: 2009/10/21 : > when doing 'make clean' it should not need to run configure first. Ist that from a release tarball or a git snapshot? Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From david at lang.hm Wed Oct 21 22:19:41 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 21 Oct 2009 13:19:41 -0700 (PDT) Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: On Wed, 21 Oct 2009, Michael Biebl wrote: > 2009/10/21 : >> when doing 'make clean' it should not need to run configure first. > > Ist that from a release tarball or a git snapshot? git snapshot, especially after doing the autoconf -fvi. the system spends several min doing lots of checks before it can remove the files, then I do a configure with the options that I need and it goes through and does all those checks again. David Lang From mbiebl at gmail.com Wed Oct 21 22:26:09 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 21 Oct 2009 22:26:09 +0200 Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: 2009/10/21 : > On Wed, 21 Oct 2009, Michael Biebl wrote: > >> 2009/10/21 ?: >>> when doing 'make clean' it should not need to run configure first. >> >> Ist that from a release tarball or a git snapshot? > > git snapshot, especially after doing the autoconf -fvi. the system spends That's the wrong command. You should use autoreconf -vfi in a fresh git checkout. After that ./configure && make clean works as expected for me. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From david at lang.hm Wed Oct 21 22:30:22 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 21 Oct 2009 13:30:22 -0700 (PDT) Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: On Wed, 21 Oct 2009, Michael Biebl wrote: > Date: Wed, 21 Oct 2009 22:26:09 +0200 > From: Michael Biebl > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] makefile tweak > > 2009/10/21 : >> On Wed, 21 Oct 2009, Michael Biebl wrote: >> >>> 2009/10/21 ?: >>>> when doing 'make clean' it should not need to run configure first. >>> >>> Ist that from a release tarball or a git snapshot? >> >> git snapshot, especially after doing the autoconf -fvi. the system spends > > That's the wrong command. > > You should use autoreconf -vfi in a fresh git checkout. sorry, I typed the wrong thing in the e-mail > After that ./configure && make clean works as expected for me. I do autoreconf -fvi make clean ./configure --enable-imfile make the make clean forces a configure David Lang From mbiebl at gmail.com Wed Oct 21 22:39:36 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 21 Oct 2009 22:39:36 +0200 Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: 2009/10/21 : > On Wed, 21 Oct 2009, Michael Biebl wrote: > >> Date: Wed, 21 Oct 2009 22:26:09 +0200 >> From: Michael Biebl >> Reply-To: rsyslog-users >> To: rsyslog-users >> Subject: Re: [rsyslog] makefile tweak >> >> 2009/10/21 ?: >>> >>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>> >>>> 2009/10/21 ?: >>>>> >>>>> when doing 'make clean' it should not need to run configure first. >>>> >>>> Ist that from a release tarball or a git snapshot? >>> >>> git snapshot, especially after doing the autoconf -fvi. the system spends >> >> That's the wrong command. >> >> You should use autoreconf -vfi in a fresh git checkout. > > sorry, I typed the wrong thing in the e-mail > >> After that ./configure && make clean works as expected for me. > > I do > autoreconf -fvi > make clean How do you expect "make" to work without running ./configure before? If you have a Makefile's from a previous ./configure run, it's absolutely correct behaviour that after a autoreconf -vfi, a configure run is first done before you can run make. To sum up, everything works as it should. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From david at lang.hm Wed Oct 21 22:52:01 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 21 Oct 2009 13:52:01 -0700 (PDT) Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: On Wed, 21 Oct 2009, Michael Biebl wrote: > 2009/10/21 : >> On Wed, 21 Oct 2009, Michael Biebl wrote: >> >>> 2009/10/21 ?: >>>> >>>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>>> >>>>> 2009/10/21 ?: >>>>>> >>>>>> when doing 'make clean' it should not need to run configure first. >>>>> >>>>> Ist that from a release tarball or a git snapshot? >>>> >>>> git snapshot, especially after doing the autoconf -fvi. the system spends >>> >>> That's the wrong command. >>> >>> You should use autoreconf -vfi in a fresh git checkout. >> >> sorry, I typed the wrong thing in the e-mail >> >>> After that ./configure && make clean works as expected for me. >> >> I do >> autoreconf -fvi >> make clean > > How do you expect "make" to work without running ./configure before? > If you have a Makefile's from a previous ./configure run, it's > absolutely correct behaviour that after a autoreconf -vfi, a configure > run is first done before you can run make. > > To sum up, everything works as it should. I would not expect to be able to do a normal make, but since a make clean just deletes files (and the same files, not matter what the config options are) I would expect it to work. David Lang From mbiebl at gmail.com Wed Oct 21 23:02:17 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 21 Oct 2009 23:02:17 +0200 Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: 2009/10/21 : > On Wed, 21 Oct 2009, Michael Biebl wrote: > >> 2009/10/21 ?: >>> >>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>> >>>> 2009/10/21 ?: >>>>> >>>>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>>>> >>>>>> 2009/10/21 ?: >>>>>>> >>>>>>> when doing 'make clean' it should not need to run configure first. >>>>>> >>>>>> Ist that from a release tarball or a git snapshot? >>>>> >>>>> git snapshot, especially after doing the autoconf -fvi. the system >>>>> spends >>>> >>>> That's the wrong command. >>>> >>>> You should use autoreconf -vfi in a fresh git checkout. >>> >>> sorry, I typed the wrong thing in the e-mail >>> >>>> After that ./configure && make clean works as expected for me. >>> >>> I do >>> autoreconf -fvi >>> make clean >> >> How do you expect "make" to work without running ./configure before? >> If you have a Makefile's from a previous ./configure run, it's >> absolutely correct behaviour that after a autoreconf -vfi, a configure >> run is first done before you can run make. >> >> To sum up, everything works as it should. > > I would not expect to be able to do a normal make, but since a make clean > just deletes files (and the same files, not matter what the config options > are) I would expect it to work. Well, that's how automake/autoconf works. If the timestamps of build input files change, the build system is automatically re-generated. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Wed Oct 21 23:09:35 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 21 Oct 2009 23:09:35 +0200 Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: 2009/10/21 Michael Biebl : > 2009/10/21 ?: >> On Wed, 21 Oct 2009, Michael Biebl wrote: >> >>> 2009/10/21 ?: >>>> >>>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>>> >>>>> 2009/10/21 ?: >>>>>> >>>>>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>>>>> >>>>>>> 2009/10/21 ?: >>>>>>>> >>>>>>>> when doing 'make clean' it should not need to run configure first. >>>>>>> >>>>>>> Ist that from a release tarball or a git snapshot? >>>>>> >>>>>> git snapshot, especially after doing the autoconf -fvi. the system >>>>>> spends >>>>> >>>>> That's the wrong command. >>>>> >>>>> You should use autoreconf -vfi in a fresh git checkout. >>>> >>>> sorry, I typed the wrong thing in the e-mail >>>> >>>>> After that ./configure && make clean works as expected for me. >>>> >>>> I do >>>> autoreconf -fvi >>>> make clean >>> >>> How do you expect "make" to work without running ./configure before? >>> If you have a Makefile's from a previous ./configure run, it's >>> absolutely correct behaviour that after a autoreconf -vfi, a configure >>> run is first done before you can run make. >>> >>> To sum up, everything works as it should. >> >> I would not expect to be able to do a normal make, but since a make clean >> just deletes files (and the same files, not matter what the config options >> are) I would expect it to work. > > Well, that's how automake/autoconf works. > If the timestamps of build input files change, the build system is > automatically re-generated. Btw, since you run autoreconf anyway there is a much nicer (and quicker way to get rid of the old build files): git clean -x -d -f autoreconf -vfi Will give you a clean working state ready to run ./configure && make -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Oct 22 06:40:52 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 22 Oct 2009 06:40:52 +0200 Subject: [rsyslog] makefile tweak Message-ID: <000101ca52d1$dea24313$100013ac@intern.adiscon.com> The point is that configure decides which files are to be created at all. So without a configure run, make does not even know what to delete. rainer ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 21.10.09 22:52 Betreff: Re: [rsyslog] makefile tweak On Wed, 21 Oct 2009, Michael Biebl wrote: > 2009/10/21 : >> On Wed, 21 Oct 2009, Michael Biebl wrote: >> >>> 2009/10/21 : >>>> >>>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>>> >>>>> 2009/10/21 : >>>>>> >>>>>> when doing 'make clean' it should not need to run configure first. >>>>> >>>>> Ist that from a release tarball or a git snapshot? >>>> >>>> git snapshot, especially after doing the autoconf -fvi. the system spends >>> >>> That's the wrong command. >>> >>> You should use autoreconf -vfi in a fresh git checkout. >> >> sorry, I typed the wrong thing in the e-mail >> >>> After that ./configure && make clean works as expected for me. >> >> I do >> autoreconf -fvi >> make clean > > How do you expect "make" to work without running ./configure before? > If you have a Makefile's from a previous ./configure run, it's > absolutely correct behaviour that after a autoreconf -vfi, a configure > run is first done before you can run make. > > To sum up, everything works as it should. I would not expect to be able to do a normal make, but since a make clean just deletes files (and the same files, not matter what the config options are) I would expect it to work. David Lang From marc.schiffbauer at mightycare.de Thu Oct 22 15:06:29 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Thu, 22 Oct 2009 15:06:29 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200910201639.43143.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> Message-ID: <200910221506.29850.marc.schiffbauer@mightycare.de> Am Dienstag, 20. Oktober 2009 18:38:16 schrieb Rainer Gerhards: > thanks! > > Interesting, I see that only one of the messages that should be in the .qi > are actually present. I wonder if this is related to the fact that ompgsql > emits an error message itself while being down (the others don't do this > AFIK). Will try to dig down to this (but I have to do so as time permits). > Thanks, please tell me if I can assist in any way. -Marc From david at lang.hm Thu Oct 22 21:29:48 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 22 Oct 2009 12:29:48 -0700 (PDT) Subject: [rsyslog] version numbers from git repository Message-ID: I've been doing a lot of testing and can have a dozen different copies of rsyslog that all claim to be the same version number. the ability of git to give more precise versions (as described in the message from the git mailing list below) would be handy in eliminating confusion. the nice thing is that if a person is using a tagged version, this just reports the tag with nothing extra, so changing the makefile to use this would not affect most people. David Lang ---------- Forwarded message ---------- Date: Thu, 22 Oct 2009 10:01:25 +0200 From: Martin Langhoff To: Robin Rosenberg Cc: newsgroups at catchall.shelter13.net, git at vger.kernel.org Subject: Re: Deciding between Git/Mercurial On Sun, Sep 27, 2009 at 8:01 PM, Robin Rosenberg wrote: > s?ndag 27 september 2009 14:24:32 skrev Anteru : >> Mercurial's revision number system: With git, I get an SHA1 hash for >> every commit, but it's not possible to see whether Hash1 is newer than >> Hash2, while Mecurial also adds a running number to each commit. What's > > But those numbers cannot be communicated since they are local to your > clone. You can use git-describe, which will look for the latest tag, and make a combo of latest tag, commits since the tag, short form of sha1. So you get "v1.6.3-33-g1234" - 33 commits after 1.6.3. Works very well to integrate in versioning. The git project itself uses it in the Makefile to set the versions, same as the kernel folk do -- I use it to version even RPM/DEBs. hth, m -- martin.langhoff at gmail.com martin at laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majordomo at vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html From rgerhards at hq.adiscon.com Thu Oct 22 21:32:58 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 22 Oct 2009 21:32:58 +0200 Subject: [rsyslog] version numbers from git repository References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> David, I like this idea, but I have to admit I have not even the slightest clue on how I should implement it... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, October 22, 2009 9:30 PM > To: rsyslog-users > Subject: [rsyslog] version numbers from git repository > > I've been doing a lot of testing and can have a dozen > different copies of > rsyslog that all claim to be the same version number. > > the ability of git to give more precise versions (as described in the > message from the git mailing list below) would be handy in > eliminating > confusion. > > the nice thing is that if a person is using a tagged version, > this just > reports the tag with nothing extra, so changing the makefile > to use this > would not affect most people. > > David Lang > > ---------- Forwarded message ---------- > Date: Thu, 22 Oct 2009 10:01:25 +0200 > From: Martin Langhoff > To: Robin Rosenberg > Cc: newsgroups at catchall.shelter13.net, git at vger.kernel.org > Subject: Re: Deciding between Git/Mercurial > > On Sun, Sep 27, 2009 at 8:01 PM, Robin Rosenberg > wrote: > > s?ndag 27 september 2009 14:24:32 skrev Anteru > : > >> Mercurial's revision number system: With git, I get an > SHA1 hash for > >> every commit, but it's not possible to see whether Hash1 > is newer than > >> Hash2, while Mecurial also adds a running number to each > commit. What's > > > > But those numbers cannot be communicated since they are > local to your > > clone. > > You can use git-describe, which will look for the latest tag, and make > a combo of latest tag, commits since the tag, short form of sha1. So > you get "v1.6.3-33-g1234" - 33 commits after 1.6.3. Works very well to > integrate in versioning. The git project itself uses it in the > Makefile to set the versions, same as the kernel folk do -- I use it > to version even RPM/DEBs. > > hth, > > > > m > -- > martin.langhoff at gmail.com > martin at laptop.org -- School Server Architect > - ask interesting questions > - don't get distracted with shiny stuff - working code first > - http://wiki.laptop.org/go/User:Martinlanghoff > -- > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo at vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu Oct 22 21:36:49 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 22 Oct 2009 12:36:49 -0700 (PDT) Subject: [rsyslog] version numbers from git repository In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 22 Oct 2009, Rainer Gerhards wrote: > David, > > I like this idea, but I have to admit I have not even the slightest clue on > how I should implement it... I don't know either, but I do know a couple projects that do implement it. git implements it the linux kernel implements it if LOCAL_VERSION_AUTO is enabled in the config if I get a chance I'll try to dig into their makefiles to find out how. David Lang P.S. I don't know what rsyslog does with the version number, but when I use checkinstall to create a package from the source it ends up detecting the version number many times instead of once so instead of version='5.3.2' I get version='5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2' > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Thursday, October 22, 2009 9:30 PM >> To: rsyslog-users >> Subject: [rsyslog] version numbers from git repository >> >> I've been doing a lot of testing and can have a dozen >> different copies of >> rsyslog that all claim to be the same version number. >> >> the ability of git to give more precise versions (as described in the >> message from the git mailing list below) would be handy in >> eliminating >> confusion. >> >> the nice thing is that if a person is using a tagged version, >> this just >> reports the tag with nothing extra, so changing the makefile >> to use this >> would not affect most people. >> >> David Lang >> >> ---------- Forwarded message ---------- >> Date: Thu, 22 Oct 2009 10:01:25 +0200 >> From: Martin Langhoff >> To: Robin Rosenberg >> Cc: newsgroups at catchall.shelter13.net, git at vger.kernel.org >> Subject: Re: Deciding between Git/Mercurial >> >> On Sun, Sep 27, 2009 at 8:01 PM, Robin Rosenberg >> wrote: >>> s?ndag 27 september 2009 14:24:32 skrev Anteru >> : >>>> Mercurial's revision number system: With git, I get an >> SHA1 hash for >>>> every commit, but it's not possible to see whether Hash1 >> is newer than >>>> Hash2, while Mecurial also adds a running number to each >> commit. What's >>> >>> But those numbers cannot be communicated since they are >> local to your >>> clone. >> >> You can use git-describe, which will look for the latest tag, and make >> a combo of latest tag, commits since the tag, short form of sha1. So >> you get "v1.6.3-33-g1234" - 33 commits after 1.6.3. Works very well to >> integrate in versioning. The git project itself uses it in the >> Makefile to set the versions, same as the kernel folk do -- I use it >> to version even RPM/DEBs. >> >> hth, >> >> >> >> m >> -- >> martin.langhoff at gmail.com >> martin at laptop.org -- School Server Architect >> - ask interesting questions >> - don't get distracted with shiny stuff - working code first >> - http://wiki.laptop.org/go/User:Martinlanghoff >> -- >> To unsubscribe from this list: send the line "unsubscribe git" in >> the body of a message to majordomo at vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Oct 23 07:54:28 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 07:54:28 +0200 Subject: [rsyslog] version numbers from git repository References: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103292@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, October 22, 2009 9:37 PM > To: rsyslog-users > Subject: Re: [rsyslog] version numbers from git repository > > On Thu, 22 Oct 2009, Rainer Gerhards wrote: > > > David, > > > > I like this idea, but I have to admit I have not even the slightest > clue on > > how I should implement it... > > I don't know either, but I do know a couple projects that do implement > it. > > git implements it > > the linux kernel implements it if LOCAL_VERSION_AUTO is enabled in the > config > > if I get a chance I'll try to dig into their makefiles to find out how. I've looked a bit into the issue. Neither of them use the GNU build system. In essence, it boils down to running "git describe" as part of the build process, generating a file with it. However, I do not know how I may achieve that. It should probably be run during the ./configure stage. Maybe someone can provide some advise ;) > > David Lang > > P.S. I don't know what rsyslog does with the version number, but when I > use checkinstall to create a package from the source it ends up > detecting > the version number many times instead of once > > so instead of > version='5.3.2' > I get > version='5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2' I have no real idea what may cause that other than the fact that the version number is probably present more than once in the constant pool because of rsyslog's modular strucutre. Rainer From rgerhards at hq.adiscon.com Fri Oct 23 08:21:01 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 08:21:01 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> Aaron, I am replying on-list so that others can follow (and I hope they have some advise). I have stripped the sensitive part of your message. In short for all others: I got a debug log and have now reviewed it. > During this debug run we saw NAS write volume increase from a "normal" > 2-3mbps and 35 iops up to 500 iops and 5.6mbps toward the end of the > test. I do understand how enabling file sync will increase iops but > I'm unclear on why the volume of data being written would increase so > much. If rsyslog has file sync "disabled" then I would expect bursts > of data and if file sync is "enabled" I would expect the same amount > of data to be written in much more frequent, but also much smaller > chunks. NFS could be a culprit here, but I'm curious what NFS > parameters would impact this behavior. The debug log does not show anything that looks specifically suspect. The incoming traffic is intense and the default queue size is overrun quickly, resulting in voluntary delays of the sender (via tcp). Looking at the traffic, I think this is perfectly legal and increasing the queue size would not bring any benefit (I think the end result would just be a larger memory footprint without any advantage). I do not see any unusual write behavior. Of course, the debug log could be more verbose as it does not specifically tell about the syncs (usually, the log is missing some information you'd like to see for a given instant, but you can't include everything...). But looking at the code base plus the log really does not reveal anything suspicious. One thing you might want to try is doing an strace so that we can look at it and see which exact OS calls are made and how long they last. > For the record, changing the sync behavior of rsyslog DOES fix the > original complaint. When we have file sync disabled we see up to 30 > second pauses between writes to files which get multiple messages per > second delivered to the rsyslog host. There is nothing in the log that points to rsyslog doing these delays. So my conclusion really is that you need to look at the NFS layer. To me, it looks like NFS does some local caching, and maybe 30 seconds is the timeout at which NFS writes data from files that are constantly being written in short intervals. With sync, NFS probably writes and rewrites each block as David has said, thus resulting in the far larger i/o footprint. But I am far from being a NFS expert, so it would probably make sense to ask one ;) > I have looked at the option > $OMFileFlushOnTXEnd and wonder if it would be a preferred method to > resolve this in the 4.1.1 version (or if it even works there). This makes no sense for v4 (I think it is not even available). Sorry I have no better answer. If get a strace, I'd gladly look at it. Rainer From david at lang.hm Fri Oct 23 08:32:05 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 22 Oct 2009 23:32:05 -0700 (PDT) Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 23 Oct 2009, Rainer Gerhards wrote: >> For the record, changing the sync behavior of rsyslog DOES fix the >> original complaint. When we have file sync disabled we see up to 30 >> second pauses between writes to files which get multiple messages per >> second delivered to the rsyslog host. > > There is nothing in the log that points to rsyslog doing these delays. > > So my conclusion really is that you need to look at the NFS layer. To me, it > looks like NFS does some local caching, and maybe 30 seconds is the timeout > at which NFS writes data from files that are constantly being written in > short intervals. With sync, NFS probably writes and rewrites each block as > David has said, thus resulting in the far larger i/o footprint. But I am far > from being a NFS expert, so it would probably make sense to ask one ;) this reminded me of something. a standard complient NFS implementation would require that any write to a NFS volume go all the way to the destination and be safe on disk before the write call returns to the calling program. this doesn't mean when you do a fsync, it means each individual write call. for the program to think the data has been written and it not be available on the NFS drive to other systems there has to be some non-standard caching in place here. note that unless you have a high-end NFS server that includes some battery-backed ram on it, standard complient behavior also means a very low write transaction rate. note that the caching could be on the reader machines instead of on the writer, and it could be something like caching of the metadata (including size and last modified timestamp) David Lang From aland at freeradius.org Fri Oct 23 08:39:35 2009 From: aland at freeradius.org (Alan T DeKok) Date: Fri, 23 Oct 2009 08:39:35 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> Message-ID: <4AE14FA7.4010008@freeradius.org> Rainer Gerhards wrote: > So my conclusion really is that you need to look at the NFS layer. Logs should NEVER be written to an NFS mount. David gave a good explanation as to why. But the above prohibition should be made clear in the docs, if it isn't already. If the NFS server goes away, then the write transaction rate will go to zero. This likely isn't what people want from a syslog server. There is already a way to get syslog data from one system to another: the syslog protocol. Using NFS as a replacement for syslog is wrong on many levels. Alan DeKok. From rgerhards at hq.adiscon.com Fri Oct 23 09:03:05 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 09:03:05 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> <4AE14FA7.4010008@freeradius.org> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> I think the main point of the OP was that he logs to a NAS device, so I do not see way to write to it without going through a network share. Other than that, I agree to your statement. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Alan T DeKok > Sent: Friday, October 23, 2009 8:40 AM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling > $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to > NAS -cansomeone helpshed light? > > Rainer Gerhards wrote: > > So my conclusion really is that you need to look at the NFS layer. > > Logs should NEVER be written to an NFS mount. > > David gave a good explanation as to why. But the above prohibition > should be made clear in the docs, if it isn't already. > > If the NFS server goes away, then the write transaction rate will go > to zero. This likely isn't what people want from a syslog server. > > There is already a way to get syslog data from one system to another: > the syslog protocol. Using NFS as a replacement for syslog is wrong on > many levels. > > Alan DeKok. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Oct 23 09:17:43 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 23 Oct 2009 00:17:43 -0700 (PDT) Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> <4AE14FA7.4010008@freeradius.org> <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 23 Oct 2009, Rainer Gerhards wrote: > I think the main point of the OP was that he logs to a NAS device, so I do > not see way to write to it without going through a network share. ahh, have we been assuming that this is NFS when all he said was NAS? if so, please clarify what protocol you are using to talk to the NAS. it could make a huge difference here. David Lang > Other than that, I agree to your statement. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Alan T DeKok >> Sent: Friday, October 23, 2009 8:40 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Enabling >> $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to >> NAS -cansomeone helpshed light? >> >> Rainer Gerhards wrote: >>> So my conclusion really is that you need to look at the NFS layer. >> >> Logs should NEVER be written to an NFS mount. >> >> David gave a good explanation as to why. But the above prohibition >> should be made clear in the docs, if it isn't already. >> >> If the NFS server goes away, then the write transaction rate will go >> to zero. This likely isn't what people want from a syslog server. >> >> There is already a way to get syslog data from one system to another: >> the syslog protocol. Using NFS as a replacement for syslog is wrong on >> many levels. >> >> Alan DeKok. >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From aland at freeradius.org Fri Oct 23 09:42:22 2009 From: aland at freeradius.org (Alan T DeKok) Date: Fri, 23 Oct 2009 09:42:22 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> <4AE14FA7.4010008@freeradius.org> <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> Message-ID: <4AE15E5E.2020100@freeradius.org> Rainer Gerhards wrote: > I think the main point of the OP was that he logs to a NAS device, so I do > not see way to write to it without going through a network share. Configure rsyslog to write to local disk, and rsync periodically to the network share. Or, use another instance of rsyslog to copy from local disk to the network share. Any other method will have problems. Even NetApp systems have issues from time to time. Alan DeKok. From rgerhards at hq.adiscon.com Fri Oct 23 09:53:32 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 09:53:32 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com><4AE14FA7.4010008@freeradius.org><9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103298@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 23, 2009 9:18 AM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling > $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to > NAS -cansomeone helpshed light? > > On Fri, 23 Oct 2009, Rainer Gerhards wrote: > > > I think the main point of the OP was that he logs to a NAS device, so > I do > > not see way to write to it without going through a network share. > > ahh, have we been assuming that this is NFS when all he said was NAS? No, he said NFS, at least in the last message (not sure about the beginning of this story...) Rainer > > if so, please clarify what protocol you are using to talk to the NAS. > it > could make a huge difference here. > > David Lang > > > Other than that, I agree to your statement. > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Alan T DeKok > >> Sent: Friday, October 23, 2009 8:40 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Enabling > >> $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) > to > >> NAS -cansomeone helpshed light? > >> > >> Rainer Gerhards wrote: > >>> So my conclusion really is that you need to look at the NFS layer. > >> > >> Logs should NEVER be written to an NFS mount. > >> > >> David gave a good explanation as to why. But the above > prohibition > >> should be made clear in the docs, if it isn't already. > >> > >> If the NFS server goes away, then the write transaction rate will > go > >> to zero. This likely isn't what people want from a syslog server. > >> > >> There is already a way to get syslog data from one system to > another: > >> the syslog protocol. Using NFS as a replacement for syslog is wrong > on > >> many levels. > >> > >> Alan DeKok. > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Oct 23 10:52:02 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 23 Oct 2009 01:52:02 -0700 (PDT) Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: <4AE15E5E.2020100@freeradius.org> References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> <4AE14FA7.4010008@freeradius.org> <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> <4AE15E5E.2020100@freeradius.org> Message-ID: On Fri, 23 Oct 2009, Alan T DeKok wrote: > Rainer Gerhards wrote: >> I think the main point of the OP was that he logs to a NAS device, so I do >> not see way to write to it without going through a network share. > > Configure rsyslog to write to local disk, and rsync periodically to > the network share. Or, use another instance of rsyslog to copy from > local disk to the network share. > > Any other method will have problems. Even NetApp systems have issues > from time to time. I've found that for every situation I've ever heard someone (including me) say 'never do that' that there is _some_ situaton where 'that' is exactly the right thing to do ;-) I agree that in almost all cases, writing logs to NFS is not a good idea. you really will need a high-end NFS server to have any chance of it working (and the load that the logging will put on the server is going to be significant, unless something does caching, which is what is causing your problems) so this is a _very_ expensive way to go to make logging work. if you are putting the logs on NFS to make them readable by a bunch of machines, consider running rsyslog on a linux system and let that linux system export the log directory via NFS. for read-only access from many machines, this is probably going to rivel the speed of a very expensive NFS server, while being much simpler, cheaper, and more reliable (if you need redundant hardware things get more complicated, but it's still not hard to make it reliable and it will definantly be much cheaper) David Lang From anichols at trumped.org Fri Oct 23 15:22:24 2009 From: anichols at trumped.org (Aaron Nichols) Date: Fri, 23 Oct 2009 07:22:24 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> <4AE14FA7.4010008@freeradius.org> <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> <4AE15E5E.2020100@freeradius.org> Message-ID: Thank you everyone for your responses - rather than reply to each post, here is some feedback. The storage system - by all measure we have a "high end" system - It's an EMC Clarion CX3-80 attached to a Celerra NS80 NAS head with 10gb interfaces. The network file system is NFS, the Celerra can also present CIFS shares but I doubt those would work any better. This is not easily replaced with a linux box so that I can do rsync. The logs must be made available to multiple hosts and our legacy implementation *is* a Linux box writing to direct attached disk sharing NFS... it is crumbling under our write/read load so we had to come up with another solution. This solution is understood to be a bit of a compromise - the ideal situation would be direct attached fiberchannel disk and a cluster filesystem but that wasn't an option - mostly for cost. For all the complaints about NFS it performs just fine - if we enable file sync the write load just goes through the roof, but the NAS keeps up. Client bufferring (or throttling) is still possible however because we are transitioning to this new system many of these systems are dual-logging to two destinations so their logging behavior can be observed on two different systems. On the old system running syslog-ng these messages are written to disk and visible near real-time and they don't seem to be doing any rate limiting or otherwise. The debug log that was taken however had file sync enable - and when that's enabled this delay is not seen so I'm not suprised it wasn't visible. I can produce a debug log with file sync disabled for comparison, that's probably something I should have done in the first place. I have tried an strace on the rsyslog process(es) when running normally and haven't been able to get much out of them but that could be my own limited knowledge of strace. `strace -p ` yields a line of information and then just stops, even though the rsyslog process is still working fine and is obviously doing something. I've tried attaching to the different threads as well with the same result. I will see if I can get a comparison debug log with file sync disabled & if there is some guidance on getting a good strace I'm happy to provide that output. Yesterday we had to move the server from RHEL 5.3 to RHEL 5.1 due to some discovered limits in tcp connections - either intentionally or unintentionally different in our installation of 5.3. The server is now performing (with file sync disabled) at an even higher capacity on RH5.1 however I haven't had a chance to review whether the write latency is still observed - hopefully today I can look at that. Thanks again for all the information. Aaron From rgerhards at hq.adiscon.com Fri Oct 23 15:31:18 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 15:31:18 +0200 Subject: [rsyslog] Enabling$ActionFileEnableSynccausedmassiveincreaseinwrite volume(bytes/sec) to NAS -cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com><4AE14FA7.4010008@freeradius.org><9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com><4AE15E5E.2020100@freeradius.org> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032A4@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Friday, October 23, 2009 3:22 PM > To: rsyslog-users > Subject: Re: [rsyslog] > Enabling$ActionFileEnableSynccausedmassiveincreaseinwrite > volume(bytes/sec) to NAS -cansomeone helpshed light? > > Thank you everyone for your responses - rather than reply to each > post, here is some feedback. > > The storage system - by all measure we have a "high end" system - It's > an EMC Clarion CX3-80 attached to a Celerra NS80 NAS head with 10gb > interfaces. The network file system is NFS, the Celerra can also > present CIFS shares but I doubt those would work any better. This is > not easily replaced with a linux box so that I can do rsync. The logs > must be made available to multiple hosts and our legacy implementation > *is* a Linux box writing to direct attached disk sharing NFS... it is > crumbling under our write/read load so we had to come up with another > solution. This solution is understood to be a bit of a compromise - > the ideal situation would be direct attached fiberchannel disk and a > cluster filesystem but that wasn't an option - mostly for cost. For > all the complaints about NFS it performs just fine - if we enable file > sync the write load just goes through the roof, but the NAS keeps up. > > Client bufferring (or throttling) is still possible however because we > are transitioning to this new system many of these systems are > dual-logging to two destinations so their logging behavior can be > observed on two different systems. On the old system running syslog-ng > these messages are written to disk and visible near real-time and they > don't seem to be doing any rate limiting or otherwise. Note that the debug run is *much* slower than the actual run. It's the usual problem, if you instrument the system you also affect it. Performance-wise, we have seen that a debug build can be up to ten times slower (sometimes more, depending on config), but even the regular debug log alone, if turned on, heavily changes performance and order of execution. The root cause is the many debug outputs, which require at least some mutex snyc and that obviously affects the overall system... > The debug log > that was taken however had file sync enable - and when that's enabled > this delay is not seen so I'm not suprised it wasn't visible. I can > produce a debug log with file sync disabled for comparison, that's > probably something I should have done in the first place. >From my POV, that actually doesn't help at all. The reason is that I don't see any unusual in the log, and that means all I can see is that data is being continously written... > > I have tried an strace on the rsyslog process(es) when running > normally and haven't been able to get much out of them but that could > be my own limited knowledge of strace. `strace -p ` yields a line > of information and then just stops, even though the rsyslog process is > still working fine and is obviously doing something. I've tried > attaching to the different threads as well with the same result. ... the strace will probably provide much better info, because here we see the API calls. One thing I forgot to mention is that you should NOT use it together with the debug log. The debug output api calls will make the strace output unreadable. For example, I have these line in one of my test scripts: strace -T -tt -f -F tools/rsyslogd -c..... I don't know out of my head what each option does, but that line may give you a starting point. The end result is not a single line but rather a complete log of each API called PLUS how much time the processing took. That, too, is a quite enormous log... Rainer > > I will see if I can get a comparison debug log with file sync disabled > & if there is some guidance on getting a good strace I'm happy to > provide that output. Yesterday we had to move the server from RHEL 5.3 > to RHEL 5.1 due to some discovered limits in tcp connections - either > intentionally or unintentionally different in our installation of 5.3. > The server is now performing (with file sync disabled) at an even > higher capacity on RH5.1 however I haven't had a chance to review > whether the write latency is still observed - hopefully today I can > look at that. > > Thanks again for all the information. > > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Oct 23 22:22:52 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 23 Oct 2009 13:22:52 -0700 (PDT) Subject: [rsyslog] queue configuration Message-ID: I know that you can create an action queue for a specific output. is there any way to create an action queue for multiple outputs? for example, in my configuration I have :fromhost, !isequal, "127.0.0.1" /var/log/messages;TraditionalFormat :fromhost, isequal, "127.0.0.1" @192.168.1.1;TraditionalForwardFormat *.* @192.168.1.115 *.* @192.168.1.241 *.* @192.168.1.242 *.* @192.168.1.6 *.* @192.168.1.7 *.* @192.168.1.122 :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 & @192.168.1.1;fixsnareForwardFormat2 & ~ :syslogtag, startswith, "MSWinEventLog#011" /var/log/messages;fixsnareFormat & @192.168.1.1;fixsnareForwardFormat & ~ *.* /var/log/messages;TraditionalFormat *.* @192.168.1.1 it would be nice to move the outbound relays off to a different queue, and to put the list of rules that have order dependancies (because they output in different formats to fix up formatting problem) to a seperate queue to spread the work across multiple processes and not slow down the basic writing to file. but as far as I can tell I would have to create a queue for each individual item, and I can't create a queue for the part of the config where I need to discard. am I missing something (including a better way to do everything ;-) David Lang From rgerhards at hq.adiscon.com Fri Oct 23 22:27:06 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 22:27:06 +0200 Subject: [rsyslog] queue configuration References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032AF@GRFEXC.intern.adiscon.com> In short: this can not currently be done and it would require considerable modification to the engine to do so. Part of the work I plan in moving queues to rule sets may actually help, but it is some way to there. Icluding that I first want to get a stable v5 and there are lots of smaller, but important things in front of it (like the DNS resolution issue. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 23, 2009 10:23 PM > To: rsyslog-users > Subject: [rsyslog] queue configuration > > I know that you can create an action queue for a specific output. > > is there any way to create an action queue for multiple outputs? > > for example, in my configuration I have > > :fromhost, !isequal, "127.0.0.1" > /var/log/messages;TraditionalFormat > :fromhost, isequal, "127.0.0.1" > @192.168.1.1;TraditionalForwardFormat > *.* @192.168.1.115 > *.* @192.168.1.241 > *.* @192.168.1.242 > *.* @192.168.1.6 > *.* @192.168.1.7 > *.* @192.168.1.122 > :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 > & @192.168.1.1;fixsnareForwardFormat2 > & ~ > :syslogtag, startswith, "MSWinEventLog#011" > /var/log/messages;fixsnareFormat > & @192.168.1.1;fixsnareForwardFormat > & ~ > *.* /var/log/messages;TraditionalFormat > *.* @192.168.1.1 > > it would be nice to move the outbound relays off to a > different queue, and > to put the list of rules that have order dependancies > (because they output > in different formats to fix up formatting problem) to a > seperate queue to > spread the work across multiple processes and not slow down the basic > writing to file. > > but as far as I can tell I would have to create a queue for each > individual item, and I can't create a queue for the part of > the config > where I need to discard. > > am I missing something (including a better way to do everything ;-) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Oct 23 22:33:35 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 23 Oct 2009 13:33:35 -0700 (PDT) Subject: [rsyslog] queue configuration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032AF@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71032AF@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 23 Oct 2009, Rainer Gerhards wrote: > In short: this can not currently be done and it would require considerable > modification to the engine to do so. Part of the work I plan in moving queues > to rule sets may actually help, but it is some way to there. Icluding that I > first want to get a stable v5 and there are lots of smaller, but important > things in front of it (like the DNS resolution issue. that's what I suspected, I just wanted to make sure. with the new queue engine can you have multiple worker threads? (I seem to remember that initially you couldn't with batch mode, and in almost every case there is no reason to as a single thread in batch mode can do so much more than it could before) David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Friday, October 23, 2009 10:23 PM >> To: rsyslog-users >> Subject: [rsyslog] queue configuration >> >> I know that you can create an action queue for a specific output. >> >> is there any way to create an action queue for multiple outputs? >> >> for example, in my configuration I have >> >> :fromhost, !isequal, "127.0.0.1" >> /var/log/messages;TraditionalFormat >> :fromhost, isequal, "127.0.0.1" >> @192.168.1.1;TraditionalForwardFormat >> *.* @192.168.1.115 >> *.* @192.168.1.241 >> *.* @192.168.1.242 >> *.* @192.168.1.6 >> *.* @192.168.1.7 >> *.* @192.168.1.122 >> :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 >> & @192.168.1.1;fixsnareForwardFormat2 >> & ~ >> :syslogtag, startswith, "MSWinEventLog#011" >> /var/log/messages;fixsnareFormat >> & @192.168.1.1;fixsnareForwardFormat >> & ~ >> *.* /var/log/messages;TraditionalFormat >> *.* @192.168.1.1 >> >> it would be nice to move the outbound relays off to a >> different queue, and >> to put the list of rules that have order dependancies >> (because they output >> in different formats to fix up formatting problem) to a >> seperate queue to >> spread the work across multiple processes and not slow down the basic >> writing to file. >> >> but as far as I can tell I would have to create a queue for each >> individual item, and I can't create a queue for the part of >> the config >> where I need to discard. >> >> am I missing something (including a better way to do everything ;-) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Oct 23 22:35:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 22:35:56 +0200 Subject: [rsyslog] queue configuration References: <9B6E2A8877C38245BFB15CC491A11DA71032AF@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032B1@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 23, 2009 10:34 PM > To: rsyslog-users > Subject: Re: [rsyslog] queue configuration > > On Fri, 23 Oct 2009, Rainer Gerhards wrote: > > > In short: this can not currently be done and it would > require considerable > > modification to the engine to do so. Part of the work I > plan in moving queues > > to rule sets may actually help, but it is some way to > there. Icluding that I > > first want to get a stable v5 and there are lots of > smaller, but important > > things in front of it (like the DNS resolution issue. > > that's what I suspected, I just wanted to make sure. > > with the new queue engine can you have multiple worker > threads? (I seem to > remember that initially you couldn't with batch mode, and in > almost every > case there is no reason to as a single thread in batch mode > can do so much > more than it could before) Yes, you can do that BUT not for disk queues. This was the restriction. It may be that one early stage of batching support limited all workers to one. But we can do multiple workers. Rainer > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > >> Sent: Friday, October 23, 2009 10:23 PM > >> To: rsyslog-users > >> Subject: [rsyslog] queue configuration > >> > >> I know that you can create an action queue for a specific output. > >> > >> is there any way to create an action queue for multiple outputs? > >> > >> for example, in my configuration I have > >> > >> :fromhost, !isequal, "127.0.0.1" > >> /var/log/messages;TraditionalFormat > >> :fromhost, isequal, "127.0.0.1" > >> @192.168.1.1;TraditionalForwardFormat > >> *.* @192.168.1.115 > >> *.* @192.168.1.241 > >> *.* @192.168.1.242 > >> *.* @192.168.1.6 > >> *.* @192.168.1.7 > >> *.* @192.168.1.122 > >> :hostname, contains ,"MSWinEventLog" > /var/log/messages;fixsnareFormat2 > >> & @192.168.1.1;fixsnareForwardFormat2 > >> & ~ > >> :syslogtag, startswith, "MSWinEventLog#011" > >> /var/log/messages;fixsnareFormat > >> & @192.168.1.1;fixsnareForwardFormat > >> & ~ > >> *.* /var/log/messages;TraditionalFormat > >> *.* @192.168.1.1 > >> > >> it would be nice to move the outbound relays off to a > >> different queue, and > >> to put the list of rules that have order dependancies > >> (because they output > >> in different formats to fix up formatting problem) to a > >> seperate queue to > >> spread the work across multiple processes and not slow > down the basic > >> writing to file. > >> > >> but as far as I can tell I would have to create a queue for each > >> individual item, and I can't create a queue for the part of > >> the config > >> where I need to discard. > >> > >> am I missing something (including a better way to do everything ;-) > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Sat Oct 24 21:10:32 2009 From: david at lang.hm (david at lang.hm) Date: Sat, 24 Oct 2009 12:10:32 -0700 (PDT) Subject: [rsyslog] is there a way to find currnt queue status? Message-ID: thinking back to the testing of the v5 code that I have been doing this last week, one thing that would be very useful is if there is a way to find out what the current queue lengths are. one possible way to do this would be to have rsyslog watch for a signal and when it receives it, spit out the queue stats. for the stats definantly the length of each queue, possibly timestamps of oldest and newest message in each queue when spitting them out, some possible approaches are to stderr pro: simple con: stderr may not appear if rsyslog is running as a daemon from startup add a log message to the main queue containing the data pro: still fairly simple con: if there is a long queue, or it is blocked the data may not be visable for a while have each queue walker thread generate a message internally and process it pro: the data will show up fast, through the normal log mechanism con: this is probably the most work to imeplement this is not a critical issue, but it could be something that would speed up development in the long run by making it easier to figure out the state of the system. or I could have just missed how something like this is already supported ;-) David Lang From rgerhards at hq.adiscon.com Sat Oct 24 21:14:57 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 24 Oct 2009 21:14:57 +0200 Subject: [rsyslog] is there a way to find currnt queue status? References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032B2@GRFEXC.intern.adiscon.com> It's very partially being supported right now via imdiag. That module enables to query the main message queue length (and only for the main message queue). There is also a very, very rudamentary Java app in the v5. This was created as part of the debugging/performance testing effort earlier this year, but unfortunately more important things got into the way. An interace needs to be designed (and implemented) for performance counters. IT's on the agenda, but unfortunately not on the top spot. In concur that it would be very useful in many cases. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Saturday, October 24, 2009 9:11 PM > To: rsyslog-users > Subject: [rsyslog] is there a way to find currnt queue status? > > thinking back to the testing of the v5 code that I have been > doing this > last week, one thing that would be very useful is if there is > a way to > find out what the current queue lengths are. > > one possible way to do this would be to have rsyslog watch > for a signal > and when it receives it, spit out the queue stats. > > for the stats definantly the length of each queue, possibly > timestamps of > oldest and newest message in each queue > > when spitting them out, some possible approaches are > > to stderr > pro: simple > con: stderr may not appear if rsyslog is running as a > daemon from startup > > add a log message to the main queue containing the data > pro: still fairly simple > con: if there is a long queue, or it is blocked the data > may not be > visable for a while > > have each queue walker thread generate a message internally > and process it > pro: the data will show up fast, through the normal log mechanism > con: this is probably the most work to imeplement > > this is not a critical issue, but it could be something that > would speed > up development in the long run by making it easier to figure > out the state > of the system. > > or I could have just missed how something like this is > already supported > ;-) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Mon Oct 26 08:17:23 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 26 Oct 2009 08:17:23 +0100 Subject: [rsyslog] queue configuration References: <9B6E2A8877C38245BFB15CC491A11DA71032AF@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032B5@GRFEXC.intern.adiscon.com> David, all, I have thought a bit more about this. Actually, I think almost everything is already present by now, just some glue is missing. The most important thing, ruleset-specific "main" queues was due to implement soon (I worked towards that the past two month). So, in short, I'll have a look at it this week while running the stability lab tests (they always take considerable time). I don't want to promise anything, but it may be possible that this feature could become available shortly. That would also build that basis for some considerabe performance improvement (for some environments), so it seems to be very relevant to my short-term goals. Thanks for bringing this up and making me think about it ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 23, 2009 10:34 PM > To: rsyslog-users > Subject: Re: [rsyslog] queue configuration > > On Fri, 23 Oct 2009, Rainer Gerhards wrote: > > > In short: this can not currently be done and it would > require considerable > > modification to the engine to do so. Part of the work I > plan in moving queues > > to rule sets may actually help, but it is some way to > there. Icluding that I > > first want to get a stable v5 and there are lots of > smaller, but important > > things in front of it (like the DNS resolution issue. > > that's what I suspected, I just wanted to make sure. > > with the new queue engine can you have multiple worker > threads? (I seem to > remember that initially you couldn't with batch mode, and in > almost every > case there is no reason to as a single thread in batch mode > can do so much > more than it could before) > > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > >> Sent: Friday, October 23, 2009 10:23 PM > >> To: rsyslog-users > >> Subject: [rsyslog] queue configuration > >> > >> I know that you can create an action queue for a specific output. > >> > >> is there any way to create an action queue for multiple outputs? > >> > >> for example, in my configuration I have > >> > >> :fromhost, !isequal, "127.0.0.1" > >> /var/log/messages;TraditionalFormat > >> :fromhost, isequal, "127.0.0.1" > >> @192.168.1.1;TraditionalForwardFormat > >> *.* @192.168.1.115 > >> *.* @192.168.1.241 > >> *.* @192.168.1.242 > >> *.* @192.168.1.6 > >> *.* @192.168.1.7 > >> *.* @192.168.1.122 > >> :hostname, contains ,"MSWinEventLog" > /var/log/messages;fixsnareFormat2 > >> & @192.168.1.1;fixsnareForwardFormat2 > >> & ~ > >> :syslogtag, startswith, "MSWinEventLog#011" > >> /var/log/messages;fixsnareFormat > >> & @192.168.1.1;fixsnareForwardFormat > >> & ~ > >> *.* /var/log/messages;TraditionalFormat > >> *.* @192.168.1.1 > >> > >> it would be nice to move the outbound relays off to a > >> different queue, and > >> to put the list of rules that have order dependancies > >> (because they output > >> in different formats to fix up formatting problem) to a > >> seperate queue to > >> spread the work across multiple processes and not slow > down the basic > >> writing to file. > >> > >> but as far as I can tell I would have to create a queue for each > >> individual item, and I can't create a queue for the part of > >> the config > >> where I need to discard. > >> > >> am I missing something (including a better way to do everything ;-) > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From tbergfeld at hq.adiscon.com Tue Oct 27 11:12:18 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Tue, 27 Oct 2009 11:12:18 +0100 Subject: [rsyslog] rsyslog 5.3.3 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032CC@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.3.3, a member of the devel branch. This version offers a partially rewritten queue engine. The functionality was streamlined and simplified, resulting in not only fewer bug potential but also greater speed. This engine sets an important basis on which all further work can build. Note that due to the magnitude of changes, there is some increased bug potential. But in general, the new engine will be able to better handle some situations where the previous engine ran into problems. Also there are some bug fixes. This is a recommended update for all users of the devel branch. ChangeLog: http://www.rsyslog.com/Article419.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-183.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From martinmie at PartyGaming.com Tue Oct 27 11:56:52 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Tue, 27 Oct 2009 11:56:52 +0100 Subject: [rsyslog] rsyslog in uninterruptable sleep Message-ID: <0B1B9163138571439EAF804646F3F6AA0892EE19@GIBSVWIN004X.partygaming.local> Hi, I just did some small changes to the rsyslog.conf and wanted to restart the process: # /etc/init.d/rsyslog restart Shutting down system logger: [FAILED] Starting system logger: Already running. [FAILED] # ps -elf | grep rsyslog 5 D root 2037 1 1 76 0 - 12040 sync_b Oct18 ? 03:32:10 rsyslogd -c 4 -x -f /etc/rsyslog.conf -i /var/run/syslogd.pid This is rsyslog 4.2.0 running on RHEL5 and it's not the first time it happens. Any ideas on why this might happened and how to solve this w/o rebooting the system?? Thanks, Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From martinmie at PartyGaming.com Tue Oct 27 11:59:19 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Tue, 27 Oct 2009 11:59:19 +0100 Subject: [rsyslog] rsyslog and snare Message-ID: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> Hi, We recently started to receive logs from a Snare client. I applied the configuration changes proposed here: http://wiki.rsyslog.com/index.php/Using_Snare_as_a_client_on_Windows After restarting rsyslog, I get the following error: -- 2009-10-27T06:45:39.067279-04:00 logserger rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="29021" x-info="http://www.rsyslog.com"] (re)start 2009-10-27T06:45:39.063213-04:00 logserger rsyslogd-3000: unknown priority name "" [try http://www.rsyslog.com/e/3000 ] 2009-10-27T06:45:39.063571-04:00 logserger rsyslogd: the last error occured in /etc/rsyslog.conf, line 18 2009-10-27T06:45:39.064044-04:00 logserger rsyslogd: warning: selector line without actions will be discarded 2009-10-27T06:45:39.064213-04:00 logserger rsyslogd-2123: CONFIG ERROR: could not interpret master config file '/etc/rsyslog.conf'. [try http://www.rsyslog.com/e/2123 ] -- If I comment out the "%msg:::space-cc%" part it starts up cleanly but I assume that the filtering won't work. Suggestions? Thanks, Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From rgerhards at hq.adiscon.com Tue Oct 27 16:02:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 27 Oct 2009 16:02:17 +0100 Subject: [rsyslog] rsyslog in uninterruptable sleep References: <0B1B9163138571439EAF804646F3F6AA0892EE19@GIBSVWIN004X.partygaming.local> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032D3@GRFEXC.intern.adiscon.com> I think we just recently had this on the mailing list or bug tracker and I made a fix for it. I guess the fix is included in 4.4.2. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Martin Mielke > Sent: Tuesday, October 27, 2009 11:57 AM > To: rsyslog-users > Subject: [rsyslog] rsyslog in uninterruptable sleep > > > Hi, > > I just did some small changes to the rsyslog.conf and wanted to restart > the process: > > # /etc/init.d/rsyslog restart > Shutting down system logger: [FAILED] > Starting system logger: Already running. > [FAILED] > > # ps -elf | grep rsyslog > 5 D root 2037 1 1 76 0 - 12040 sync_b Oct18 ? > 03:32:10 rsyslogd -c 4 -x -f /etc/rsyslog.conf -i /var/run/syslogd.pid > > > This is rsyslog 4.2.0 running on RHEL5 and it's not the first time it > happens. > > > Any ideas on why this might happened and how to solve this w/o > rebooting > the system?? > > > Thanks, > Martin > > > This email and any attachments are confidential, and may be legally > privileged and protected by copyright. If you are not the intended > recipient dissemination or copying of this email is prohibited. If you > have received this in error, please notify the sender by replying by > email and then delete the email completely from your system. > > Any views or opinions are solely those of the sender. This > communication is not intended to form a binding contract unless > expressly indicated to the contrary and properly authorised. Any > actions taken on the basis of this email are at the recipient's own > risk. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Oct 27 17:53:20 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 27 Oct 2009 17:53:20 +0100 Subject: [rsyslog] important enhancement in rsyslog upcoming Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032D6@GRFEXC.intern.adiscon.com> Hi all, I was finally able to add some new featueres to rsyslog (stability is still my priority), and I think it is a very exciting one. In short: we can now partition input message into different queues, dramatically decreasing lock contention potential. Please read yourself: http://blog.gerhards.net/2009/10/next-round-of-performance-enhancement.html The change is committed, but I'll probably be busy a day or two with lab testing and testbench enhancement before I continue to add more enhancements. Rainer From epiphani at gmail.com Tue Oct 27 17:59:27 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 27 Oct 2009 12:59:27 -0400 Subject: [rsyslog] important enhancement in rsyslog upcoming In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032D6@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71032D6@GRFEXC.intern.adiscon.com> Message-ID: This looks like it has a lot of potential! I look forward to trying it. -Aaron On Tue, Oct 27, 2009 at 12:53 PM, Rainer Gerhards wrote: > Hi all, > > I was finally able to add some new featueres to rsyslog (stability is still > my priority), and I think it is a very exciting one. In short: we can now > partition input message into different queues, dramatically decreasing lock > contention potential. > > Please read yourself: > > http://blog.gerhards.net/2009/10/next-round-of-performance-enhancement.html > > The change is committed, but I'll probably be busy a day or two with lab > testing and testbench enhancement before I continue to add more enhancements. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Oct 27 18:09:06 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 27 Oct 2009 18:09:06 +0100 Subject: [rsyslog] important enhancement in rsyslog upcoming References: <9B6E2A8877C38245BFB15CC491A11DA71032D6@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032D7@GRFEXC.intern.adiscon.com> As a side-note, this does not yet mean that the input module itself runs on multiple threads. So the new functionality probably works best across different inputs. Massiv-parallel inputs are on my agenda as well, but some other things need to be done first so that I will have the necessary basis to build on (I'll not use a simple "one thread per connection" model, as this does not scale well). I'll probably go for multiplexed io with potentially one queue per worker, and quick worker engagement into that queue where no wait is necessary ... but more later ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Tuesday, October 27, 2009 5:59 PM > To: rsyslog-users > Subject: Re: [rsyslog] important enhancement in rsyslog upcoming > > This looks like it has a lot of potential! I look forward to trying > it. > > -Aaron > > On Tue, Oct 27, 2009 at 12:53 PM, Rainer Gerhards > wrote: > > Hi all, > > > > I was finally able to add some new featueres to rsyslog (stability is > still > > my priority), and I think it is a very exciting one. In short: we can > now > > partition input message into different queues, dramatically > decreasing lock > > contention potential. > > > > Please read yourself: > > > > http://blog.gerhards.net/2009/10/next-round-of-performance- > enhancement.html > > > > The change is committed, but I'll probably be busy a day or two with > lab > > testing and testbench enhancement before I continue to add more > enhancements. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jbondc at openmv.com Tue Oct 27 20:40:42 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Tue, 27 Oct 2009 15:40:42 -0400 Subject: [rsyslog] Property-Based Filters, not working? Message-ID: <002101ca573d$5f417f70$1dc47e50$@com> I tried adding the following to my config file: :msg,contains,"MSWinEventLog" *.* /var/log/windows.log :msg, !contains, "MSWinEventLog" *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err /var/log/messages security.* /var/log/security auth.info;authpriv.info /var/log/auth.log However the filter doesn't seem to apply, when debugging the parse sequence goes like this: 5250.008750501:main thread: Action 0x80a8600: queue 0x8094c00 created 5250.008823471:main thread: cfline: ':msg,contains,"MSWinEventLog"' 5250.008884805:main thread: selector line successfully processed 5250.008943799:main thread: - property-based filter 5250.009059262:main thread: tried selector action for builtin-file: -2001 5250.009130893:main thread: tried selector action for builtin-fwd: -2001 5250.009195089:main thread: tried selector action for builtin-shell: -2001 5250.009259497:main thread: tried selector action for builtin-discard: -2001 5250.009324602:main thread: tried selector action for builtin-usrmsg: -2001 5250.009394008:main thread: tried selector action for ompgsql.so: -2001 5250.009454057:main thread: config line NOT successfully processed Am I missing something? rsyslogd 4.4.2, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: Yes FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: No FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: No Runtime Instrumentation (slow code): No From david at lang.hm Tue Oct 27 21:03:01 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 27 Oct 2009 13:03:01 -0700 (PDT) Subject: [rsyslog] Property-Based Filters, not working? In-Reply-To: <002101ca573d$5f417f70$1dc47e50$@com> References: <002101ca573d$5f417f70$1dc47e50$@com> Message-ID: On Tue, 27 Oct 2009, Jonathan Bond-Caron wrote: > Date: Tue, 27 Oct 2009 15:40:42 -0400 > From: Jonathan Bond-Caron > Reply-To: rsyslog-users > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Property-Based Filters, not working? > > I tried adding the following to my config file: > > > > :msg,contains,"MSWinEventLog" > > > > *.* /var/log/windows.log > > > > :msg, !contains, "MSWinEventLog" > > > > *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err > /var/log/messages this is not how the property based filters work. I made the same mistake instead of :msg,contains,"MSWinEventLog" *.* /var/log/windows.log it should be :msg,contains,"MSWinEventLog" /var/log/windows.log when you do filter rules those filters are everything, you don't also do facility/priority filters as well (or if you need to do so, you need to make a compound if (() and ()) type rule) FYI, what I do with snare is $template fixsnareFormat,"%timereported% %HOSTNAME% MSWinEventLog %syslogtag:18:$%%msg:::drop-last-lf%\n" $template fixsnareForwardFormat,"<%pri%>%timereported% %HOSTNAME% MSWinEventLog %syslogtag:18:$:%%msg:::drop-last-lf%\n" $template fixsnareFormat2,"%timereported% %fromhost-ip% broken-MSWinEventLog %HOSTNAME% %syslogtag:18:$%%msg:::drop-last-lf%\n" $template fixsnareForwardFormat2,"<%pri%>%timereported% %fromhost-ip% broken-MSWinEventLog %HOSTNAME% %syslogtag:18:$:%%msg:::drop-last-lf%\n" :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 & @192.168.210.8;fixsnareForwardFormat2 & ~ :syslogtag, startswith, "MSWinEventLog\011" /var/log/messages;fixsnareFormat & @192.168.210.8;fixsnareForwardFormat & ~ *.* /var/log/messages;TraditionalFormat *.* @192.168.210.8 the fixsnare*2 formats are to handle cases where I haven't tracked down why snare is sending things incorrectly (8 of a couple hundred servers) David Lang > security.* /var/log/security > > auth.info;authpriv.info /var/log/auth.log > > > > > > However the filter doesn't seem to apply, when debugging the parse sequence > goes like this: > > > > 5250.008750501:main thread: Action 0x80a8600: queue 0x8094c00 created > > 5250.008823471:main thread: cfline: ':msg,contains,"MSWinEventLog"' > > 5250.008884805:main thread: selector line successfully processed > > 5250.008943799:main thread: - property-based filter > > 5250.009059262:main thread: tried selector action for builtin-file: -2001 > > 5250.009130893:main thread: tried selector action for builtin-fwd: -2001 > > 5250.009195089:main thread: tried selector action for builtin-shell: -2001 > > 5250.009259497:main thread: tried selector action for builtin-discard: -2001 > > 5250.009324602:main thread: tried selector action for builtin-usrmsg: -2001 > > 5250.009394008:main thread: tried selector action for ompgsql.so: -2001 > > 5250.009454057:main thread: config line NOT successfully processed > > > > Am I missing something? > > > > rsyslogd 4.4.2, compiled with: > > FEATURE_REGEXP: Yes > > FEATURE_LARGEFILE: Yes > > FEATURE_NETZIP (message compression): Yes > > GSSAPI Kerberos 5 support: No > > FEATURE_DEBUG (debug build, slow code): No > > Atomic operations supported: No > > Runtime Instrumentation (slow code): No > > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Tue Oct 27 21:05:47 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 27 Oct 2009 13:05:47 -0700 (PDT) Subject: [rsyslog] control character escaping change? Message-ID: it appears that the 5.3 development series recently changed the default for escaping control characters from # to \ so a tab changed from #011 to \011 was this a deliberate change going forward, or a mistake? David Lang From david at lang.hm Tue Oct 27 21:32:39 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 27 Oct 2009 13:32:39 -0700 (PDT) Subject: [rsyslog] rsyslog and snare In-Reply-To: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> Message-ID: On Tue, 27 Oct 2009, Martin Mielke wrote: > Hi, > > We recently started to receive logs from a Snare client. > I applied the configuration changes proposed here: > http://wiki.rsyslog.com/index.php/Using_Snare_as_a_client_on_Windows I just got a chance to read this, and I disagree with it's recommendation the #011 is a tab. snare uses it to seperate fields from the windows event log. if you just replace it with a space you loose the ability to do things like look for field 10 because field 8 may contain spaces. In addition there are two formats of logs that snare can output to syslog 1. the snare format 2. 'syslog' format the difference seems to be that in snare format it does Jan 1 01:01:01 mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri while in syslog format it does Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri the biggest change is the space between the host and MSWinEventLog there is also sometime a "1#011" before the field with Security in it in this example and sometimes not. I have not figured out the pattern here yet (it may be the format difference, my parsing logic just looks for a '1' in the first field, and if it finds it, drops that field) what I do is to setup the following rules to reformat the syslog formatted messages to look like Jan 1 01:01:01 mail.abc.com MSWinEventLog 1#011Security#0114169#011Fri I can then filter on the program name MSWinEventLog and later parsing tools can examine specific fields of the log message I have not fully figured out the snare formatted message, but the *2 formats at lease prevent lots of garbage from ending up in the host and program fields. $template fixsnareFormat,"%timereported% %HOSTNAME% MSWinEventLog %syslogtag:18:$%%msg:::drop-last-lf%\n" $template fixsnareForwardFormat,"<%pri%>%timereported% %HOSTNAME% MSWinEventLog %syslogtag:18:$:%%msg:::drop-last-lf%\n" $template fixsnareFormat2,"%timereported% %fromhost-ip% broken-MSWinEventLog %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" $template fixsnareForwardFormat2,"<%pri%>%timereported% %fromhost-ip% broken-MSWinEventLog %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 & @192.168.1.8;fixsnareForwardFormat2 & ~ :syslogtag, startswith, "MSWinEventLog#011" /var/log/messages;fixsnareFormat & @192.168.1.8;fixsnareForwardFormat & ~ *.* /var/log/messages;TraditionalFormat *.* @192.168.1.8 I updated to wiki to show this option > After restarting rsyslog, I get the following error: > -- > 2009-10-27T06:45:39.067279-04:00 logserger rsyslogd: [origin > software="rsyslogd" swVersion="4.2.0" x-pid="29021" > x-info="http://www.rsyslog.com"] (re)start > 2009-10-27T06:45:39.063213-04:00 logserger rsyslogd-3000: unknown > priority name "" [try http://www.rsyslog.com/e/3000 ] > 2009-10-27T06:45:39.063571-04:00 logserger rsyslogd: the last error > occured in /etc/rsyslog.conf, line 18 > 2009-10-27T06:45:39.064044-04:00 logserger rsyslogd: warning: selector > line without actions will be discarded > 2009-10-27T06:45:39.064213-04:00 logserger rsyslogd-2123: CONFIG ERROR: > could not interpret master config file '/etc/rsyslog.conf'. [try > http://www.rsyslog.com/e/2123 ] > -- > > If I comment out the "%msg:::space-cc%" part it starts up cleanly but I > assume that the filtering won't work. the %msg:::space-cc% shouldn't have been listed as a seperate line, that would replace %msg% in a format string. David Lang > > Suggestions? > > > Thanks, > Martin > > > > > > This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. > > Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Oct 27 22:09:58 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 27 Oct 2009 22:09:58 +0100 Subject: [rsyslog] control character escaping change? References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032D8@GRFEXC.intern.adiscon.com> This smells like a mistake. At least I don't remember changing that, but I'll look at the changelog tomorrow... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, October 27, 2009 9:06 PM > To: rsyslog-users > Subject: [rsyslog] control character escaping change? > > it appears that the 5.3 development series recently changed > the default > for escaping control characters from # to \ so a tab changed > from #011 to > \011 > > was this a deliberate change going forward, or a mistake? > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From r.bhatia at ipax.at Wed Oct 28 11:40:09 2009 From: r.bhatia at ipax.at (Raoul Bhatia [IPAX]) Date: Wed, 28 Oct 2009 11:40:09 +0100 Subject: [rsyslog] version numbers from git repository In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103292@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103292@GRFEXC.intern.adiscon.com> Message-ID: <4AE81F89.3070409@ipax.at> On 10/23/2009 07:54 AM, Rainer Gerhards wrote: > I've looked a bit into the issue. Neither of them use the GNU build system. > In essence, it boils down to running "git describe" as part of the build > process, generating a file with it. However, I do not know how I may achieve > that. It should probably be run during the ./configure stage. Maybe someone > can provide some advise ;) i like the way drbd includes the git-hash in its version output: > # cat /proc/drbd > version: 8.3.2 (api:88/proto:86-90) > GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by phil at fat-tyre, 2009-07-03 15:35:39 > ... take a look at line 112ff at (especially line 125) http://git.drbd.org/?p=drbd-8.3.git;a=blob;f=drbd/Makefile;hb=HEAD there, a drbd_buildtag.c file is created with the help of: > git rev-parse HEAD the output of this Makefile section is: > /* automatically generated. DO NOT EDIT. */ > #include > const char * drbd_buildtag(void) > { > return "GIT-hash: 9ce425f51860f1205395bf7127cad3c427070837" > " build by raoul at dev.ipax.at, 2008-09-04 17:00:24"; > } one can use this, or something similar using e.g. git-describe. cheers, raoul -- ____________________________________________________________________ DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at Technischer Leiter IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at Barawitzkagasse 10/2/2/11 email. office at ipax.at 1190 Wien tel. +43 1 3670030 FN 277995t HG Wien fax. +43 1 3670030 15 ____________________________________________________________________ From rgerhards at hq.adiscon.com Wed Oct 28 12:06:32 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 12:06:32 +0100 Subject: [rsyslog] version numbers from git repository References: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103292@GRFEXC.intern.adiscon.com> <4AE81F89.3070409@ipax.at> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032DC@GRFEXC.intern.adiscon.com> Hi Raoul, this is useful, but unfortunately it does not tell how to do it with GNU autotools. I'll keep this on my mind when I try to tackle it, but so far it looks like I need too much time to find out the details. If someone could suggest a solution within the GNU build system, I'd love to include that ASAP. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX] > Sent: Wednesday, October 28, 2009 11:40 AM > To: rsyslog-users > Subject: Re: [rsyslog] version numbers from git repository > > On 10/23/2009 07:54 AM, Rainer Gerhards wrote: > > I've looked a bit into the issue. Neither of them use the GNU build > system. > > In essence, it boils down to running "git describe" as part of the > build > > process, generating a file with it. However, I do not know how I may > achieve > > that. It should probably be run during the ./configure stage. Maybe > someone > > can provide some advise ;) > > i like the way drbd includes the git-hash in its version output: > > # cat /proc/drbd > > version: 8.3.2 (api:88/proto:86-90) > > GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by phil at fat- > tyre, 2009-07-03 15:35:39 > > ... > > take a look at line 112ff at (especially line 125) > http://git.drbd.org/?p=drbd-8.3.git;a=blob;f=drbd/Makefile;hb=HEAD > > there, a drbd_buildtag.c file is created with the help of: > > git rev-parse HEAD > > the output of this Makefile section is: > > /* automatically generated. DO NOT EDIT. */ > > #include > > const char * drbd_buildtag(void) > > { > > return "GIT-hash: 9ce425f51860f1205395bf7127cad3c427070837" > > " build by raoul at dev.ipax.at, 2008-09-04 17:00:24"; > > } > > one can use this, or something similar using e.g. git-describe. > > cheers, > raoul > -- > ____________________________________________________________________ > DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at > Technischer Leiter > > IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at > Barawitzkagasse 10/2/2/11 email. office at ipax.at > 1190 Wien tel. +43 1 3670030 > FN 277995t HG Wien fax. +43 1 3670030 15 > ____________________________________________________________________ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jbondc at openmv.com Wed Oct 28 12:32:35 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Wed, 28 Oct 2009 07:32:35 -0400 Subject: [rsyslog] Property-Based Filters, not working? In-Reply-To: References: <002101ca573d$5f417f70$1dc47e50$@com> Message-ID: <001b01ca57c2$593248a0$0b96d9e0$@com> On Tue Oct 27 04:03 PM, david at lang.hm wrote: > On Tue, 27 Oct 2009, Jonathan Bond-Caron wrote: > > > Date: Tue, 27 Oct 2009 15:40:42 -0400 > > From: Jonathan Bond-Caron > > Reply-To: rsyslog-users > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] Property-Based Filters, not working? > > > > I tried adding the following to my config file: > > > > > > :msg,contains,"MSWinEventLog" > > > > *.* /var/log/windows.log > > > > :msg, !contains, "MSWinEventLog" > > > > *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err > > /var/log/messages > > this is not how the property based filters work. > > I made the same mistake > > instead of > > :msg,contains,"MSWinEventLog" *.* /var/log/windows.log > > it should be > > :msg,contains,"MSWinEventLog" /var/log/windows.log > > when you do filter rules those filters are everything, you don't also > do facility/priority filters as well (or if you need to do so, you > need to make a compound if (() and ()) type rule) > Thanks a lot, that works well. The docs are bit misleading: http://www.rsyslog.com/doc-rsyslog_conf_filter.html It could show something like this: :msg, contains, "error" /var/log/error.log :msg, !contains, "error" ~ All messages that contain the word "error" are logged to /var/log/error.log All messages that do not contain the word "error" are thrown away. From rgerhards at hq.adiscon.com Wed Oct 28 12:35:25 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 12:35:25 +0100 Subject: [rsyslog] Property-Based Filters, not working? References: <002101ca573d$5f417f70$1dc47e50$@com> <001b01ca57c2$593248a0$0b96d9e0$@com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032DD@GRFEXC.intern.adiscon.com> > Thanks a lot, that works well. > > The docs are bit misleading: > http://www.rsyslog.com/doc-rsyslog_conf_filter.html > > It could show something like this: > :msg, contains, "error" /var/log/error.log > :msg, !contains, "error" ~ > > All messages that contain the word "error" are logged to > /var/log/error.log > All messages that do not contain the word "error" are thrown away. The doc in general could be much improved (any volunteers?), but here I think it is right. The link you quote defines what filters are, but what you post is not only a filter but a full selector line, consisting from a filter and the associated action. I agree that it would be useful to have more scenario-based cases which contain all the pieces put together (again, any volunteers?). Rainer From jbondc at openmv.com Wed Oct 28 13:25:40 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Wed, 28 Oct 2009 08:25:40 -0400 Subject: [rsyslog] Property-Based Filters, not working? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032DD@GRFEXC.intern.adiscon.com> References: <002101ca573d$5f417f70$1dc47e50$@com> <001b01ca57c2$593248a0$0b96d9e0$@com> <9B6E2A8877C38245BFB15CC491A11DA71032DD@GRFEXC.intern.adiscon.com> Message-ID: <001c01ca57c9$c3870810$4a951830$@com> On Wed Oct 28 07:35 AM, Rainer Gerhards wrote: > > Thanks a lot, that works well. > > > > The docs are bit misleading: > > http://www.rsyslog.com/doc-rsyslog_conf_filter.html > > > > It could show something like this: > > :msg, contains, "error" /var/log/error.log > > :msg, !contains, "error" ~ > > > > All messages that contain the word "error" are logged to > > /var/log/error.log All messages that do not contain the word "error" > > are thrown away. > > The doc in general could be much improved (any volunteers?), but here > I think it is right. The link you quote defines what filters are, but > what you post is not only a filter but a full selector line, > consisting from a filter and the associated action. > > I agree that it would be useful to have more scenario-based cases > which contain all the pieces put together (again, any volunteers?). Noted ;) I'll prepare a patch for 4.4.2 this week, will also fix an issue with the syslog 'tag' parsing. It should terminate when it sees a tab or escape control character. Makes it a little more RFC3164 compliant. From a.smith at ukgrid.net Wed Oct 28 13:53:21 2009 From: a.smith at ukgrid.net (Andy Smith) Date: Wed, 28 Oct 2009 12:53:21 +0000 Subject: [rsyslog] config prob with mail.info messages Message-ID: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> Hi, I?m finding rsyslog is logging all routine (I assume mail.info) mail messages to the main messages file (FreeBSD 7.0). Can someone suggest why based on the config: *.err;kern.warning;auth.notice;mail.crit;local7.none /dev/console *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err;local7.none /var/log/messages security.* /var/log/security auth.info;authpriv.info /var/log/auth.log mail.info /var/log/maillog lpr.info /var/log/lpd-errs ftp.info /var/log/xferlog cron.* /var/log/cron *.=debug;local7.none /var/log/debug.log *.emerg * local7.* :ommysql:127.0.0.1,Syslog,syslog,lisboa08 !startslip *.* /var/log/slip.log !ppp *.* /var/log/ppp.log The messages in the messages file dont include a facility or criticality value so makes it tough to work out. For example: 2009-10-28T12:21:36.782834+00:00 ourmailserver tpop3d[90983]: connections_post_select: client [3]dave at somedomain.co.uk(1.8.1.1): disconnected; 93/602717 bytes read/written Can anyone give me a clue?? thanks, Andy. From rgerhards at hq.adiscon.com Wed Oct 28 13:56:08 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 13:56:08 +0100 Subject: [rsyslog] config prob with mail.info messages References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032E2@GRFEXC.intern.adiscon.com> I have to admit I do not really understand the question, but to record severity and facility, you can follow this guide: http://www.rsyslog.com/doc-rsyslog_recording_pri.html HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Andy Smith > Sent: Wednesday, October 28, 2009 1:53 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] config prob with mail.info messages > > Hi, > > I?m finding rsyslog is logging all routine (I assume mail.info) > mail messages to the main messages file (FreeBSD 7.0). Can someone > suggest why based on the config: > > *.err;kern.warning;auth.notice;mail.crit;local7.none > /dev/console > *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err;local7.no > ne > /var/log/messages > security.* /var/log/security > auth.info;authpriv.info /var/log/auth.log > mail.info /var/log/maillog > lpr.info /var/log/lpd-errs > ftp.info /var/log/xferlog > cron.* /var/log/cron > *.=debug;local7.none > /var/log/debug.log > *.emerg * > local7.* :ommysql:127.0.0.1,Syslog,syslog,lisboa08 > !startslip > *.* /var/log/slip.log > !ppp > *.* /var/log/ppp.log > > The messages in the messages file dont include a facility or > criticality value so makes it tough to work out. For example: > > 2009-10-28T12:21:36.782834+00:00 ourmailserver tpop3d[90983]: > connections_post_select: client [3]dave at somedomain.co.uk(1.8.1.1): > disconnected; 93/602717 bytes read/written > > > Can anyone give me a clue?? > > thanks, Andy. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jbondc at openmv.com Wed Oct 28 14:04:24 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Wed, 28 Oct 2009 09:04:24 -0400 Subject: [rsyslog] rsyslog and snare In-Reply-To: References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> Message-ID: <002301ca57cf$2cece590$86c6b0b0$@com> On Tue Oct 27 04:32 PM, david at lang.hm wrote: > On Tue, 27 Oct 2009, Martin Mielke wrote: > > > Hi, > > > > We recently started to receive logs from a Snare client. > > I applied the configuration changes proposed here: > > http://wiki.rsyslog.com/index.php/Using_Snare_as_a_client_on_Windows > > I just got a chance to read this, and I disagree with it's > recommendation > > the #011 is a tab. > > snare uses it to seperate fields from the windows event log. > > if you just replace it with a space you loose the ability to do things > like look for field 10 because field 8 may contain spaces. > > In addition there are two formats of logs that snare can output to > syslog > > 1. the snare format > 2. 'syslog' format > > the difference seems to be that in snare format it does > > Jan 1 01:01:01 > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > > while in syslog format it does > > Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri > > Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric character will terminate the TAG field and will be assumed to be the starting character of the CONTENT field.) The result would be: TAG: '' MSG: '#011MSWinEventLog#0111#011Security#0114169#011Fri...' At least I think, any thoughts Rainer? The following works for me: // If first character not alpha-numeric, skip tag parsing if( !isalphanum((int)*p2parse) ) bTAGCharDetected = 1; Is this bad for other loggers? From rgerhards at hq.adiscon.com Wed Oct 28 14:12:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 14:12:53 +0100 Subject: [rsyslog] rsyslog and snare References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> <002301ca57cf$2cece590$86c6b0b0$@com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> > > Jan 1 01:01:01 > > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > > > > while in syslog format it does > > > > Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri > > > > First things first: I really don't like to dig into snare, as I do not think it is a really good solution (but how could the designer of http://www.eventreporter.com say otherwise? ;)). so I assume you are talking about this message: mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric > character > will terminate the TAG field and will be assumed to be the starting > character of the CONTENT field.) Another thing to put straight: RFC3164 is NOT a standard, it is an informational document that describes things seen on the wire. And it does not even describe all *popular* cases. If it all, it gains some standard function via RFC3195, but this is questionable. Just for the records ;) So the key point here is that we have nothing to "respect" or "obey to", but rather something that describes things seen in practice. For any decent syslog receiver, this means it must try to work equally well with everything that comes in via legacy syslog. But, granted, RFC3164 is useful if we have nothing else to look at. So let's do that. We'll immediately see that snare is broken, because no control characters are allowed in the hostname. So rsyslog does the right thing and escapes these characters. So everthing belongs to the hostname and thus you have problems with tag. Damn... I didn't want to say somethign about snare. Now it has happened again. My conclusion: fix snare or use something that works ;) Rainer > > The result would be: > TAG: '' > MSG: '#011MSWinEventLog#0111#011Security#0114169#011Fri...' > > At least I think, any thoughts Rainer? > > The following works for me: > // If first character not alpha-numeric, skip tag parsing > if( !isalphanum((int)*p2parse) ) > bTAGCharDetected = 1; > > Is this bad for other loggers? > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From a.smith at ukgrid.net Wed Oct 28 14:17:41 2009 From: a.smith at ukgrid.net (Andy Smith) Date: Wed, 28 Oct 2009 13:17:41 +0000 Subject: [rsyslog] config prob with mail.info messages In-Reply-To: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> Message-ID: <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> Hi Rainer, ok, the question is because based on my config I do not expect or want routine mail messages logged to my messages file, it is over run by messages from courier IMAP and TPOP3 daemons. When using the built in FreeBSD syslog these types of messages are written to the maillog log file, and I dont understand why the same isnt true of rsyslog given the config file I am using, thanks Andy. Quoting Rainer Gerhards: > I have to admit I do not really understand the question From rgerhards at hq.adiscon.com Wed Oct 28 14:43:43 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 14:43:43 +0100 Subject: [rsyslog] config prob with mail.info messages References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032E5@GRFEXC.intern.adiscon.com> Ah, I see. It would probably be useful to see the priority values, as specified in the other link I posted. Maybe the problem is rooted there. Ultimately, a debug log could bring the information how selectors are processed. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Andy Smith > Sent: Wednesday, October 28, 2009 2:18 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] config prob with mail.info messages > > Hi Rainer, > > ok, the question is because based on my config I do not expect or > want routine mail messages logged to my messages file, it is over run > by messages from courier IMAP and TPOP3 daemons. When using the built > in FreeBSD syslog these types of messages are written to the maillog > log file, and I dont understand why the same isnt true of rsyslog > given the config file I am using, > > thanks Andy. > > Quoting Rainer Gerhards: > > > I have to admit I do not really understand the question > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Oct 28 15:50:29 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 07:50:29 -0700 (PDT) Subject: [rsyslog] Property-Based Filters, not working? In-Reply-To: <001c01ca57c9$c3870810$4a951830$@com> References: <002101ca573d$5f417f70$1dc47e50$@com> <001b01ca57c2$593248a0$0b96d9e0$@com> <9B6E2A8877C38245BFB15CC491A11DA71032DD@GRFEXC.intern.adiscon.com> <001c01ca57c9$c3870810$4a951830$@com> Message-ID: On Wed, 28 Oct 2009, Jonathan Bond-Caron wrote: > On Wed Oct 28 07:35 AM, Rainer Gerhards wrote: >>> Thanks a lot, that works well. >>> >>> The docs are bit misleading: >>> http://www.rsyslog.com/doc-rsyslog_conf_filter.html >>> >>> It could show something like this: >>> :msg, contains, "error" /var/log/error.log >>> :msg, !contains, "error" ~ >>> >>> All messages that contain the word "error" are logged to >>> /var/log/error.log All messages that do not contain the word "error" >>> are thrown away. >> >> The doc in general could be much improved (any volunteers?), but here >> I think it is right. The link you quote defines what filters are, but >> what you post is not only a filter but a full selector line, >> consisting from a filter and the associated action. >> >> I agree that it would be useful to have more scenario-based cases >> which contain all the pieces put together (again, any volunteers?). > > Noted ;) > > I'll prepare a patch for 4.4.2 this week, will also fix an issue with the > syslog 'tag' parsing. It should terminate when it sees a tab or escape > control character. > > Makes it a little more RFC3164 compliant. it should also terminate parsing a hostname when it hits a tab (the RFC says it should be a space, but I've got one case (snare) that sometimes uses a tab. David Lang From david at lang.hm Wed Oct 28 15:53:32 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 07:53:32 -0700 (PDT) Subject: [rsyslog] rsyslog and snare In-Reply-To: <002301ca57cf$2cece590$86c6b0b0$@com> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> <002301ca57cf$2cece590$86c6b0b0$@com> Message-ID: On Wed, 28 Oct 2009, Jonathan Bond-Caron wrote: > On Tue Oct 27 04:32 PM, david at lang.hm wrote: >> On Tue, 27 Oct 2009, Martin Mielke wrote: >> >>> Hi, >>> >>> We recently started to receive logs from a Snare client. >>> I applied the configuration changes proposed here: >>> http://wiki.rsyslog.com/index.php/Using_Snare_as_a_client_on_Windows >> >> I just got a chance to read this, and I disagree with it's >> recommendation >> >> the #011 is a tab. >> >> snare uses it to seperate fields from the windows event log. >> >> if you just replace it with a space you loose the ability to do things >> like look for field 10 because field 8 may contain spaces. >> >> In addition there are two formats of logs that snare can output to >> syslog >> >> 1. the snare format >> 2. 'syslog' format >> >> the difference seems to be that in snare format it does >> >> Jan 1 01:01:01 >> mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri >> >> while in syslog format it does >> >> Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri >> >> > > Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric character > will terminate the TAG field and will be assumed to be the starting > character of the CONTENT field.) > > The result would be: > TAG: '' > MSG: '#011MSWinEventLog#0111#011Security#0114169#011Fri...' > > At least I think, any thoughts Rainer? > > The following works for me: > // If first character not alpha-numeric, skip tag parsing > if( !isalphanum((int)*p2parse) ) > bTAGCharDetected = 1; > > Is this bad for other loggers? if it stops on the tab it should end up hostname =vmail.abc.com then it hits a tab, so stops putting things as part of the hostname tag =vMSWinEventLog then it hits a tab, so stops putting things as part of the tag message = 1#011Security#0114169#011Fri... David Lang From david at lang.hm Wed Oct 28 15:56:48 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 07:56:48 -0700 (PDT) Subject: [rsyslog] rsyslog and snare In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> <002301ca57cf$2cece590$86c6b0b0$@com> <9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 28 Oct 2009, Rainer Gerhards wrote: >>> Jan 1 01:01:01 >>> mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri >>> >>> while in syslog format it does >>> >>> Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri >>> >>> > > First things first: I really don't like to dig into snare, as I do not think > it is a really good solution (but how could the designer of > http://www.eventreporter.com say otherwise? ;)). > > so I assume you are talking about this message: > > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > >> Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric >> character >> will terminate the TAG field and will be assumed to be the starting >> character of the CONTENT field.) > > Another thing to put straight: RFC3164 is NOT a standard, it is an > informational document that describes things seen on the wire. And it does > not even describe all *popular* cases. If it all, it gains some standard > function via RFC3195, but this is questionable. Just for the records ;) > > So the key point here is that we have nothing to "respect" or "obey to", but > rather something that describes things seen in practice. For any decent > syslog receiver, this means it must try to work equally well with everything > that comes in via legacy syslog. But, granted, RFC3164 is useful if we have > nothing else to look at. > > So let's do that. We'll immediately see that snare is broken, because no > control characters are allowed in the hostname. So rsyslog does the right > thing and escapes these characters. So everthing belongs to the hostname and > thus you have problems with tag. two other valid behaviors 1. it has a control character in the first text field, so that cannot be a hostname or a tag, so it must be part of the message (after doing the escaping) 2. treat running into a tab like running into a space I think #1 would be better than what we do today, but #2 would be the best for users. David Lang > Damn... I didn't want to say somethign about snare. Now it has happened > again. My conclusion: fix snare or use something that works ;) > > Rainer > >> >> The result would be: >> TAG: '' >> MSG: '#011MSWinEventLog#0111#011Security#0114169#011Fri...' >> >> At least I think, any thoughts Rainer? >> >> The following works for me: >> // If first character not alpha-numeric, skip tag parsing >> if( !isalphanum((int)*p2parse) ) >> bTAGCharDetected = 1; >> >> Is this bad for other loggers? >> >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 28 16:00:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 16:00:12 +0100 Subject: [rsyslog] rsyslog and snare References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local><002301ca57cf$2cece590$86c6b0b0$@com><9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032E8@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, October 28, 2009 3:57 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog and snare > > On Wed, 28 Oct 2009, Rainer Gerhards wrote: > > >>> Jan 1 01:01:01 > >>> mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > >>> > >>> while in syslog format it does > >>> > >>> Jan 1 01:01:01 mail.abc.com > MSWinEventLog#011Security#0114169#011Fri > >>> > >>> > > > > First things first: I really don't like to dig into snare, as I do > not think > > it is a really good solution (but how could the designer of > > http://www.eventreporter.com say otherwise? ;)). > > > > so I assume you are talking about this message: > > > > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > > > >> Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric > >> character > >> will terminate the TAG field and will be assumed to be the starting > >> character of the CONTENT field.) > > > > Another thing to put straight: RFC3164 is NOT a standard, it is an > > informational document that describes things seen on the wire. And it > does > > not even describe all *popular* cases. If it all, it gains some > standard > > function via RFC3195, but this is questionable. Just for the records > ;) > > > > So the key point here is that we have nothing to "respect" or "obey > to", but > > rather something that describes things seen in practice. For any > decent > > syslog receiver, this means it must try to work equally well with > everything > > that comes in via legacy syslog. But, granted, RFC3164 is useful if > we have > > nothing else to look at. > > > > So let's do that. We'll immediately see that snare is broken, because > no > > control characters are allowed in the hostname. So rsyslog does the > right > > thing and escapes these characters. So everthing belongs to the > hostname and > > thus you have problems with tag. > > two other valid behaviors > > 1. it has a control character in the first text field, so that cannot > be a > hostname or a tag, so it must be part of the message (after doing the > escaping) > > 2. treat running into a tab like running into a space > > I think #1 would be better than what we do today, but #2 would be the > best > for users. The problem with all that is that it breaks message sanitation. When a message hits rsyslog, it first is sanitized (which is important for security reasons, e.g. to prevent NUL characters to make parts of the messages unreadable and, later, to prevent the myriad of Unicode-based Vulnerabilites). So when it hits the parser, the is no such thing like a HT present anymore. What we could do, however, is add an option that tells the sanitizer to replace HT by SP in all cases. Rainer From david at lang.hm Wed Oct 28 16:08:22 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 08:08:22 -0700 (PDT) Subject: [rsyslog] rsyslog and snare In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032E8@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local><002301ca57cf$2cece590$86c6b0b0$@com><9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71032E8@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 28 Oct 2009, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >> two other valid behaviors >> >> 1. it has a control character in the first text field, so that cannot >> be a >> hostname or a tag, so it must be part of the message (after doing the >> escaping) >> >> 2. treat running into a tab like running into a space >> >> I think #1 would be better than what we do today, but #2 would be the >> best >> for users. > > The problem with all that is that it breaks message sanitation. When a > message hits rsyslog, it first is sanitized (which is important for security > reasons, e.g. to prevent NUL characters to make parts of the messages > unreadable and, later, to prevent the myriad of Unicode-based > Vulnerabilites). > > So when it hits the parser, the is no such thing like a HT present anymore. > What we could do, however, is add an option that tells the sanitizer to > replace HT by SP in all cases. this is not as good because the tabs are useful in the message itself, it acts as a field seperator for the different fields from the windows logs, and the fields themselves can contain spaces, so if you replace them with tabs you no longer have any way to identify individual fields. the parser could look for # (which is not a valid character in a hostname, and I don't think it's valid in a tag), then if it's #011 treat is as a space for seperating the hoatname/tag/message and if it's anything else, make that the start of the message. David Lang From david at lang.hm Wed Oct 28 16:08:55 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 08:08:55 -0700 (PDT) Subject: [rsyslog] control character escaping change? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032D8@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71032D8@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 27 Oct 2009, Rainer Gerhards wrote: > This smells like a mistake. At least I don't remember changing that, but I'll > look at the changelog tomorrow... did you get a chance to look at this? David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Tuesday, October 27, 2009 9:06 PM >> To: rsyslog-users >> Subject: [rsyslog] control character escaping change? >> >> it appears that the 5.3 development series recently changed >> the default >> for escaping control characters from # to \ so a tab changed >> from #011 to >> \011 >> >> was this a deliberate change going forward, or a mistake? >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From epiphani at gmail.com Wed Oct 28 16:14:03 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Wed, 28 Oct 2009 11:14:03 -0400 Subject: [rsyslog] Queue sizing Message-ID: Greetings, I'm trying to get a good handle on queue sizing and memory usage. In a higher-traffic environment, I have a dedicated rsyslog machine with a large amount of ram. I want to try and find the right balance between large queues for burst traffic, but I also want to make sure that the queue sizes are small enough that we don't run out of memory and crash. I'm using 4.5.5 currently. I'm currently using disk-backed memory queues, configured in fixedarray, like so: $MainMsgQueueSize 2000000 $MainMsgQueueMaxFileSize 50g $MainMsgQueueSaveOnShutdown on $MainMsgQueueHighWaterMark 15000000 $MainMsgQueueLowWaterMark 10000000 $MainMsgQueueType FixedArray $MainMsgQueueWorkerThreads 2 $ActionQueueSize 200000 $ActionQueueMaxFileSize 50g $ActionQueueSaveOnShutdown on $ActionQueueHighWaterMark 1500000 $ActionQueueLowWaterMark 1000000 $ActionQueueType FixedArray $ActionQueueWorkerThreads 1 I usually have upwards of a dozen action queue definitions, and my omfile outputs are based on hostname of the incoming connection. Is there some way I can calculate maximum memory use by the queuing system, or force the system to a set memory maximum where I -know- that it will simply start pushing back on the incoming connection instead of adding more data to the system? Thoughts? -Aaron From rgerhards at hq.adiscon.com Wed Oct 28 16:25:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 16:25:27 +0100 Subject: [rsyslog] control character escaping change? References: <9B6E2A8877C38245BFB15CC491A11DA71032D8@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032EB@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, October 28, 2009 4:09 PM > To: rsyslog-users > Subject: Re: [rsyslog] control character escaping change? > > On Tue, 27 Oct 2009, Rainer Gerhards wrote: > > > This smells like a mistake. At least I don't remember changing that, > but I'll > > look at the changelog tomorrow... > > did you get a chance to look at this? yes, sorry, this is a mistake. I'll fix it soon, but today (and maybe up to the end of the week), I am working on a journal article whom's deadline *is* end of the week ;) Rainer From rgerhards at hq.adiscon.com Wed Oct 28 16:28:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 16:28:53 +0100 Subject: [rsyslog] rsyslog and snare References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local><002301ca57cf$2cece590$86c6b0b0$@com><9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71032E8@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032EC@GRFEXC.intern.adiscon.com> > > So when it hits the parser, the is no such thing like a HT present > anymore. > > What we could do, however, is add an option that tells the sanitizer > to > > replace HT by SP in all cases. > > this is not as good because the tabs are useful in the message itself, > it > acts as a field seperator for the different fields from the windows > logs, > and the fields themselves can contain spaces, so if you replace them > with > tabs you no longer have any way to identify individual fields. > > the parser could look for # (which is not a valid character in a > hostname, > and I don't think it's valid in a tag), then if it's #011 treat is as a > space for seperating the hoatname/tag/message and if it's anything > else, > make that the start of the message. Yeah, but (not only because it is snare ;)) wouldn't it be the right thing to do to fix the broken sender? Of course, I can add another exception AND "#" may validly be inside a tag. I am all for trying to guess right, but keep in mind that all this also has a performance toll. One of my plans is to add a custom parser interface. With it, custom parsers could be added and they could take care of all the anomalies of the real world (and there *are* many!). But that is up for another increment, after the queue work. I don't like to change the parsers that have proven to work well in practice just because one application misbehaves... Rainer From david at lang.hm Wed Oct 28 17:15:17 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 09:15:17 -0700 (PDT) Subject: [rsyslog] rsyslog and snare In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032EC@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local><002301ca57cf$2cece590$86c6b0b0$@com><9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71032E8@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71032EC@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 28 Oct 2009, Rainer Gerhards wrote: >>> So when it hits the parser, the is no such thing like a HT present >> anymore. >>> What we could do, however, is add an option that tells the sanitizer >> to >>> replace HT by SP in all cases. >> >> this is not as good because the tabs are useful in the message itself, >> it >> acts as a field seperator for the different fields from the windows >> logs, >> and the fields themselves can contain spaces, so if you replace them >> with >> tabs you no longer have any way to identify individual fields. >> >> the parser could look for # (which is not a valid character in a >> hostname, >> and I don't think it's valid in a tag), then if it's #011 treat is as a >> space for seperating the hoatname/tag/message and if it's anything >> else, >> make that the start of the message. > > Yeah, but (not only because it is snare ;)) wouldn't it be the right thing to > do to fix the broken sender? yes it would, but doing so is much harder than it should be :-( David Lang > Of course, I can add another exception AND "#" > may validly be inside a tag. I am all for trying to guess right, but keep in > mind that all this also has a performance toll. > > One of my plans is to add a custom parser interface. With it, custom parsers > could be added and they could take care of all the anomalies of the real > world (and there *are* many!). But that is up for another increment, after > the queue work. > > I don't like to change the parsers that have proven to work well in practice > just because one application misbehaves... > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From a.smith at ukgrid.net Wed Oct 28 19:30:59 2009 From: a.smith at ukgrid.net (Andy Smith) Date: Wed, 28 Oct 2009 18:30:59 +0000 Subject: [rsyslog] config prob with mail.info messages In-Reply-To: <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> Message-ID: <20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> Hi Rainer, ok thanks, I turned on the custom template as described in the link you posted and could immediately see that the events were being logged as mail.notice, which strangely isnt specifically configured in the FreeBSD built in syslog conf but these are none the less logged to maillog instead of messages. Anyway, thanks to knowing this I have been easily able to add an extra line in the config to direct these to the maillog as I want :) thanks again! Andy. Quoting Rainer Gerhards: > Ah, I see. It would probably be useful to see the priority values, as > specified in the other link I posted. From anichols at trumped.org Wed Oct 28 20:00:22 2009 From: anichols at trumped.org (Aaron Nichols) Date: Wed, 28 Oct 2009 13:00:22 -0600 Subject: [rsyslog] config prob with mail.info messages In-Reply-To: <20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> <20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> Message-ID: On Wed, Oct 28, 2009 at 12:30 PM, Andy Smith wrote: > ? ok thanks, I turned on the custom template as described in the link > you posted and could immediately see that the events were being logged > as mail.notice, which strangely isnt specifically configured in the > FreeBSD built in syslog conf but these are none the less logged to > maillog instead of messages. Your original configuration contained this: *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err;local7.none /var/log/messages If you have messages being delivered to mail.notice they will match the "*.notice" selector at the beginning of that line. It's easy enough to filter them out as you've found, but I figured I'd point out that the configuration did actually specify that these should be sent to /var/log/messages. Aaron From rgerhards at hq.adiscon.com Wed Oct 28 21:02:45 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 21:02:45 +0100 Subject: [rsyslog] config prob with mail.info messages References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net><20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net><20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032EF@GRFEXC.intern.adiscon.com> Aaron, Good catch - I overlooked that :) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Wednesday, October 28, 2009 8:00 PM > To: rsyslog-users > Subject: Re: [rsyslog] config prob with mail.info messages > > On Wed, Oct 28, 2009 at 12:30 PM, Andy Smith > wrote: > > ? ok thanks, I turned on the custom template as described > in the link > > you posted and could immediately see that the events were > being logged > > as mail.notice, which strangely isnt specifically configured in the > > FreeBSD built in syslog conf but these are none the less logged to > > maillog instead of messages. > > Your original configuration contained this: > *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err; > local7.none > /var/log/messages > > If you have messages being delivered to mail.notice they will match > the "*.notice" selector at the beginning of that line. It's easy > enough to filter them out as you've found, but I figured I'd point out > that the configuration did actually specify that these should be sent > to /var/log/messages. > > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From a.smith at ukgrid.net Wed Oct 28 21:51:44 2009 From: a.smith at ukgrid.net (Andy Smith) Date: Wed, 28 Oct 2009 20:51:44 +0000 Subject: [rsyslog] config prob with mail.info messages In-Reply-To: <20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> <20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> Message-ID: <20091028205144.17377b2djm644r8c@horde.ukgrid.net> Ah yep all is clear now, thanks Aaron, well spotted as Rainer said :P From martinmie at PartyGaming.com Sat Oct 31 02:28:22 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Sat, 31 Oct 2009 02:28:22 +0100 Subject: [rsyslog] rsyslog and snare In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local><002301ca57cf$2cece590$86c6b0b0$@com> <9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> Message-ID: <0B1B9163138571439EAF804646F3F6AA0892F6F9@GIBSVWIN004X.partygaming.local> Semi off-topic: release EventReporter under the same license as rsyslog and you'll win a lot of happy friends ;-) Cheers > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: 28 October 2009 14:13 > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog and snare > > > > Jan 1 01:01:01 > > > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > > > > > > while in syslog format it does > > > > > > Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri > > > > > > > > First things first: I really don't like to dig into snare, as I do not > think > it is a really good solution (but how could the designer of > http://www.eventreporter.com say otherwise? ;)). > > so I assume you are talking about this message: > > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > > > Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric > > character > > will terminate the TAG field and will be assumed to be the starting > > character of the CONTENT field.) > > Another thing to put straight: RFC3164 is NOT a standard, it is an > informational document that describes things seen on the wire. And it does > not even describe all *popular* cases. If it all, it gains some standard > function via RFC3195, but this is questionable. Just for the records ;) > > So the key point here is that we have nothing to "respect" or "obey to", > but > rather something that describes things seen in practice. For any decent > syslog receiver, this means it must try to work equally well with > everything > that comes in via legacy syslog. But, granted, RFC3164 is useful if we > have > nothing else to look at. > > So let's do that. We'll immediately see that snare is broken, because no > control characters are allowed in the hostname. So rsyslog does the right > thing and escapes these characters. So everthing belongs to the hostname > and > thus you have problems with tag. > > Damn... I didn't want to say somethign about snare. Now it has happened > again. My conclusion: fix snare or use something that works ;) > > Rainer > > > > > The result would be: > > TAG: '' > > MSG: '#011MSWinEventLog#0111#011Security#0114169#011Fri...' > > > > At least I think, any thoughts Rainer? > > > > The following works for me: > > // If first character not alpha-numeric, skip tag parsing > > if( !isalphanum((int)*p2parse) ) > > bTAGCharDetected = 1; > > > > Is this bad for other loggers? > > > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > /rsyslog > http://www.rsyslog.com This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From igalvarez at gmail.com Thu Oct 1 05:42:56 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Wed, 30 Sep 2009 22:42:56 -0500 Subject: [rsyslog] sending logs outside /etc/rsyslog.conf to remote server Message-ID: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> I've some daemons with their logs not define in /etc/rsyslog.conf and files outside /var/log folder. I have a central rsyslog receiving logs from severral servers. My question is: How can setup rsyslog on clients to send these logs outside /etc/rsyslog.conf to the central rsyslog server? Is this possible? -- Regards; Israel Garcia From david at lang.hm Thu Oct 1 06:17:19 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 21:17:19 -0700 (PDT) Subject: [rsyslog] sending logs outside /etc/rsyslog.conf to remote server In-Reply-To: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> References: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> Message-ID: On Wed, 30 Sep 2009, Israel Garcia wrote: > I've some daemons with their logs not define in /etc/rsyslog.conf and > files outside /var/log folder. I have a central rsyslog receiving logs > from severral servers. > > My question is: > > How can setup rsyslog on clients to send these logs outside > /etc/rsyslog.conf to the central rsyslog server? Is this possible? the first problem is to get the messages to rsyslog. if the applications don't have an option to log to syslog you can use the imfile input module in rsyslog to read lines from the file and process them. David Lang From rgerhards at hq.adiscon.com Thu Oct 1 11:01:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 1 Oct 2009 11:01:17 +0200 Subject: [rsyslog] DNS cache and expiration References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103110@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710311A@GRFEXC.intern.adiscon.com> I have now reviewed the discussion in depth. Many thanks for all the good thoughts. I will now first begin to move the name resolution calls to the "backend" threads (away from the inputs). However, I will create an option that the name resolution may be done before enqueing the message, which may be used by some folks for whatever reason (plus, it is trivially to do). I also think I will leave in the capability that inputs do the name resolution themselfs, at least if they prefer. For example, imuxsock always has the local host as input, so it makes limited sense to do name resolution calls for messages received from that input. Similarly, imtcp uses the name that was (once) obtained when the session was created (btw, this is also some expiry-less "name resolution", but I think here it is even harder to argue against it ;)). When I am done with this, I have a nice central place where the name resolution is done. In that place, I will then build a central name cache, at first with only two options - cache disabled - cache without expiration (requires shutdown) After that, I'll probably look into HUP plus probably provide a maximum TTL to rsyslog that says e.g. "check entry after n seconds". Even if n is low (maybe 10 seconds), the extra effort for doing these lookups is minimal: If the system has low throughput, the lookup cost is irrelevant in any case. If it is a high performance system, with more than e.g. 10,000 msgs/s, then n=10 means we do one lookup for at most 1/100,000th message, which doesn't matter either. Most importantly, I do not like the idea to use a third-party resolver library, as this creates another set of dependencies, and this for a core module. Booting rsyslog would get even more complicated. So I think the "expire ever n seconds" approach is a very practical one. You will see move these changes gradually into the master branch. One word on the performance: an avl-tree based internal cache can resolve names *very* fast. It has a depth of almost log n where n being the number of names cached. Building and maintaining the tree is almost for free, because the structure is rather static. Expiration, as I intend to implement it, is also a very simple operation, the node does not need to be removed and re-inserted but rather I need to update the resolution info by means of a new lookup (what costs the majority of time). So for a typical search, I would assume that the name can be resolved with less than 500 machine instructions (what I consider a conservative choice). The way I intend to implement it probably does not include any mutex calls, so there is no context switch involved. I consider the cost of a single context switch to be much higher than such an avl-tree lookup. Now consider any external solution. You need to do at least four context switches (rsyslog -> OS -> resolver -> OS -> rsyslog), probably more. Also, at least the OS scheduler is involved to activate the threads that are synchronized, plus probably a bit more of OS logic. Then, the external resolver can at best perform at the same rate rsyslog itself can (using a similar algorithm), but probably slower because it must be more generic. Looking at this picture, I think, clarifies why an (proper) internal implementation is so much more efficient than any external solution. Also note that I assume that in almost all cases, due to the structure of syslog traffic, no real lookup needs to be done, everything sits already in the resolver caches. I hope this is useful information. If someone has any additional comments, I would love to hear them. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Wednesday, September 30, 2009 9:57 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Wednesday, September 30, 2009 9:53 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] DNS cache and expiration > > > > On Wed, 30 Sep 2009, Rainer Gerhards wrote: > > > > > Thanks for the good discussion. I am lacking somewhat > > behind, but will review > > > it in depth tomorrow morning. > > > > > > I just wanted to stress the point that an external cache > > does not really > > > help, much for the reason David mentioned: if you process > > messages at very > > > high data rates, the context switch overhead involved with > > any external > > > solution is extremely costly. Also, in the usual cases, I > > may do several > > > million queries within a few seconds for just a handful of > > hosts. With an > > > internal cache, the overhead in doing so is very minimal. > > With an external > > > solution, the overhead in calling the external cache causes a lot > of > > > performance degredation, what in the case of UDP also > > implies (heavy!) > > > message loss. > > > > the message loss problem with UDP will not be solved completely by an > > internal cache. when the source is not in the cache and you > > have to go out > > to find it the lookup can take several seconds. > > > > moving the lookup out of the input module and into the output > > module would > > address this, anything else would leave you with losses as > > the cache gets > > populated. > > That's right and that's one reason why I intend to move this > (optionally) > over to the "backend" processing. However, even that does not > completely > solve the message loss problem, as we, in extreme cases, may loose > messages > when the queue is full - and for a myriad of other reasons, like > routers > discarding frames and such. Of course, you know that, but I'd like to > mention > if for those folks that at some time find our conversation via Google > ;) > > Rainer > > > > David Lang > > > > > Rainer > > > > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com > > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > david at lang.hm > > >> Sent: Wednesday, September 30, 2009 8:54 PM > > >> To: rsyslog-users > > >> Subject: Re: [rsyslog] DNS cache and expiration > > >> > > >> On Wed, 30 Sep 2009, Kenneth Marshall wrote: > > >> > > >>>>> mechanism to map IP addresses to names for the purposes > > of logging > > >>>>> error messages. As such, pretty much the only piece > > that needs to > > >>>>> be tracked within rsyslog is the TTL for the entry and the ip - > > > > >>>>> name mapping. A thread would be responsible for expiring > entries > > >>>>> from the cache (or refreshing the timeout) after validating the > > >>>>> correctness of the mapping. I think the DNS lookups should be > > >>>>> handled by a good resolver like pdns-recursor, djbdns,... The > > >>>>> goal here is to allow names in the log entries and not just IP > > >>>>> addresses and in a very high performance logging environment. > > >>>> > > >>>> the trouble is that doing _proper_ TTL expiration isn't as > > >> simple as it > > >>>> sounds. > > >>>> > > >>>> and if you are willing to back away from 'proper' > > >> expiration to something > > >>>> that will work in practice, why not go much further (as I > > >> have detailed in > > >>>> the other messages) > > >>>> > > >>>> David Lang > > >>> > > >>> I agree. I only mention TTL values as a reasonable > > upperbound of the > > >>> refresh check. The advantage is to remove the large pause > > in logging > > >>> due to a DNS refresh after a HUP to rsyslog, if that were the > only > > >>> method to flush/refresh. Like you mention, the IPs/names > > of systems > > >>> being logged change rarely so we should tune this for speed and > > >>> not worry about expiration correctness. I do not agree so other > > >>> statements that an external cache will perform as well as > > >> an internal > > >>> cache. Too many software products that I work with have > > >> needed exactly > > >>> that functionality to support very high levels of performance. > > >> > > >> actually, you could have the cache be configurable in three modes > > >> > > >> 1. no caching > > >> > > >> 2. blank the cache on HUP > > >> > > >> 3. never blank the cache (i.e. require a full restart to clear it) > > >> > > >> David Lang > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From igalvarez at gmail.com Thu Oct 1 17:09:04 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Thu, 1 Oct 2009 10:09:04 -0500 Subject: [rsyslog] sending logs outside /etc/rsyslog.conf to remote server In-Reply-To: References: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> Message-ID: <194a2c240910010809k99cd7e7o46d40e428923912f@mail.gmail.com> On 9/30/09, david at lang.hm wrote: > On Wed, 30 Sep 2009, Israel Garcia wrote: > >> I've some daemons with their logs not define in /etc/rsyslog.conf and >> files outside /var/log folder. I have a central rsyslog receiving logs >> from severral servers. >> >> My question is: >> >> How can setup rsyslog on clients to send these logs outside >> /etc/rsyslog.conf to the central rsyslog server? Is this possible? > > the first problem is to get the messages to rsyslog. > > if the applications don't have an option to log to syslog you can use the > imfile input module in rsyslog to read lines from the file and process > them. Hi David, Has anybody on the list setup apache, proftpd, momgrel or any other daemon logs to forward to a centralized syslog server? I'd appreciate your experiences. regards, Israel. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From rgerhards at hq.adiscon.com Thu Oct 1 17:25:24 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 1 Oct 2009 17:25:24 +0200 Subject: [rsyslog] sending logs outside /etc/rsyslog.conf to remoteserver References: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> <194a2c240910010809k99cd7e7o46d40e428923912f@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710312D@GRFEXC.intern.adiscon.com> no experience on my side, but I wanted to make you aware of the wiki, which (I think) contains a number of samples: http://wiki.rsyslog.com Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Israel Garcia > Sent: Thursday, October 01, 2009 5:09 PM > To: rsyslog-users > Subject: Re: [rsyslog] sending logs outside /etc/rsyslog.conf to > remoteserver > > On 9/30/09, david at lang.hm wrote: > > On Wed, 30 Sep 2009, Israel Garcia wrote: > > > >> I've some daemons with their logs not define in /etc/rsyslog.conf > and > >> files outside /var/log folder. I have a central rsyslog receiving > logs > >> from severral servers. > >> > >> My question is: > >> > >> How can setup rsyslog on clients to send these logs outside > >> /etc/rsyslog.conf to the central rsyslog server? Is this possible? > > > > the first problem is to get the messages to rsyslog. > > > > if the applications don't have an option to log to syslog you can use > the > > imfile input module in rsyslog to read lines from the file and > process > > them. > > Hi David, > > Has anybody on the list setup apache, proftpd, momgrel or any other > daemon logs to forward to a centralized syslog server? I'd appreciate > your experiences. > > regards, > Israel. > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > -- > Regards; > Israel Garcia > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Thu Oct 1 19:06:56 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 1 Oct 2009 10:06:56 -0700 (PDT) Subject: [rsyslog] sending logs outside /etc/rsyslog.conf to remote server In-Reply-To: <194a2c240910010809k99cd7e7o46d40e428923912f@mail.gmail.com> References: <194a2c240909302042j3a556dc0v13a0d9fce9ebbd36@mail.gmail.com> <194a2c240910010809k99cd7e7o46d40e428923912f@mail.gmail.com> Message-ID: On Thu, 1 Oct 2009, Israel Garcia wrote: > On 9/30/09, david at lang.hm wrote: >> On Wed, 30 Sep 2009, Israel Garcia wrote: >> >>> I've some daemons with their logs not define in /etc/rsyslog.conf and >>> files outside /var/log folder. I have a central rsyslog receiving logs >>> from severral servers. >>> >>> My question is: >>> >>> How can setup rsyslog on clients to send these logs outside >>> /etc/rsyslog.conf to the central rsyslog server? Is this possible? >> >> the first problem is to get the messages to rsyslog. >> >> if the applications don't have an option to log to syslog you can use the >> imfile input module in rsyslog to read lines from the file and process >> them. > > Hi David, > > Has anybody on the list setup apache, proftpd, momgrel or any other > daemon logs to forward to a centralized syslog server? I'd appreciate > your experiences. I haven't done it yet, but this page makes it seem pretty easy to setup apache http://www.oreillynet.com/pub/a/sysadmin/2006/10/12/httpd-syslog.html I've had proftpd send things to syslog in the past (IIRC it was a trivial config option) no expriance with mongrel David Lang From david at lang.hm Thu Oct 1 21:26:23 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 1 Oct 2009 12:26:23 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710311A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103110@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710311A@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 1 Oct 2009, Rainer Gerhards wrote: > I have now reviewed the discussion in depth. Many thanks for all the good > thoughts. > > I will now first begin to move the name resolution calls to the "backend" > threads (away from the inputs). However, I will create an option that the > name resolution may be done before enqueing the message, which may be used by > some folks for whatever reason (plus, it is trivially to do). I also think I > will leave in the capability that inputs do the name resolution themselfs, at > least if they prefer. For example, imuxsock always has the local host as > input, so it makes limited sense to do name resolution calls for messages > received from that input. Similarly, imtcp uses the name that was (once) > obtained when the session was created (btw, this is also some expiry-less > "name resolution", but I think here it is even harder to argue against it > ;)). on note on this, if you can delay the resolution to the point where something actually accesses the property that uses the name you may be able to skip doing name resolution entirely in many cases. David Lang > When I am done with this, I have a nice central place where the name > resolution is done. In that place, I will then build a central name cache, at > first with only two options > > - cache disabled > - cache without expiration (requires shutdown) > > After that, I'll probably look into HUP plus probably provide a maximum TTL > to rsyslog that says e.g. "check entry after n seconds". Even if n is low > (maybe 10 seconds), the extra effort for doing these lookups is minimal: > > If the system has low throughput, the lookup cost is irrelevant in any case. > If it is a high performance system, with more than e.g. 10,000 msgs/s, then > n=10 means we do one lookup for at most 1/100,000th message, which doesn't > matter either. > > Most importantly, I do not like the idea to use a third-party resolver > library, as this creates another set of dependencies, and this for a core > module. Booting rsyslog would get even more complicated. So I think the > "expire ever n seconds" approach is a very practical one. > > You will see move these changes gradually into the master branch. > > One word on the performance: an avl-tree based internal cache can resolve > names *very* fast. It has a depth of almost log n where n being the number of > names cached. Building and maintaining the tree is almost for free, because > the structure is rather static. Expiration, as I intend to implement it, is > also a very simple operation, the node does not need to be removed and > re-inserted but rather I need to update the resolution info by means of a new > lookup (what costs the majority of time). So for a typical search, I would > assume that the name can be resolved with less than 500 machine instructions > (what I consider a conservative choice). The way I intend to implement it > probably does not include any mutex calls, so there is no context switch > involved. > > I consider the cost of a single context switch to be much higher than such an > avl-tree lookup. Now consider any external solution. You need to do at least > four context switches (rsyslog -> OS -> resolver -> OS -> rsyslog), probably > more. Also, at least the OS scheduler is involved to activate the threads > that are synchronized, plus probably a bit more of OS logic. Then, the > external resolver can at best perform at the same rate rsyslog itself can > (using a similar algorithm), but probably slower because it must be more > generic. Looking at this picture, I think, clarifies why an (proper) internal > implementation is so much more efficient than any external solution. Also > note that I assume that in almost all cases, due to the structure of syslog > traffic, no real lookup needs to be done, everything sits already in the > resolver caches. > > I hope this is useful information. If someone has any additional comments, I > would love to hear them. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Wednesday, September 30, 2009 9:57 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] DNS cache and expiration >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com >>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Wednesday, September 30, 2009 9:53 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] DNS cache and expiration >>> >>> On Wed, 30 Sep 2009, Rainer Gerhards wrote: >>> >>>> Thanks for the good discussion. I am lacking somewhat >>> behind, but will review >>>> it in depth tomorrow morning. >>>> >>>> I just wanted to stress the point that an external cache >>> does not really >>>> help, much for the reason David mentioned: if you process >>> messages at very >>>> high data rates, the context switch overhead involved with >>> any external >>>> solution is extremely costly. Also, in the usual cases, I >>> may do several >>>> million queries within a few seconds for just a handful of >>> hosts. With an >>>> internal cache, the overhead in doing so is very minimal. >>> With an external >>>> solution, the overhead in calling the external cache causes a lot >> of >>>> performance degredation, what in the case of UDP also >>> implies (heavy!) >>>> message loss. >>> >>> the message loss problem with UDP will not be solved completely by an >>> internal cache. when the source is not in the cache and you >>> have to go out >>> to find it the lookup can take several seconds. >>> >>> moving the lookup out of the input module and into the output >>> module would >>> address this, anything else would leave you with losses as >>> the cache gets >>> populated. >> >> That's right and that's one reason why I intend to move this >> (optionally) >> over to the "backend" processing. However, even that does not >> completely >> solve the message loss problem, as we, in extreme cases, may loose >> messages >> when the queue is full - and for a myriad of other reasons, like >> routers >> discarding frames and such. Of course, you know that, but I'd like to >> mention >> if for those folks that at some time find our conversation via Google >> ;) >> >> Rainer >>> >>> David Lang >>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com >>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >>> david at lang.hm >>>>> Sent: Wednesday, September 30, 2009 8:54 PM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] DNS cache and expiration >>>>> >>>>> On Wed, 30 Sep 2009, Kenneth Marshall wrote: >>>>> >>>>>>>> mechanism to map IP addresses to names for the purposes >>> of logging >>>>>>>> error messages. As such, pretty much the only piece >>> that needs to >>>>>>>> be tracked within rsyslog is the TTL for the entry and the ip - >>> >>>>>>>> name mapping. A thread would be responsible for expiring >> entries >>>>>>>> from the cache (or refreshing the timeout) after validating the >>>>>>>> correctness of the mapping. I think the DNS lookups should be >>>>>>>> handled by a good resolver like pdns-recursor, djbdns,... The >>>>>>>> goal here is to allow names in the log entries and not just IP >>>>>>>> addresses and in a very high performance logging environment. >>>>>>> >>>>>>> the trouble is that doing _proper_ TTL expiration isn't as >>>>> simple as it >>>>>>> sounds. >>>>>>> >>>>>>> and if you are willing to back away from 'proper' >>>>> expiration to something >>>>>>> that will work in practice, why not go much further (as I >>>>> have detailed in >>>>>>> the other messages) >>>>>>> >>>>>>> David Lang >>>>>> >>>>>> I agree. I only mention TTL values as a reasonable >>> upperbound of the >>>>>> refresh check. The advantage is to remove the large pause >>> in logging >>>>>> due to a DNS refresh after a HUP to rsyslog, if that were the >> only >>>>>> method to flush/refresh. Like you mention, the IPs/names >>> of systems >>>>>> being logged change rarely so we should tune this for speed and >>>>>> not worry about expiration correctness. I do not agree so other >>>>>> statements that an external cache will perform as well as >>>>> an internal >>>>>> cache. Too many software products that I work with have >>>>> needed exactly >>>>>> that functionality to support very high levels of performance. >>>>> >>>>> actually, you could have the cache be configurable in three modes >>>>> >>>>> 1. no caching >>>>> >>>>> 2. blank the cache on HUP >>>>> >>>>> 3. never blank the cache (i.e. require a full restart to clear it) >>>>> >>>>> David Lang >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu Oct 1 21:27:40 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 1 Oct 2009 12:27:40 -0700 (PDT) Subject: [rsyslog] fromhost-ip not working with -x Message-ID: I am running all my servers with -x to disable name resolution. I have found that fromhost exists, but fromhost-ip does not this is either a bug in the documentation, or fromhost-ip doesn't get created if no name resolution is done. david Lang From rgerhards at hq.adiscon.com Thu Oct 1 21:45:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 1 Oct 2009 21:45:56 +0200 Subject: [rsyslog] DNS cache and expiration References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103110@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710311A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103133@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, October 01, 2009 9:26 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > On Thu, 1 Oct 2009, Rainer Gerhards wrote: > > > I have now reviewed the discussion in depth. Many thanks > for all the good > > thoughts. > > > > I will now first begin to move the name resolution calls to > the "backend" > > threads (away from the inputs). However, I will create an > option that the > > name resolution may be done before enqueing the message, > which may be used by > > some folks for whatever reason (plus, it is trivially to > do). I also think I > > will leave in the capability that inputs do the name > resolution themselfs, at > > least if they prefer. For example, imuxsock always has the > local host as > > input, so it makes limited sense to do name resolution > calls for messages > > received from that input. Similarly, imtcp uses the name > that was (once) > > obtained when the session was created (btw, this is also > some expiry-less > > "name resolution", but I think here it is even harder to > argue against it > > ;)). > > on note on this, if you can delay the resolution to the point where > something actually accesses the property that uses the name > you may be > able to skip doing name resolution entirely in many cases. This I plan to do, but in the TCP case I think it probably is better to do a name resolution once per *session* even if the name may never be used. Typically, there are many messages in a session and the overhead if the one name lookup does not really matter. At least this is my current thinking. Rainer From david at lang.hm Thu Oct 1 22:07:31 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 1 Oct 2009 13:07:31 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103133@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103110@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710311A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103133@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 1 Oct 2009, Rainer Gerhards wrote: >> -----Original Message----- >> On Thu, 1 Oct 2009, Rainer Gerhards wrote: >> >>> I have now reviewed the discussion in depth. Many thanks >> for all the good >>> thoughts. >>> >>> I will now first begin to move the name resolution calls to >> the "backend" >>> threads (away from the inputs). However, I will create an >> option that the >>> name resolution may be done before enqueing the message, >> which may be used by >>> some folks for whatever reason (plus, it is trivially to >> do). I also think I >>> will leave in the capability that inputs do the name >> resolution themselfs, at >>> least if they prefer. For example, imuxsock always has the >> local host as >>> input, so it makes limited sense to do name resolution >> calls for messages >>> received from that input. Similarly, imtcp uses the name >> that was (once) >>> obtained when the session was created (btw, this is also >> some expiry-less >>> "name resolution", but I think here it is even harder to >> argue against it >>> ;)). >> >> on note on this, if you can delay the resolution to the point where >> something actually accesses the property that uses the name >> you may be >> able to skip doing name resolution entirely in many cases. > > This I plan to do, but in the TCP case I think it probably is better to do a > name resolution once per *session* even if the name may never be used. > Typically, there are many messages in a session and the overhead if the one > name lookup does not really matter. At least this is my current thinking. no problem with filling it out earlier if the info happens to be available (or in the case of thinks like TCP, RELP, etc where it really is static). I'm thinking of things like my central server that receives the aggregated feed from several relays. everything that it does is based on the name in the log, and since the logs come from other rsyslog relay boxes, those boxes will have cleaned things up, so there is never (outside of debugging) a need for that central box to do _any_ lookups, no matter how fast they end up being. David Lang From anichols at trumped.org Fri Oct 2 01:52:22 2009 From: anichols at trumped.org (Aaron Nichols) Date: Thu, 1 Oct 2009 17:52:22 -0600 Subject: [rsyslog] "too many tcp sessions - dropping incoming request" - tried increasing max sessions to no avail Message-ID: Hi all, I am seeing the following in my logs running rsyslog 4.4.1: Oct 1 23:29:24 host rsyslogd-2079: too many tcp sessions - dropping incoming request [try http://www.rsyslog.com/e/2079 ] Oct 1 23:29:29 host rsyslogd-2079: too many tcp sessions - dropping incoming request [try http://www.rsyslog.com/e/2079 ] Oct 1 23:29:38 host rsyslogd-2079: too many tcp sessions - dropping incoming request [try http://www.rsyslog.com/e/2079 ] I went to the URL in the message and the answer makes sense so I added the following to my configuration: $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) I added it before the imtcp module is loaded so the config looks like: # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) $ModLoad imtcp.so # load module $InputTCPServerRun 514 # start up TCP listener at port 514 I am still seeing the "too many tcp sessions" in the logs. When I take a look at netstat (assuming this is the relevant data) I see less than 1000 *connections* - but I'm not sure if each of these is considered a "session" $ netstat -an | grep :514 | wc -l 269 If I look for only ESTABLISHED connections I see less: [anichols at inlwr01p1 inlwr01p1]$ netstat -an | grep :514 | grep ESTABLISHED | wc -l 231 Many of the connections show a SYN_RECV state on the server side - I assume these are cases where we aren't completing the tcp handshake due to too many connections. Are there other factors which may be impacting the max connections or is my configuration incorrect? Thanks, Aaron From anichols at trumped.org Fri Oct 2 04:44:50 2009 From: anichols at trumped.org (Aaron Nichols) Date: Thu, 1 Oct 2009 20:44:50 -0600 Subject: [rsyslog] "too many tcp sessions - dropping incoming request" - tried increasing max sessions to no avail In-Reply-To: References: Message-ID: I found the cause of my problem I believe - the $InputTCPMaxSessions value needs to come after the imtcp module is loaded. This makes sense in hindsight. Of course, the evidence was right in front of me had I looked in the right place: Oct 2 02:34:54 host rsyslogd-3003: invalid or yet-unknown config file command - have you forgotten to load a module? [try http://www.rsyslog.com/e/3003 ] Oct 2 02:34:54 host rsyslogd: the last error occured in /etc/rsyslog.d/rsyslog-server.conf, line 4 I have to thank you guys for making an effort to provide informative errors with URL references. Thanks, Aaron On Thu, Oct 1, 2009 at 5:52 PM, Aaron Nichols wrote: > Hi all, > > I am seeing the following in my logs running rsyslog 4.4.1: > > Oct 1 23:29:24 host rsyslogd-2079: too many tcp sessions - dropping > incoming request [try http://www.rsyslog.com/e/2079 ] > Oct 1 23:29:29 host rsyslogd-2079: too many tcp sessions - dropping > incoming request [try http://www.rsyslog.com/e/2079 ] > Oct 1 23:29:38 host rsyslogd-2079: too many tcp sessions - dropping > incoming request [try http://www.rsyslog.com/e/2079 ] > > I went to the URL in the message and the answer makes sense so I added the > following to my configuration: > > $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) > > I added it before the imtcp module is loaded so the config looks like: > > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) > $ModLoad imtcp.so # load module > $InputTCPServerRun 514 # start up TCP listener at port 514 > > I am still seeing the "too many tcp sessions" in the logs. When I take a > look at netstat (assuming this is the relevant data) I see less than 1000 > *connections* - but I'm not sure if each of these is considered a "session" > > $ netstat -an | grep :514 | wc -l > 269 > > If I look for only ESTABLISHED connections I see less: > > [anichols at inlwr01p1 inlwr01p1]$ netstat -an | grep :514 | grep ESTABLISHED > | wc -l > 231 > > Many of the connections show a SYN_RECV state on the server side - I assume > these are cases where we aren't completing the tcp handshake due to too many > connections. > > Are there other factors which may be impacting the max connections or is my > configuration incorrect? > > Thanks, > Aaron > From tbergfeld at hq.adiscon.com Mon Oct 5 15:13:52 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Mon, 5 Oct 2009 15:13:52 +0200 Subject: [rsyslog] rsyslog 5.3.1 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710315D@GRFEXC.intern.adiscon.com> Hi all, We have just released 5.3.1, a member of the devel branch. This is a bug-fixing release, providing some important fixes from the v4-stable and the current v4-beta, like a bugfix for a potential segfault in stream writer on destruction. This is a recommended update for all users of the devel branch. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article407.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-178.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From salim.engin at gmail.com Wed Oct 7 15:23:21 2009 From: salim.engin at gmail.com (Salim Engin) Date: Wed, 7 Oct 2009 16:23:21 +0300 Subject: [rsyslog] Sender IP or Hostname Message-ID: <43660480910070623s6eeb20b3gf200de18cff7d553@mail.gmail.com> Hello all, I have just installed Debain 5.0.3 with its default syslog server which is rsyslog 3.18.6 . and I configured it to receive external syslog messages from our network devices. What the problem is that, the sender IP or hostname of external devices which send syslog messages is not displayed in the log . Please see the example below; 2009-10-07T16:09:39.407770+03:00 debian dhcpd: DHCPACK on 172.10.10.101 to 00:24:7e:11:eb:49 (stajerstac) via vlan110 --------> "debian" hostname is OK 2009-09-06T14:04:53+03:00 - rCT0[184]: craft(8) : RTRV-HDR:::P6099; -----------------------------------------------------------------------------------> Neither IP nor hostname is printed 2009-09-06T14:04:56+03:00 - clij[258]: isadmin at 10.5.5.23(21) : info configure system sntp ----------------------------------------------------> Neither IP nor hostname is printed. Instead there is dash after the timestamp. I tried many things but was not able to make it work. Can someone help me to fix this ? Thanks Salim Engin From rgerhards at hq.adiscon.com Wed Oct 7 15:33:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 7 Oct 2009 15:33:17 +0200 Subject: [rsyslog] Sender IP or Hostname References: <43660480910070623s6eeb20b3gf200de18cff7d553@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103191@GRFEXC.intern.adiscon.com> Hi Salim, sounds like a malformed message is causing this: http://www.rsyslog.com/doc-syslog_parsing.html Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Salim Engin > Sent: Wednesday, October 07, 2009 3:23 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Sender IP or Hostname > > Hello all, > > I have just installed Debain 5.0.3 with its default syslog server which > is > rsyslog 3.18.6 . and I configured it to receive external syslog > messages > from our network devices. > > What the problem is that, the sender IP or hostname of external devices > which send syslog messages is not displayed in the log . Please see the > example below; > > > 2009-10-07T16:09:39.407770+03:00 debian dhcpd: DHCPACK on 172.10.10.101 > to > 00:24:7e:11:eb:49 (stajerstac) via vlan110 --------> "debian" hostname > is > OK > 2009-09-06T14:04:53+03:00 - rCT0[184]: craft(8) : RTRV-HDR:::P6099; > ----------------------------------------------------------------------- > ------------> > Neither IP nor hostname is printed > 2009-09-06T14:04:56+03:00 - clij[258]: isadmin at 10.5.5.23(21) : info > configure system sntp ------------------------------------------------- > ---> > Neither IP nor hostname is printed. Instead there is dash after the > timestamp. > > > I tried many things but was not able to make it work. > > Can someone help me to fix this ? > > Thanks > > Salim Engin > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From iamsayan at gmail.com Wed Oct 7 16:35:55 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Wed, 7 Oct 2009 20:05:55 +0530 Subject: [rsyslog] limiting message size while forwarding messages Message-ID: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> Hello All,RFC 3164 says that max rsyslog message length is 1024 bytes. Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 and has a default max message size of 2K. Now, I don't want to restrict the size while receiving messages.However when I forward message out of the box, I want to truncate it to 1K, as I am not sure what implementations I will have to interop with. I could not find an easy way to do this. $MaxMessageSize seems to control both sending and receiving. Is there a way in which I can achieve this? Regards, Sayan From rgerhards at hq.adiscon.com Wed Oct 7 16:43:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 7 Oct 2009 16:43:56 +0200 Subject: [rsyslog] limiting message size while forwarding messages References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> you need to look at the property replacer, and do something like %msg:1:950% in your template. I am not sure, but I think the default template already does this. Did you experience oversize messages on a receiver? > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Wednesday, October 07, 2009 4:36 PM > To: rsyslog-users > Subject: [rsyslog] limiting message size while forwarding messages > > Hello All,RFC 3164 says that max rsyslog message length is 1024 bytes. > Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 and > has a > default max message size of 2K. > Now, I don't want to restrict the size while receiving messages.However > when > I forward message out of the box, I want to truncate it > to 1K, as I am not sure what implementations I will have to interop > with. > I could not find an easy way to do this. $MaxMessageSize seems to > control > both sending and receiving. > Is there a way in which I can achieve this? > Regards, > Sayan > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From dirk.schulz at kinzesberg.de Wed Oct 7 16:31:34 2009 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Wed, 07 Oct 2009 16:31:34 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG Message-ID: <4ACCA646.8050209@kinzesberg.de> Hi folks, I am not sure if this is the right list to ask, but I hope that there is more of the related knowledge here. I have several Syslog-NG servers reporting to Rsyslog central servers (just because nobody could tell me how to run Rsyslog on NetBSD ...). Now I fould that the facility is not transferred to/evaluated on the Rsyslog servers; e.g. mail messages are written to syslog instead of maillog on the Rsyslog machines. Is there something special I have to configure on the Rsyslog servers or the Syslog-NG servers to fix that? Any hint or help is appreciated. Dirk From iamsayan at gmail.com Wed Oct 7 16:53:08 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Wed, 7 Oct 2009 20:23:08 +0530 Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> thanks Rainer, I will try that.So that means if I use the default template for forwarding, i think it should just work then. I did actually get a message from an application running on my box(uses standard syslog api) which was around 4K in size rsyslog did the right thing and truncated it to 2K.and logged into the file as per the configured rule. now I have requirement to support messages up to 5K while logging to the local file, but truncating it to 1K while sending it out. On Wed, Oct 7, 2009 at 8:13 PM, Rainer Gerhards wrote: > you need to look at the property replacer, and do something like > %msg:1:950% > in your template. I am not sure, but I think the default template already > does this. Did you experience oversize messages on a receiver? > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > Sent: Wednesday, October 07, 2009 4:36 PM > > To: rsyslog-users > > Subject: [rsyslog] limiting message size while forwarding messages > > > > Hello All,RFC 3164 says that max rsyslog message length is 1024 bytes. > > Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 and > > has a > > default max message size of 2K. > > Now, I don't want to restrict the size while receiving messages.However > > when > > I forward message out of the box, I want to truncate it > > to 1K, as I am not sure what implementations I will have to interop > > with. > > I could not find an easy way to do this. $MaxMessageSize seems to > > control > > both sending and receiving. > > Is there a way in which I can achieve this? > > Regards, > > Sayan > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 7 18:32:52 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 7 Oct 2009 18:32:52 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG References: <4ACCA646.8050209@kinzesberg.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dirk H. Schulz > Sent: Wednesday, October 07, 2009 4:32 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Getting Facility from Synlog-NG > > Hi folks, > > I am not sure if this is the right list to ask, but I hope that there > is > more of the related knowledge here. > > I have several Syslog-NG servers reporting to Rsyslog central servers > (just because nobody could tell me how to run Rsyslog on NetBSD ...). > Now I fould that the facility is not transferred to/evaluated on the > Rsyslog servers; e.g. mail messages are written to syslog instead of > maillog on the Rsyslog machines. This is strange. Facility is a fairly standard field. Could you tell me the content of %rawmwg%. This is done along these lines (if it does not work, check doc for exact syntax): $template raw,"%rawmsg%\n" *.* /path/to/file;raw Rainer > > Is there something special I have to configure on the Rsyslog servers > or > the Syslog-NG servers to fix that? > > Any hint or help is appreciated. > > Dirk > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 7 18:37:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 7 Oct 2009 18:37:17 +0200 Subject: [rsyslog] limiting message size while forwarding messages References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> I just checked, and the default is to send the full message. See: http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/syslogd.c;h=0f4f8a233c97 a44e72c97a555ff6f95eb367573a;hb=HEAD#l382 But all you need to do is use %msg:1:950...% in a template that otherwise is the same as the canned template. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Wednesday, October 07, 2009 4:53 PM > To: rsyslog-users > Subject: Re: [rsyslog] limiting message size while forwarding messages > > thanks Rainer, I will try that.So that means if I use the default > template > for forwarding, i think it should just work then. I did actually get a > message from an application running on my box(uses standard syslog api) > which was around 4K in size > rsyslog did the right thing and truncated it to 2K.and logged into the > file > as per the configured rule. > > now I have requirement to support messages up to 5K while logging to > the > local file, but truncating it to 1K while sending it out. > > > On Wed, Oct 7, 2009 at 8:13 PM, Rainer Gerhards > wrote: > > > you need to look at the property replacer, and do something like > > %msg:1:950% > > in your template. I am not sure, but I think the default template > already > > does this. Did you experience oversize messages on a receiver? > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > Sent: Wednesday, October 07, 2009 4:36 PM > > > To: rsyslog-users > > > Subject: [rsyslog] limiting message size while forwarding messages > > > > > > Hello All,RFC 3164 says that max rsyslog message length is 1024 > bytes. > > > Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 > and > > > has a > > > default max message size of 2K. > > > Now, I don't want to restrict the size while receiving > messages.However > > > when > > > I forward message out of the box, I want to truncate it > > > to 1K, as I am not sure what implementations I will have to interop > > > with. > > > I could not find an easy way to do this. $MaxMessageSize seems to > > > control > > > both sending and receiving. > > > Is there a way in which I can achieve this? > > > Regards, > > > Sayan > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From salim.engin at gmail.com Wed Oct 7 19:00:38 2009 From: salim.engin at gmail.com (Salim Engin) Date: Wed, 7 Oct 2009 20:00:38 +0300 Subject: [rsyslog] Sender IP or Hostname In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103191@GRFEXC.intern.adiscon.com> References: <43660480910070623s6eeb20b3gf200de18cff7d553@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103191@GRFEXC.intern.adiscon.com> Message-ID: <43660480910071000o6dbd760cn66ee567ccb844c28@mail.gmail.com> That is right. I configured an other networking device to use rsyslog and saw that the sender IP is displayed in the log file. But the WA mentioned in the document did not work ... Thanks On Wed, Oct 7, 2009 at 4:33 PM, Rainer Gerhards wrote: > Hi Salim, > > sounds like a malformed message is causing this: > > http://www.rsyslog.com/doc-syslog_parsing.html > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Salim Engin > > Sent: Wednesday, October 07, 2009 3:23 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] Sender IP or Hostname > > > > Hello all, > > > > I have just installed Debain 5.0.3 with its default syslog server which > > is > > rsyslog 3.18.6 . and I configured it to receive external syslog > > messages > > from our network devices. > > > > What the problem is that, the sender IP or hostname of external devices > > which send syslog messages is not displayed in the log . Please see the > > example below; > > > > > > 2009-10-07T16:09:39.407770+03:00 debian dhcpd: DHCPACK on 172.10.10.101 > > to > > 00:24:7e:11:eb:49 (stajerstac) via vlan110 --------> "debian" hostname > > is > > OK > > 2009-09-06T14:04:53+03:00 - rCT0[184]: craft(8) : RTRV-HDR:::P6099; > > ----------------------------------------------------------------------- > > ------------> > > Neither IP nor hostname is printed > > 2009-09-06T14:04:56+03:00 - clij[258]: isadmin at 10.5.5.23(21) : info > > configure system sntp ------------------------------------------------- > > ---> > > Neither IP nor hostname is printed. Instead there is dash after the > > timestamp. > > > > > > I tried many things but was not able to make it work. > > > > Can someone help me to fix this ? > > > > Thanks > > > > Salim Engin > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 7 19:05:28 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 7 Oct 2009 10:05:28 -0700 (PDT) Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> Message-ID: On Wed, 7 Oct 2009, Sayan Chowdhury wrote: > thanks Rainer, I will try that.So that means if I use the default template > for forwarding, i think it should just work then. I did actually get a > message from an application running on my box(uses standard syslog api) > which was around 4K in size > rsyslog did the right thing and truncated it to 2K.and logged into the file > as per the configured rule. > > now I have requirement to support messages up to 5K while logging to the > local file, but truncating it to 1K while sending it out. RFC 3164 says that a syslog implementation should truncate any oversized messages that it gets, so it should be safe to send a 2k message to a syslog daemon that truncates it to 1k note that in rsyslog the max message size is configrable. you may want to just set it to 1k until everything in your infrastructure handles larger messages (unless you want rsyslog to get the full message and send a portion of it, in which case you may want to set it larger) David Lang > > On Wed, Oct 7, 2009 at 8:13 PM, Rainer Gerhards wrote: > >> you need to look at the property replacer, and do something like >> %msg:1:950% >> in your template. I am not sure, but I think the default template already >> does this. Did you experience oversize messages on a receiver? >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury >>> Sent: Wednesday, October 07, 2009 4:36 PM >>> To: rsyslog-users >>> Subject: [rsyslog] limiting message size while forwarding messages >>> >>> Hello All,RFC 3164 says that max rsyslog message length is 1024 bytes. >>> Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 and >>> has a >>> default max message size of 2K. >>> Now, I don't want to restrict the size while receiving messages.However >>> when >>> I forward message out of the box, I want to truncate it >>> to 1K, as I am not sure what implementations I will have to interop >>> with. >>> I could not find an easy way to do this. $MaxMessageSize seems to >>> control >>> both sending and receiving. >>> Is there a way in which I can achieve this? >>> Regards, >>> Sayan >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iamsayan at gmail.com Wed Oct 7 20:11:49 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Wed, 7 Oct 2009 23:41:49 +0530 Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> Yes, I saw that too ... I made the template based out of Rsyslog traditional forward format and then used the property replacer as you mentioned. It works like a charm. Thanks a bunch! Regards, Sayan On Wed, Oct 7, 2009 at 10:07 PM, Rainer Gerhards wrote: > I just checked, and the default is to send the full message. See: > > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/syslogd.c;h=0f4f8a233c97 > a44e72c97a555ff6f95eb367573a;hb=HEAD#l382 > > But all you need to do is use %msg:1:950...% in a template that otherwise > is > the same as the canned template. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > Sent: Wednesday, October 07, 2009 4:53 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] limiting message size while forwarding messages > > > > thanks Rainer, I will try that.So that means if I use the default > > template > > for forwarding, i think it should just work then. I did actually get a > > message from an application running on my box(uses standard syslog api) > > which was around 4K in size > > rsyslog did the right thing and truncated it to 2K.and logged into the > > file > > as per the configured rule. > > > > now I have requirement to support messages up to 5K while logging to > > the > > local file, but truncating it to 1K while sending it out. > > > > > > On Wed, Oct 7, 2009 at 8:13 PM, Rainer Gerhards > > wrote: > > > > > you need to look at the property replacer, and do something like > > > %msg:1:950% > > > in your template. I am not sure, but I think the default template > > already > > > does this. Did you experience oversize messages on a receiver? > > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > > Sent: Wednesday, October 07, 2009 4:36 PM > > > > To: rsyslog-users > > > > Subject: [rsyslog] limiting message size while forwarding messages > > > > > > > > Hello All,RFC 3164 says that max rsyslog message length is 1024 > > bytes. > > > > Rsyslog version 4.2.0 which I am using, is compliant to rfc 5424 > > and > > > > has a > > > > default max message size of 2K. > > > > Now, I don't want to restrict the size while receiving > > messages.However > > > > when > > > > I forward message out of the box, I want to truncate it > > > > to 1K, as I am not sure what implementations I will have to interop > > > > with. > > > > I could not find an easy way to do this. $MaxMessageSize seems to > > > > control > > > > both sending and receiving. > > > > Is there a way in which I can achieve this? > > > > Regards, > > > > Sayan > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From jbondc at openmv.com Thu Oct 8 03:25:21 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Wed, 7 Oct 2009 21:25:21 -0400 Subject: [rsyslog] Queue System Message-ID: <007901ca47b6$34ac0870$9e041950$@com> Hi everyone, I'm been experimenting with rsyslogd to see if I could avoid losing messages/events when the central server is rebooted. I'll start with what works, then what doesn't L Both the server & client are running rsyslogd 3.22.1 The client has the following configuration: $ModLoad imklog.so # kernel logging $ModLoad imuxsock.so # provides support for local system logging $WorkDirectory /var/spool/rsyslog $ActionQueueType Disk $ActionQueueFileName srvrfwd $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on *.err;*.crit;*.emerg;*.warn;auth.*;user.notice @@server.domain.com # Stop the central server [root at server] $ service rsyslogd stop # Start client logger [root at client] $ service rsyslogd start ** (client) syslog(test a, NOTICE) ** (client) syslog(test b, NOTICE) Since the central server is down I expect that the client will keep messages in the disk queue and send them when the server is back online. # Start the central server [root at server] $ service rsyslogd start . 2009-10-08T00:44:21-04:00 client php: test a 2009-10-08T00:45:39-04:00 client php: test b . Great it works! ****** Now the problem, when client and server are both connected and exchanging messages # Stop the central server [root at server] $ service rsyslogd stop ** (client) syslog(test c - lost message, NOTICE) With some debugging: 4241.782860452:action 10 queue:Reg/w0: action 10 queue: entry deleted, state 0, size now 0 entries 4241.782860452:action 10 queue:Reg/w0: server 4241.782860452:action 10 queue:Reg/w0: server:10514/tcp 4241.782860452:action 10 queue:Reg/w0: TCP sent 36 bytes, requested 36 The client 'sends the message over tcp' and actually empties the queue! Is this expected behavior? rsyslogd was shutdown 'cleanly' on the server. Shouldn't it force the client to close the TCP connection? Using netstat, the two servers are : Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 client.10514 server..60726 FIN_WAIT_2 Proto Recv-Q Send-Q Local Address Foreign Address State tcp 1 36 server:60726 client:10514 CLOSE_WAIT Have I misunderstood something? I've read Rainer's blog and issues with TCP "unreliability" / case of a power failure but don't think this related From david at lang.hm Thu Oct 8 06:58:21 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 7 Oct 2009 21:58:21 -0700 (PDT) Subject: [rsyslog] Queue System In-Reply-To: <007901ca47b6$34ac0870$9e041950$@com> References: <007901ca47b6$34ac0870$9e041950$@com> Message-ID: On Wed, 7 Oct 2009, Jonathan Bond-Caron wrote: > The client 'sends the message over tcp' and actually empties the queue! > > Is this expected behavior? rsyslogd was shutdown 'cleanly' on the server. > Shouldn't it force the client to close the TCP connection? > > > > Using netstat, the two servers are : > > > > Proto Recv-Q Send-Q Local Address Foreign Address (state) > > tcp4 0 0 client.10514 server..60726 FIN_WAIT_2 > > > > Proto Recv-Q Send-Q Local Address Foreign Address > State > > tcp 1 36 server:60726 client:10514 CLOSE_WAIT > > > > Have I misunderstood something? I've read Rainer's blog and issues with TCP > "unreliability" / case of a power failure but don't think this related I think it's the same problem, one side thinks the connection is closed, the other doesn't and so the one that doesn't happily sends the data out. try reconfiguring to use relp and things should work as expected. now, I don't know why the close isn't getting through from the one system to the other rapidly. is there any sort of firewall in between them? David Lang From rgerhards at hq.adiscon.com Thu Oct 8 09:27:30 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 09:27:30 +0200 Subject: [rsyslog] limiting message size while forwarding messages References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com><87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Wednesday, October 07, 2009 8:12 PM > To: rsyslog-users > Subject: Re: [rsyslog] limiting message size while forwarding messages > > Yes, I saw that too ... I made the template based out of Rsyslog > traditional > forward format and then used the property replacer as you mentioned. > It works like a charm. Thanks a bunch! I am glad it works, but I also tend to think that the default template is probably not correct. As it claims to be "traditional" format, I think it should really limit itself to 1K message size, so that another template must be picked if messages of "non-traditional large size" are to be transmitted. However, changing that default would potentially break a number of existing deployments. Does anybody else have an opinion on that? Thanks, Rainer > Regards, > Sayan > > > On Wed, Oct 7, 2009 at 10:07 PM, Rainer Gerhards > wrote: > > > I just checked, and the default is to send the full message. See: > > > > > > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/syslogd.c;h=0f4f8a > 233c97 > > a44e72c97a555ff6f95eb367573a;hb=HEAD#l382 > > > > But all you need to do is use %msg:1:950...% in a template that > otherwise > > is > > the same as the canned template. > > > > Rainer > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > Sent: Wednesday, October 07, 2009 4:53 PM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] limiting message size while forwarding > messages > > > > > > thanks Rainer, I will try that.So that means if I use the default > > > template > > > for forwarding, i think it should just work then. I did actually > get a > > > message from an application running on my box(uses standard syslog > api) > > > which was around 4K in size > > > rsyslog did the right thing and truncated it to 2K.and logged into > the > > > file > > > as per the configured rule. > > > > > > now I have requirement to support messages up to 5K while logging > to > > > the > > > local file, but truncating it to 1K while sending it out. > > > > > > > > > On Wed, Oct 7, 2009 at 8:13 PM, Rainer Gerhards > > > wrote: > > > > > > > you need to look at the property replacer, and do something like > > > > %msg:1:950% > > > > in your template. I am not sure, but I think the default template > > > already > > > > does this. Did you experience oversize messages on a receiver? > > > > > > > > > -----Original Message----- > > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > > > Sent: Wednesday, October 07, 2009 4:36 PM > > > > > To: rsyslog-users > > > > > Subject: [rsyslog] limiting message size while forwarding > messages > > > > > > > > > > Hello All,RFC 3164 says that max rsyslog message length is 1024 > > > bytes. > > > > > Rsyslog version 4.2.0 which I am using, is compliant to rfc > 5424 > > > and > > > > > has a > > > > > default max message size of 2K. > > > > > Now, I don't want to restrict the size while receiving > > > messages.However > > > > > when > > > > > I forward message out of the box, I want to truncate it > > > > > to 1K, as I am not sure what implementations I will have to > interop > > > > > with. > > > > > I could not find an easy way to do this. $MaxMessageSize seems > to > > > > > control > > > > > both sending and receiving. > > > > > Is there a way in which I can achieve this? > > > > > Regards, > > > > > Sayan > > > > > _______________________________________________ > > > > > rsyslog mailing list > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > http://www.rsyslog.com > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Thu Oct 8 09:41:16 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 8 Oct 2009 00:41:16 -0700 (PDT) Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com><87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 8 Oct 2009, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury >> >> Yes, I saw that too ... I made the template based out of Rsyslog >> traditional >> forward format and then used the property replacer as you mentioned. >> It works like a charm. Thanks a bunch! > > I am glad it works, but I also tend to think that the default template is > probably not correct. As it claims to be "traditional" format, I think it > should really limit itself to 1K message size, so that another template must > be picked if messages of "non-traditional large size" are to be transmitted. > > However, changing that default would potentially break a number of existing > deployments. > > Does anybody else have an opinion on that? existing syslog implementations already need to be able to deal with oversized messages (almost nothing checks the size before it gets sent), so I don't see a big benifit in changing rsyslog to limit what it outputs. David Lang From rgerhards at hq.adiscon.com Thu Oct 8 09:43:47 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 09:43:47 +0200 Subject: [rsyslog] Queue System References: <007901ca47b6$34ac0870$9e041950$@com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710319B@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, October 08, 2009 6:58 AM > To: rsyslog-users > Subject: Re: [rsyslog] Queue System > > On Wed, 7 Oct 2009, Jonathan Bond-Caron wrote: > > > The client 'sends the message over tcp' and actually empties the > queue! > > > > Is this expected behavior? rsyslogd was shutdown 'cleanly' on the > server. > > Shouldn't it force the client to close the TCP connection? > > > > > > > > Using netstat, the two servers are : > > > > > > > > Proto Recv-Q Send-Q Local Address Foreign Address > (state) > > > > tcp4 0 0 client.10514 server..60726 FIN_WAIT_2 > > > > > > > > Proto Recv-Q Send-Q Local Address Foreign Address > > State > > > > tcp 1 36 server:60726 client:10514 > CLOSE_WAIT > > > > > > > > Have I misunderstood something? I've read Rainer's blog and issues > with TCP > > "unreliability" / case of a power failure but don't think this > related > > I think it's the same problem, one side thinks the connection is > closed, > the other doesn't and so the one that doesn't happily sends the data > out. > > try reconfiguring to use relp and things should work as expected. > > > now, I don't know why the close isn't getting through from the one > system > to the other rapidly. is there any sort of firewall in between them? The close may get through. It is a (kind of) race condition, inside the tcp stack. Assume the following happens (System S being the server, system C being the client): S : closes connection, sends close info C1: rsyslog writes data (not yet sent, but acknowledged by tcp stack) C2: receives close request C3: discards data The race is in C1/C2. In this order, data is lost. If C2 happens before C1, no data is lost. No chance to solve that with plain tcp without app-level acks. Rainer From david at lang.hm Thu Oct 8 09:48:08 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 8 Oct 2009 00:48:08 -0700 (PDT) Subject: [rsyslog] Queue System In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710319B@GRFEXC.intern.adiscon.com> References: <007901ca47b6$34ac0870$9e041950$@com> <9B6E2A8877C38245BFB15CC491A11DA710319B@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 8 Oct 2009, Rainer Gerhards wrote: >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >> On Wed, 7 Oct 2009, Jonathan Bond-Caron wrote: >> >>> The client 'sends the message over tcp' and actually empties the >> queue! >>> >>> Is this expected behavior? rsyslogd was shutdown 'cleanly' on the >> server. >>> Shouldn't it force the client to close the TCP connection? >>> >>> >>> >>> Using netstat, the two servers are : >>> >>> >>> >>> Proto Recv-Q Send-Q Local Address Foreign Address >> (state) >>> >>> tcp4 0 0 client.10514 server..60726 FIN_WAIT_2 >>> >>> >>> >>> Proto Recv-Q Send-Q Local Address Foreign Address >>> State >>> >>> tcp 1 36 server:60726 client:10514 >> CLOSE_WAIT >>> >>> >>> >>> Have I misunderstood something? I've read Rainer's blog and issues >> with TCP >>> "unreliability" / case of a power failure but don't think this >> related >> >> I think it's the same problem, one side thinks the connection is >> closed, >> the other doesn't and so the one that doesn't happily sends the data >> out. >> >> try reconfiguring to use relp and things should work as expected. >> >> >> now, I don't know why the close isn't getting through from the one >> system >> to the other rapidly. is there any sort of firewall in between them? > > The close may get through. It is a (kind of) race condition, inside the tcp > stack. Assume the following happens (System S being the server, system C > being the client): > > S : closes connection, sends close info > C1: rsyslog writes data (not yet sent, but acknowledged by tcp stack) > C2: receives close request > C3: discards data > > The race is in C1/C2. In this order, data is lost. If C2 happens before C1, > no data is lost. No chance to solve that with plain tcp without app-level > acks. but that window should be very short, for it to last long enough to show up in the netstat commands sounds odd to me. David Lang From rgerhards at hq.adiscon.com Thu Oct 8 09:51:39 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 09:51:39 +0200 Subject: [rsyslog] Queue System References: <007901ca47b6$34ac0870$9e041950$@com><9B6E2A8877C38245BFB15CC491A11DA710319B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710319D@GRFEXC.intern.adiscon.com> > -----Original Message----- > > The close may get through. It is a (kind of) race condition, inside > the tcp > > stack. Assume the following happens (System S being the server, > system C > > being the client): > > > > S : closes connection, sends close info > > C1: rsyslog writes data (not yet sent, but acknowledged by tcp stack) > > C2: receives close request > > C3: discards data > > > > The race is in C1/C2. In this order, data is lost. If C2 happens > before C1, > > no data is lost. No chance to solve that with plain tcp without app- > level > > acks. > > but that window should be very short, for it to last long enough to > show > up in the netstat commands sounds odd to me. Ah, I didn't pay attention to the netstat. You are right, in *this* case the close looks like it does not go through. HOWEVER, for the argument I have given, this race exists in general. The window is extremely short (at least on a local LAN), but I have learned if there is a potential for a race, it will happen sooner or later. Chances go up very soon as quickly as you have millions and millions of cases... Just yesterday I was able to find a race with a much lower probability in v5-beta during shutdown... And it really happened (thankfully only in my lab, where I set parameters to make such races more probable). Rainer From martinmie at PartyGaming.com Thu Oct 8 10:50:13 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 8 Oct 2009 10:50:13 +0200 Subject: [rsyslog] URGENT - rsyslog stops sshd Message-ID: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Hi all, odd behavior here... Today one of our core systems stopped accepting ssh connections. -- $ telnet sshserver 22 Trying 1.2.3.4... Connected to sshserver. Escape character is '^]'. Connection closed. -- Using a pre-established session rsyslog was shutdown: -- # /etc/init.d/rsyslog stop Shutting down system logger: [ OK ][/quote] -- ..and the system started accepting connections again: -- $ ssh sshserver Last login: Thu Oct 8 01:12:59 2009 from foo.bar.tld -- We are just one step before dropping rsyslog and use syslog-ng again (which was dismissed some time ago to favor rsyslog) but I would like to know what's the reason for this and, most important, what's the solution :-) Some additional info: -- # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 32767 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 32767 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited -- Thanks! Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From david at lang.hm Thu Oct 8 11:16:44 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 8 Oct 2009 02:16:44 -0700 (PDT) Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Message-ID: On Thu, 8 Oct 2009, Martin Mielke wrote: > Hi all, > > odd behavior here... > > Today one of our core systems stopped accepting ssh connections. > -- > $ telnet sshserver 22 > Trying 1.2.3.4... > Connected to sshserver. > Escape character is '^]'. > Connection closed. > -- > > Using a pre-established session rsyslog was shutdown: > -- > # /etc/init.d/rsyslog stop > Shutting down system logger: [ OK > ][/quote] > -- > > ..and the system started accepting connections again: > -- > $ ssh sshserver > Last login: Thu Oct 8 01:12:59 2009 from foo.bar.tld > -- > > We are just one step before dropping rsyslog and use syslog-ng again > (which was dismissed some time ago to favor rsyslog) but I would like to > know what's the reason for this and, most important, what's the solution > :-) ssh logs connections, if the syslog process cannot process the message, ssh is designed to stop and wait until it does (it deems the log so important that it refuses to do anything until the log is written) conole logins do the same thing. so something causes rsyslog to stop accepting messages. the same thing can happen to syslog-ng or to plain sysklog, just under different conditions. so the question is why rsyslog stopped. what is rsyslog configured to do with log messages? is there any chance that it was unable to do something with a message and so would have had to keep it in it's queue until the queue filled up? (logging via TCP to a remote server that stops responding will do this, so will writing to a full filesystem) rsyslog can be configured to accept and discard log entries when the queue is full, doing this can avoid this sort of situation. David Lang > Some additional info: > -- > # ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 32767 > max locked memory (kbytes, -l) 32 > max memory size (kbytes, -m) unlimited > open files (-n) 1024 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 10240 > cpu time (seconds, -t) unlimited > max user processes (-u) 32767 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > -- > > > Thanks! > > Martin > > > This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. > > Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iamsayan at gmail.com Thu Oct 8 11:16:14 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Thu, 8 Oct 2009 05:16:14 -0400 Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540910080216y1caef6begd013ff6b722e76ef@mail.gmail.com> I agree ... As long as rsyslog has a mechanism to do the truncation if somebody wants on the egress(which it has), it is fine. I don't think the default template should change. BTW, there are implementations out there which behaves badly when they receive large messages/control characters etc. I want to be defensive about what send out and hence my requirement. On Thu, Oct 8, 2009 at 3:41 AM, wrote: > On Thu, 8 Oct 2009, Rainer Gerhards wrote: > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > >> > >> Yes, I saw that too ... I made the template based out of Rsyslog > >> traditional > >> forward format and then used the property replacer as you mentioned. > >> It works like a charm. Thanks a bunch! > > > > I am glad it works, but I also tend to think that the default template is > > probably not correct. As it claims to be "traditional" format, I think it > > should really limit itself to 1K message size, so that another template > must > > be picked if messages of "non-traditional large size" are to be > transmitted. > > > > However, changing that default would potentially break a number of > existing > > deployments. > > > > Does anybody else have an opinion on that? > > existing syslog implementations already need to be able to deal with > oversized messages (almost nothing checks the size before it gets sent), > so I don't see a big benifit in changing rsyslog to limit what it outputs. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu Oct 8 11:26:15 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 8 Oct 2009 02:26:15 -0700 (PDT) Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <87a1ae540910080216y1caef6begd013ff6b722e76ef@mail.gmail.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> <87a1ae540910080216y1caef6begd013ff6b722e76ef@mail.gmail.com> Message-ID: On Thu, 8 Oct 2009, Sayan Chowdhury wrote: > I agree ... As long as rsyslog has a mechanism to do the truncation if > somebody wants on the egress(which it has), it is fine. I don't think the > default template should change. > BTW, there are implementations out there which behaves badly when they > receive large messages/control characters etc. I want to be defensive about > what send out and hence my requirement. which implementation caused you grief? (so I know what to avoid) David Lang > > > > On Thu, Oct 8, 2009 at 3:41 AM, wrote: > >> On Thu, 8 Oct 2009, Rainer Gerhards wrote: >> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury >>>> >>>> Yes, I saw that too ... I made the template based out of Rsyslog >>>> traditional >>>> forward format and then used the property replacer as you mentioned. >>>> It works like a charm. Thanks a bunch! >>> >>> I am glad it works, but I also tend to think that the default template is >>> probably not correct. As it claims to be "traditional" format, I think it >>> should really limit itself to 1K message size, so that another template >> must >>> be picked if messages of "non-traditional large size" are to be >> transmitted. >>> >>> However, changing that default would potentially break a number of >> existing >>> deployments. >>> >>> Does anybody else have an opinion on that? >> >> existing syslog implementations already need to be able to deal with >> oversized messages (almost nothing checks the size before it gets sent), >> so I don't see a big benifit in changing rsyslog to limit what it outputs. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Oct 8 11:28:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 11:28:56 +0200 Subject: [rsyslog] limiting message size while forwarding messages References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com><87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com><87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com><87a1ae540910080216y1caef6begd013ff6b722e76ef@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031A2@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, October 08, 2009 11:26 AM > To: rsyslog-users > Subject: Re: [rsyslog] limiting message size while forwarding messages > > On Thu, 8 Oct 2009, Sayan Chowdhury wrote: > > > I agree ... As long as rsyslog has a mechanism to do the truncation > if > > somebody wants on the egress(which it has), it is fine. I don't think > the > > default template should change. > > BTW, there are implementations out there which behaves badly when > they > > receive large messages/control characters etc. I want to be defensive > about > > what send out and hence my requirement. > > which implementation caused you grief? (so I know what to avoid) > The one I know about is darn old Solaris syslogd, which reliably died with messages of 1025 bytes and above. However, I think this is fixed for 10 years or so now ;) I've seen no other problems with implementations, but I definitely have not seen everything ;) Rainer > David Lang > > > > > > > > > On Thu, Oct 8, 2009 at 3:41 AM, wrote: > > > >> On Thu, 8 Oct 2009, Rainer Gerhards wrote: > >> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > >>>> > >>>> Yes, I saw that too ... I made the template based out of Rsyslog > >>>> traditional > >>>> forward format and then used the property replacer as you > mentioned. > >>>> It works like a charm. Thanks a bunch! > >>> > >>> I am glad it works, but I also tend to think that the default > template is > >>> probably not correct. As it claims to be "traditional" format, I > think it > >>> should really limit itself to 1K message size, so that another > template > >> must > >>> be picked if messages of "non-traditional large size" are to be > >> transmitted. > >>> > >>> However, changing that default would potentially break a number of > >> existing > >>> deployments. > >>> > >>> Does anybody else have an opinion on that? > >> > >> existing syslog implementations already need to be able to deal with > >> oversized messages (almost nothing checks the size before it gets > sent), > >> so I don't see a big benifit in changing rsyslog to limit what it > outputs. > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From dag at wieers.com Thu Oct 8 12:36:38 2009 From: dag at wieers.com (Dag Wieers) Date: Thu, 8 Oct 2009 12:36:38 +0200 (CEST) Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Message-ID: On Thu, 8 Oct 2009, david at lang.hm wrote: > rsyslog can be configured to accept and discard log entries when the queue > is full, doing this can avoid this sort of situation. Hi Martin, We were in a similar situation recently (because of a limit of open file descriptors) and I am very interested to learn how you can do the above. Thanks in advance, -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] From martinmie at PartyGaming.com Thu Oct 8 13:28:36 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 8 Oct 2009 13:28:36 +0200 Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Message-ID: <0B1B9163138571439EAF804646F3F6AA08771181@GIBSVWIN004X.partygaming.local> Hi Dag, The number of open file descriptors wasn't the issue here... and I'm also interested in how to configure rsyslog as described by David ;-) Cheers, Martin > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dag Wieers > Sent: 08 October 2009 12:37 > To: rsyslog-users > Subject: Re: [rsyslog] URGENT - rsyslog stops sshd > > On Thu, 8 Oct 2009, david at lang.hm wrote: > > > rsyslog can be configured to accept and discard log entries when the > queue > > is full, doing this can avoid this sort of situation. > > Hi Martin, > > We were in a similar situation recently (because of a limit of open file > descriptors) and I am very interested to learn how you can do the above. > > Thanks in advance, > -- > -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- > [Any errors in spelling, tact or fact are transmission errors] > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > nfo/rsyslog > http://www.rsyslog.com This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From martinmie at PartyGaming.com Thu Oct 8 13:32:23 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 8 Oct 2009 13:32:23 +0200 Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Message-ID: <0B1B9163138571439EAF804646F3F6AA08771183@GIBSVWIN004X.partygaming.local> Hi, > > ssh logs connections, if the syslog process cannot process the message, > ssh is designed to stop and wait until it does (it deems the log so > important that it refuses to do anything until the log is written) > > conole logins do the same thing. Is there any way I can test this? Strace? Ltrace? Sshd configuration directive?... > > so something causes rsyslog to stop accepting messages. the same thing can > happen to syslog-ng or to plain sysklog, just under different conditions. > > so the question is why rsyslog stopped. > > what is rsyslog configured to do with log messages? > > is there any chance that it was unable to do something with a message and > so would have had to keep it in it's queue until the queue filled up? > (logging via TCP to a remote server that stops responding will do this, so > will writing to a full filesystem) The system sends its logs over TCP to a remote logserver. Initial analysis yield a routing problem which should have been fixed so I hope not to see such a problem due to this. But, my question now is: shouldn't logfiles be spooled under $WorkDirectory /var/spool/rsyslog and sent over to the logserver once it becomes available again? There's plenty of space on /var ... > > rsyslog can be configured to accept and discard log entries when the queue > is full, doing this can avoid this sort of situation. Could you please tell us how? :-) Regards, Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From rgerhards at hq.adiscon.com Thu Oct 8 14:11:45 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 14:11:45 +0200 Subject: [rsyslog] URGENT - rsyslog stops sshd References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031A5@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dag Wieers > Sent: Thursday, October 08, 2009 12:37 PM > To: rsyslog-users > Subject: Re: [rsyslog] URGENT - rsyslog stops sshd > > On Thu, 8 Oct 2009, david at lang.hm wrote: > > > rsyslog can be configured to accept and discard log entries when the > queue > > is full, doing this can avoid this sort of situation. > > Hi Martin, > > We were in a similar situation recently (because of a limit of open > file > descriptors) and I am very interested to learn how you can do the > above. Well, it depends on your configuration. If you use a plain default config, this situation is expected to never happen, because we assume that log files can be written and discard messages when this is not the case. However, if you begin to configure the system to be reliable, you need to think about the implications. In almost all cases where I have seen such a problem (assuming it was not a real bug, of course), the configuration demanded reliable delivery, but it was not thought about the fact that if the message could not be delivered, the system would stall. Some organizations actually prefer this mode, so it may be useful. If you post your config, I could check if there is something in it that demands such reliability and prevents the engine from discarding messages when needed. Rainer From martinmie at PartyGaming.com Thu Oct 8 14:24:06 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 8 Oct 2009 14:24:06 +0200 Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031A5@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> <9B6E2A8877C38245BFB15CC491A11DA71031A5@GRFEXC.intern.adiscon.com> Message-ID: <0B1B9163138571439EAF804646F3F6AA087711B7@GIBSVWIN004X.partygaming.local> Hi Rainer, This is the rsyslog.conf used - quite simple as you can see: -- $ cat /etc/rsyslog.conf # rsyslog v3: load input modules # If you do not load inputs, nothing happens! # You may need to set the module load path if modules are not found. $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # provides kernel logging support (previously done by rklogd) # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none -/var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* -/var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit -/var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. $WorkDirectory /var/spool/rsyslog # where to place spool files $ActionQueueFileName uniqName # unique name prefix for spool files $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) $ActionQueueSaveOnShutdown on # save messages to disk on shutdown $ActionQueueType LinkedList # run asynchronously $ActionResumeRetryCount 5 # five retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional *.* @@logserver -- Do you see anything suspicious? Thanks, Martin > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: 08 October 2009 14:12 > To: rsyslog-users > Subject: Re: [rsyslog] URGENT - rsyslog stops sshd > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Dag Wieers > > Sent: Thursday, October 08, 2009 12:37 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] URGENT - rsyslog stops sshd > > > > On Thu, 8 Oct 2009, david at lang.hm wrote: > > > > > rsyslog can be configured to accept and discard log entries when the > > queue > > > is full, doing this can avoid this sort of situation. > > > > Hi Martin, > > > > We were in a similar situation recently (because of a limit of open > > file > > descriptors) and I am very interested to learn how you can do the > > above. > > Well, it depends on your configuration. If you use a plain default config, > this situation is expected to never happen, because we assume that log > files > can be written and discard messages when this is not the case. However, if > you begin to configure the system to be reliable, you need to think about > the > implications. In almost all cases where I have seen such a problem > (assuming > it was not a real bug, of course), the configuration demanded reliable > delivery, but it was not thought about the fact that if the message could > not > be delivered, the system would stall. Some organizations actually prefer > this > mode, so it may be useful. > > If you post your config, I could check if there is something in it that > demands such reliability and prevents the engine from discarding messages > when needed. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > /rsyslog > http://www.rsyslog.com This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From jbondc at openmv.com Thu Oct 8 15:17:44 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Thu, 8 Oct 2009 09:17:44 -0400 Subject: [rsyslog] Queue System In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710319D@GRFEXC.intern.adiscon.com> References: <007901ca47b6$34ac0870$9e041950$@com><9B6E2A8877C38245BFB15CC491A11DA710319B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710319D@GRFEXC.intern.adiscon.com> Message-ID: <000e01ca4819$b942b760$2bc82620$@com> On Thu Oct 8 03:51 AM, Rainer Gerhards wrote: > > -----Original Message----- > > > The close may get through. It is a (kind of) race condition, > > > inside > > the tcp > > > stack. Assume the following happens (System S being the server, > > system C > > > being the client): > > Ah, I didn't pay attention to the netstat. You are right, in *this* > case the close looks like it does not go through. HOWEVER, for the > argument I have given, this race exists in general. The window is > extremely short (at least on a local LAN), but I have learned if there > is a potential for a race, it will happen sooner or later. Chances go > up very soon as quickly as you have millions and millions of cases... > Just yesterday I was able to find a race with a much lower probability > in v5-beta during shutdown... And it really happened (thankfully only > in my lab, where I set parameters to make such races more probable). > I decided to look at the code and I must say, wow documented open source code! I haven't written c in a while but easily found my way, so big congrats on keeping the code so clean. After some googling, I found that whenever TCP is in status CLOSE_WAIT, the host is expected to close the socket sometime soon. It might not want to close it right away for example to keep-alive the connection... So I searched for all close() calls on the socket. I found sockClose() then static void CheckConnection(nsd_t *pNsd) -- runtime/nsd_ptcp.c http://blog.gerhards.net/2008/06/getting-bit-more-reliability-from-plain.htm l With debug mode I found: 5543.432507331:action 10 queue:Reg/w0: server:10514/tcp 5543.432507331:action 10 queue:Reg/w0: TCP sent 36 bytes, requested 36 It turns out I was pointing to the SSL config which had: $DefaultNetstreamDriver gtls For this, I found: void CheckConnection(nsd_t *pNsd) -- runtime/nsd_gtls.c /* This function checks if the connection is still alive - well, kind of... * This is a dummy here. For details, check function common in ptcp driver. * rgerhards, 2008-06-09 */ static void CheckConnection(nsd_t *pNsd) { dbgprintf("CheckConnection SSL - do something\n"); nsd_gtls_t *pThis = (nsd_gtls_t*) pNsd; ISOBJ_TYPE_assert(pThis, nsd_gtls); nsd_ptcp.CheckConnection(pThis->pTcp); } 5649.998580185:action 10 queue:Reg/w0: server:10514/tcp 5649.998580185:action 10 queue:Reg/w0: CheckConnection SSL - do something 5649.998580185:action 10 queue:Reg/w0: CheckConnection detected broken connection - closing it Wonderful! It works as I would expect. But, is there any reason why the socket check wasn't added for SSL? I'm currently testing this 'patch' with a couple of 'live' servers and randomly stopping the central logging server. From dirk.schulz at kinzesberg.de Thu Oct 8 17:58:51 2009 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Thu, 08 Oct 2009 17:58:51 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> References: <4ACCA646.8050209@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> Message-ID: <4ACE0C3B.7090207@kinzesberg.de> Rainer Gerhards schrieb: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Dirk H. Schulz >> Sent: Wednesday, October 07, 2009 4:32 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] Getting Facility from Synlog-NG >> >> Hi folks, >> >> I am not sure if this is the right list to ask, but I hope that there >> is >> more of the related knowledge here. >> >> I have several Syslog-NG servers reporting to Rsyslog central servers >> (just because nobody could tell me how to run Rsyslog on NetBSD ...). >> Now I fould that the facility is not transferred to/evaluated on the >> Rsyslog servers; e.g. mail messages are written to syslog instead of >> maillog on the Rsyslog machines. >> > > This is strange. Facility is a fairly standard field. Could you tell me the > content of %rawmwg%. This is done along these lines (if it does not work, > check doc for exact syntax): > > $template raw,"%rawmsg%\n" > *.* /path/to/file;raw > I did not find these terms, neither in rsyslog.conf on the central logserver servers nor in syslog-ng.conf on the log client servers. Rsyslog is 3.18.6-4 on Debian 5. Dirk > > Rainer > >> Is there something special I have to configure on the Rsyslog servers >> or >> the Syslog-NG servers to fix that? >> >> Any hint or help is appreciated. >> >> Dirk >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Oct 8 18:03:54 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 8 Oct 2009 18:03:54 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG References: <4ACCA646.8050209@kinzesberg.de><9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> <4ACE0C3B.7090207@kinzesberg.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031AC@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dirk H. Schulz > Sent: Thursday, October 08, 2009 5:59 PM > To: rsyslog-users > Subject: Re: [rsyslog] Getting Facility from Synlog-NG > > Rainer Gerhards schrieb: > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Dirk H. Schulz > >> Sent: Wednesday, October 07, 2009 4:32 PM > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] Getting Facility from Synlog-NG > >> > >> Hi folks, > >> > >> I am not sure if this is the right list to ask, but I hope that > there > >> is > >> more of the related knowledge here. > >> > >> I have several Syslog-NG servers reporting to Rsyslog central > servers > >> (just because nobody could tell me how to run Rsyslog on NetBSD > ...). > >> Now I fould that the facility is not transferred to/evaluated on the > >> Rsyslog servers; e.g. mail messages are written to syslog instead of > >> maillog on the Rsyslog machines. > >> > > > > This is strange. Facility is a fairly standard field. Could you tell > me the > > content of %rawmwg%. This is done along these lines (if it does not > work, > > check doc for exact syntax): > > > > $template raw,"%rawmsg%\n" > > *.* /path/to/file;raw > > > > I did not find these terms, neither in rsyslog.conf on the central > logserver servers nor in syslog-ng.conf on the log client servers. > Rsyslog is 3.18.6-4 on Debian 5. Well, these are the config statements for the central rsyslog instance... Rainer From dirk.schulz at kinzesberg.de Thu Oct 8 18:22:49 2009 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Thu, 08 Oct 2009 18:22:49 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031AC@GRFEXC.intern.adiscon.com> References: <4ACCA646.8050209@kinzesberg.de><9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> <4ACE0C3B.7090207@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA71031AC@GRFEXC.intern.adiscon.com> Message-ID: <4ACE11D9.1020806@kinzesberg.de> Rainer Gerhards schrieb: >>> check doc for exact syntax): >>> >>> $template raw,"%rawmsg%\n" >>> *.* /path/to/file;raw >>> >>> >> I did not find these terms, neither in rsyslog.conf on the central >> logserver servers nor in syslog-ng.conf on the log client servers. >> Rsyslog is 3.18.6-4 on Debian 5. >> > > Well, these are the config statements for the central rsyslog instance... > Does that mean that I am lacking a template statement in the central rsyslog config? Dirk > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iamsayan at gmail.com Thu Oct 8 18:48:33 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Thu, 8 Oct 2009 22:18:33 +0530 Subject: [rsyslog] limiting message size while forwarding messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031A2@GRFEXC.intern.adiscon.com> References: <87a1ae540910070735u6a3e0eadk868cd28ed102798a@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103194@GRFEXC.intern.adiscon.com> <87a1ae540910070753v5ca6f5a5s67257590f7a42c37@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103197@GRFEXC.intern.adiscon.com> <87a1ae540910071111u75bdb66dob0ee65e158054b62@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103199@GRFEXC.intern.adiscon.com> <87a1ae540910080216y1caef6begd013ff6b722e76ef@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71031A2@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540910080948t1859f0dav90e3cd15c1f951b7@mail.gmail.com> it was an in house properietary "data mining" implementation. I don't know why people try to reinvent the wheel :) On Thu, Oct 8, 2009 at 2:58 PM, Rainer Gerhards wrote: > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Thursday, October 08, 2009 11:26 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] limiting message size while forwarding messages > > > > On Thu, 8 Oct 2009, Sayan Chowdhury wrote: > > > > > I agree ... As long as rsyslog has a mechanism to do the truncation > > if > > > somebody wants on the egress(which it has), it is fine. I don't think > > the > > > default template should change. > > > BTW, there are implementations out there which behaves badly when > > they > > > receive large messages/control characters etc. I want to be defensive > > about > > > what send out and hence my requirement. > > > > which implementation caused you grief? (so I know what to avoid) > > > > The one I know about is darn old Solaris syslogd, which reliably died with > messages of 1025 bytes and above. However, I think this is fixed for 10 > years > or so now ;) I've seen no other problems with implementations, but I > definitely have not seen everything ;) > > Rainer > > > David Lang > > > > > > > > > > > > > > On Thu, Oct 8, 2009 at 3:41 AM, wrote: > > > > > >> On Thu, 8 Oct 2009, Rainer Gerhards wrote: > > >> > > >>>> -----Original Message----- > > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >>>> bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > >>>> > > >>>> Yes, I saw that too ... I made the template based out of Rsyslog > > >>>> traditional > > >>>> forward format and then used the property replacer as you > > mentioned. > > >>>> It works like a charm. Thanks a bunch! > > >>> > > >>> I am glad it works, but I also tend to think that the default > > template is > > >>> probably not correct. As it claims to be "traditional" format, I > > think it > > >>> should really limit itself to 1K message size, so that another > > template > > >> must > > >>> be picked if messages of "non-traditional large size" are to be > > >> transmitted. > > >>> > > >>> However, changing that default would potentially break a number of > > >> existing > > >>> deployments. > > >>> > > >>> Does anybody else have an opinion on that? > > >> > > >> existing syslog implementations already need to be able to deal with > > >> oversized messages (almost nothing checks the size before it gets > > sent), > > >> so I don't see a big benifit in changing rsyslog to limit what it > > outputs. > > >> > > >> David Lang > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From iamsayan at gmail.com Thu Oct 8 18:51:45 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Thu, 8 Oct 2009 22:21:45 +0530 Subject: [rsyslog] Getting Facility from Synlog-NG In-Reply-To: <4ACE11D9.1020806@kinzesberg.de> References: <4ACCA646.8050209@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> <4ACE0C3B.7090207@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA71031AC@GRFEXC.intern.adiscon.com> <4ACE11D9.1020806@kinzesberg.de> Message-ID: <87a1ae540910080951g6c7106fby53979808642cb537@mail.gmail.com> Hello Dirk, I think Rainer wants you to add this to your rsyslog.conf file, and send him the file that gets saved because of adding this statement. So as to debug the issue. You will need to restart rsyslog to pick up the config change Hope this helps. Sayan On Thu, Oct 8, 2009 at 9:52 PM, Dirk H. Schulz wrote: > Rainer Gerhards schrieb: > >>> check doc for exact syntax): > >>> > >>> $template raw,"%rawmsg%\n" > >>> *.* /path/to/file;raw > >>> > >>> > >> I did not find these terms, neither in rsyslog.conf on the central > >> logserver servers nor in syslog-ng.conf on the log client servers. > >> Rsyslog is 3.18.6-4 on Debian 5. > >> > > > > Well, these are the config statements for the central rsyslog instance... > > > Does that mean that I am lacking a template statement in the central > rsyslog config? > > Dirk > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu Oct 8 20:31:09 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 8 Oct 2009 11:31:09 -0700 (PDT) Subject: [rsyslog] URGENT - rsyslog stops sshd In-Reply-To: <0B1B9163138571439EAF804646F3F6AA087711B7@GIBSVWIN004X.partygaming.local> References: <0B1B9163138571439EAF804646F3F6AA087710E5@GIBSVWIN004X.partygaming.local> <9B6E2A8877C38245BFB15CC491A11DA71031A5@GRFEXC.intern.adiscon.com> <0B1B9163138571439EAF804646F3F6AA087711B7@GIBSVWIN004X.partygaming.local> Message-ID: On Thu, 8 Oct 2009, Martin Mielke wrote: > Hi Rainer, > > This is the rsyslog.conf used - quite simple as you can see: > -- > $ cat /etc/rsyslog.conf > # rsyslog v3: load input modules > # If you do not load inputs, nothing happens! > # You may need to set the module load path if modules are not found. > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # provides kernel logging support (previously done by > rklogd) > > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none > -/var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* > -/var/log/maillog > > > # Log cron stuff > cron.* -/var/log/cron > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit > -/var/log/spooler > > # Save boot messages also to boot.log > local7.* > /var/log/boot.log > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/spool/rsyslog # where to place spool files > $ActionQueueFileName uniqName # unique name prefix for spool files > $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > possible) > $ActionQueueSaveOnShutdown on # save messages to disk on shutdown > $ActionQueueType LinkedList # run asynchronously > $ActionResumeRetryCount 5 # five retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > *.* @@logserver > -- > > Do you see anything suspicious? this TCP logging would cause it. you have it limited to 1g of disk space, how large did it get? also, disk queues are pretty slow, it may be that you had messages arriving faster than they could be put into the queue also, you still have it using the default main queue size (something tiny like 1000 messages IIRC) I would bump the main queue size up significantly (assuming you can afford the ram) also, look at the high and low watermark config variables, they (along with one other variable I don't remember offhand) tell the system what to do in crisis mode and how to define that mode. unfortunantly I don't have time until late today to go through the config page and dig up the exact details. David Lang From dag at wieers.com Thu Oct 8 21:10:22 2009 From: dag at wieers.com (Dag Wieers) Date: Thu, 8 Oct 2009 21:10:22 +0200 (CEST) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole Message-ID: Hi, We have been looking at why we don't see messages from netconsole ending up in our logfiles. netconsole is a kernel module that sends information of kernel panics to a syslog server. The documentation indicates it should work with syslog as an endpoint but that is not true with rsyslog, apparently :) https://bugzilla.redhat.com/show_bug.cgi?id=432160 Also, I have noticed that our central rsyslog-server (more than 400 systems log to it) have directories with names like: Detected/, exiting/, ext3_abort/, EXT3-fs/, journal/, last/, martian/, program/, Remounting/, Restarting/, ... So it is obvious that something is not working as expected coming from the kernel. >From information I found online, a message formatted like: <4>Testing through netcat should work for testing the syslog connectivity, but it is clear that rsyslog only accepts: <4>Oct 7 12:34:56 hostname Testing but of course I cannot influence our production kernels to do the right thing. What can I do to have rsyslog accept the "wrong" thing ? :) Advice is appreciated :) -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] From aoz.syn at gmail.com Thu Oct 8 21:28:38 2009 From: aoz.syn at gmail.com (RB) Date: Thu, 8 Oct 2009 13:28:38 -0600 Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: References: Message-ID: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> On Thu, Oct 8, 2009 at 13:10, Dag Wieers wrote: > We have been looking at why we don't see messages from netconsole ending > up in our logfiles. netconsole is a kernel module that sends information > of kernel panics to a syslog server. The documentation indicates it should > work with syslog as an endpoint but that is not true with rsyslog, > apparently :) Mostly because said syslog daemons are not only very loose in their interpretation of a valid syslog entry, but in their default configurations they also tend to silently interpolate data sloppy senders leave out. > Also, I have noticed that our central rsyslog-server (more than 400 > systems log to it) have directories with names like: Detected/, exiting/, > ext3_abort/, EXT3-fs/, journal/, last/, martian/, program/, Remounting/, > Restarting/, ... So it is obvious that something is not working as > expected coming from the kernel. More than likely because you're trying to make directories based on the %hostname% property, which rsyslog assumes is a specific field. With the sloppier daemons (FreeBSD in particular), I've had far more luck using the %fromhost-ip% property (as well as the $ system properties for timestamps). Of course, that breaks down if you're doing relaying, but relying on values the other end sends you to create filesystem artifacts is dangerous at best anyway > but of course I cannot influence our production kernels to do the right > thing. What can I do to have rsyslog accept the "wrong" thing ? :) Use %fromhost% or %fromhost-ip% to make the directory structures/filenames, and make a custom format if you need to handle the remaning lack of data (again, timestamp & host). I'm sure there are many other ways to approach it, but that's the way I've solved it. From mrdemeanour at jackpot.uk.net Fri Oct 9 13:59:52 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 09 Oct 2009 12:59:52 +0100 Subject: [rsyslog] Deviant PRI values causing dropped messages Message-ID: <4ACF25B8.3040002@jackpot.uk.net> Hi, I have a Speedtouch router that produces firewall events whose PRI value is <84> (auth.notice, according to RFC3164). I have a script that retrieves events from the router via Telnet, and passes them on to rsyslog via UDP. The firewall events with PRI=<84> are dropped. If I modify the script so that they are submitted to rsyslog with a PRI of <36> (facility changed from 10 to 4), they are logged. They are also logged if the facility is 13 ("audit"?). According to RFC3614, facilities numbers 4 and 10 are both "auth". I haven't inspected the source, but I suspect that rsyslog doesn't like facility number 10, and is ignoring it. Is this a correct interpretation? Why should it object to this facility in particular? Regards, -- Jack. From mrdemeanour at jackpot.uk.net Fri Oct 9 14:08:38 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 09 Oct 2009 13:08:38 +0100 Subject: [rsyslog] Deviant PRI values causing dropped messages In-Reply-To: <4ACF25B8.3040002@jackpot.uk.net> References: <4ACF25B8.3040002@jackpot.uk.net> Message-ID: <4ACF27C6.5080705@jackpot.uk.net> Mr. Demeanour wrote: > Hi, > > I have a Speedtouch router that produces firewall events whose PRI value > is <84> (auth.notice, according to RFC3164). Sorry; that should be "auth.warn". -- Jack. From rgerhards at hq.adiscon.com Fri Oct 9 14:10:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 9 Oct 2009 14:10:27 +0200 Subject: [rsyslog] Deviant PRI values causing dropped messages References: <4ACF25B8.3040002@jackpot.uk.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031BB@GRFEXC.intern.adiscon.com> Hi Jack, rsyslog does not do any processing per se on the message. It only does what you instruct it to do. So there is probably something in your rsyslog.conf that does either not process messages with this facility or drops them. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mr. Demeanour > Sent: Friday, October 09, 2009 2:00 PM > To: rsyslog-users > Subject: [rsyslog] Deviant PRI values causing dropped messages > > Hi, > > I have a Speedtouch router that produces firewall events whose PRI > value > is <84> (auth.notice, according to RFC3164). I have a script that > retrieves events from the router via Telnet, and passes them on to > rsyslog via UDP. > > The firewall events with PRI=<84> are dropped. If I modify the script > so > that they are submitted to rsyslog with a PRI of <36> (facility changed > from 10 to 4), they are logged. They are also logged if the facility is > 13 ("audit"?). > > According to RFC3614, facilities numbers 4 and 10 are both "auth". I > haven't inspected the source, but I suspect that rsyslog doesn't like > facility number 10, and is ignoring it. Is this a correct > interpretation? Why should it object to this facility in particular? From mrdemeanour at jackpot.uk.net Fri Oct 9 14:15:21 2009 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Fri, 09 Oct 2009 13:15:21 +0100 Subject: [rsyslog] Deviant PRI values causing dropped messages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031BB@GRFEXC.intern.adiscon.com> References: <4ACF25B8.3040002@jackpot.uk.net> <9B6E2A8877C38245BFB15CC491A11DA71031BB@GRFEXC.intern.adiscon.com> Message-ID: <4ACF2959.2020707@jackpot.uk.net> Rainer Gerhards wrote: > Hi Jack, > > rsyslog does not do any processing per se on the message. It only > does what you instruct it to do. So there is probably something in > your rsyslog.conf that does either not process messages with this > facility or drops them. Oh, thanks, Rainer. I had this entry in the config: authpriv.info ~ Commenting it out fixed the problem. -- Jack. From tbergfeld at hq.adiscon.com Fri Oct 9 14:34:49 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 9 Oct 2009 14:34:49 +0200 Subject: [rsyslog] rsyslog 4.4.2 (v4-stable) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031BE@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.4.2, a member of the v4-stable branch. This is a bug-fixing release containing some important fixes. It is a recommended update for all users of the v4-stabled. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article409.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-179.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From dag at wieers.com Fri Oct 9 14:38:01 2009 From: dag at wieers.com (Dag Wieers) Date: Fri, 9 Oct 2009 14:38:01 +0200 (CEST) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> Message-ID: On Thu, 8 Oct 2009, RB wrote: > On Thu, Oct 8, 2009 at 13:10, Dag Wieers wrote: > >> Also, I have noticed that our central rsyslog-server (more than 400 >> systems log to it) have directories with names like: Detected/, exiting/, >> ext3_abort/, EXT3-fs/, journal/, last/, martian/, program/, Remounting/, >> Restarting/, ... So it is obvious that something is not working as >> expected coming from the kernel. > > More than likely because you're trying to make directories based on > the %hostname% property, which rsyslog assumes is a specific field. > With the sloppier daemons (FreeBSD in particular), I've had far more > luck using the %fromhost-ip% property (as well as the $ system > properties for timestamps). Of course, that breaks down if you're > doing relaying, but relying on values the other end sends you to > create filesystem artifacts is dangerous at best anyway You are correct, that is exactly what we do. However with rsyslog v2.0.6 it seems there is no %FROMHOST-IP% and the %FROMHOST% property only contains the IP address. Maybe there is something else I need to do to get the short hostname from DNS, rather than an IP on rsyslog v2 ? >> but of course I cannot influence our production kernels to do the right >> thing. What can I do to have rsyslog accept the "wrong" thing ? :) > > Use %fromhost% or %fromhost-ip% to make the directory > structures/filenames, and make a custom format if you need to handle > the remaning lack of data (again, timestamp & host). I'm sure there > are many other ways to approach it, but that's the way I've solved it. Thanks for the feedback. I hope more people can chime into this. -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] From rgerhards at hq.adiscon.com Fri Oct 9 14:47:39 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 9 Oct 2009 14:47:39 +0200 Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031BF@GRFEXC.intern.adiscon.com> Unfortunately, 2.0.6 is a very old and deprecated version. I really don't dig any longer into that code base (except, of course, for folks with support contracts). It really is not worth the hassle. You'll run into a couple of issues in the long term. I suggest upgrading at least to v3-stable. Project status is here: http://www.rsyslog.com/doc-status.html Sorry I have no better reply, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dag Wieers > Sent: Friday, October 09, 2009 2:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] Wrongly formatted messages from > kernel/netconsole > > On Thu, 8 Oct 2009, RB wrote: > > > On Thu, Oct 8, 2009 at 13:10, Dag Wieers wrote: > > > >> Also, I have noticed that our central rsyslog-server (more than 400 > >> systems log to it) have directories with names like: Detected/, > exiting/, > >> ext3_abort/, EXT3-fs/, journal/, last/, martian/, program/, > Remounting/, > >> Restarting/, ... So it is obvious that something is not working as > >> expected coming from the kernel. > > > > More than likely because you're trying to make directories based on > > the %hostname% property, which rsyslog assumes is a specific field. > > With the sloppier daemons (FreeBSD in particular), I've had far more > > luck using the %fromhost-ip% property (as well as the $ system > > properties for timestamps). Of course, that breaks down if you're > > doing relaying, but relying on values the other end sends you to > > create filesystem artifacts is dangerous at best anyway > > You are correct, that is exactly what we do. However with rsyslog > v2.0.6 > it seems there is no %FROMHOST-IP% and the %FROMHOST% property only > contains the IP address. Maybe there is something else I need to do to > get > the short hostname from DNS, rather than an IP on rsyslog v2 ? > > > >> but of course I cannot influence our production kernels to do the > right > >> thing. What can I do to have rsyslog accept the "wrong" thing ? :) > > > > Use %fromhost% or %fromhost-ip% to make the directory > > structures/filenames, and make a custom format if you need to handle > > the remaning lack of data (again, timestamp & host). I'm sure there > > are many other ways to approach it, but that's the way I've solved > it. > > Thanks for the feedback. I hope more people can chime into this. > > -- > -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- > [Any errors in spelling, tact or fact are transmission errors] > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From dag at wieers.com Fri Oct 9 16:17:27 2009 From: dag at wieers.com (Dag Wieers) Date: Fri, 9 Oct 2009 16:17:27 +0200 (CEST) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031BF@GRFEXC.intern.adiscon.com> References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71031BF@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 9 Oct 2009, Rainer Gerhards wrote: > Unfortunately, 2.0.6 is a very old and deprecated version. I really don't dig > any longer into that code base (except, of course, for folks with support > contracts). It really is not worth the hassle. You'll run into a couple of > issues in the long term. I suggest upgrading at least to v3-stable. Project > status is here: > > http://www.rsyslog.com/doc-status.html > > Sorry I have no better reply, Don't feel sorry, I understand. We are looking with Red Hat to rebase rsyslog to a newer version asap. We expect by RHEL5.5 to have either v3 or v4. No word yet from Red Hat what their target will be for RHEL5.5. (Hopefully the same as RHEL6) This is related to the bugzilla reports I send to the list recently. But it would have been nice to know whether there was a solution for this on v2 as well, even when I accept that it is old and unsupported here :-) Kind regards, -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] From aoz.syn at gmail.com Fri Oct 9 16:19:30 2009 From: aoz.syn at gmail.com (RB) Date: Fri, 9 Oct 2009 08:19:30 -0600 Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> Message-ID: <4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> On Fri, Oct 9, 2009 at 06:38, Dag Wieers wrote: > You are correct, that is exactly what we do. However with rsyslog v2.0.6 > it seems there is no %FROMHOST-IP% and the %FROMHOST% property only > contains the IP address. Maybe there is something else I need to do to get > the short hostname from DNS, rather than an IP on rsyslog v2 ? This usually has more to do with the host's DNS configuration than the syslog daemon - I've not looked at the 2.x code (ever), but if your host is configured with a resolver that will perform reverse lookups of the IPs and with an appropriate domain search list, you should be able to get the short hostnames. That can be shortcut by placing entries in /etc/hosts, but that obviously doesn't scale very well. From dag at wieers.com Fri Oct 9 16:36:23 2009 From: dag at wieers.com (Dag Wieers) Date: Fri, 9 Oct 2009 16:36:23 +0200 (CEST) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: <4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> <4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> Message-ID: On Fri, 9 Oct 2009, RB wrote: > On Fri, Oct 9, 2009 at 06:38, Dag Wieers wrote: >> You are correct, that is exactly what we do. However with rsyslog v2.0.6 >> it seems there is no %FROMHOST-IP% and the %FROMHOST% property only >> contains the IP address. Maybe there is something else I need to do to get >> the short hostname from DNS, rather than an IP on rsyslog v2 ? > > This usually has more to do with the host's DNS configuration than the > syslog daemon - I've not looked at the 2.x code (ever), but if your > host is configured with a resolver that will perform reverse lookups > of the IPs and with an appropriate domain search list, you should be > able to get the short hostnames. That can be shortcut by placing > entries in /etc/hosts, but that obviously doesn't scale very well. We do have a working DNS resolver and reverse lookups work fine. I assume that the problem is that with rsyslog v2 %FROMHOST% simply is not resolved. To be sure, I will have to dig into the sources. -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] From david at lang.hm Sat Oct 10 08:06:17 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 9 Oct 2009 23:06:17 -0700 (PDT) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: <4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com> <4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> Message-ID: On Fri, 9 Oct 2009, RB wrote: > On Fri, Oct 9, 2009 at 06:38, Dag Wieers wrote: >> You are correct, that is exactly what we do. However with rsyslog v2.0.6 >> it seems there is no %FROMHOST-IP% and the %FROMHOST% property only >> contains the IP address. Maybe there is something else I need to do to get >> the short hostname from DNS, rather than an IP on rsyslog v2 ? > > This usually has more to do with the host's DNS configuration than the > syslog daemon - I've not looked at the 2.x code (ever), but if your > host is configured with a resolver that will perform reverse lookups > of the IPs and with an appropriate domain search list, you should be > able to get the short hostnames. That can be shortcut by placing > entries in /etc/hosts, but that obviously doesn't scale very well. even the 4.x and 5.x do not populate the fromhost-ip if you have -x on the command line. I think this is a bug, but there are bigger bugs that are being worked on. David Lang From rgerhards at hq.adiscon.com Mon Oct 12 16:01:23 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 12 Oct 2009 16:01:23 +0200 Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com><4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031DA@GRFEXC.intern.adiscon.com> David, All, > even the 4.x and 5.x do not populate the fromhost-ip if you have -x on > the > command line. > > I think this is a bug, but there are bigger bugs that are being worked > on. Does anybody have a debug log where this happens? I do not see it in my lab (but I did not try hard). For the DNS cache, I am shuffling code in v5, and that would be a good time to address the bug. Raienr From marc.schiffbauer at mightycare.de Tue Oct 13 13:20:45 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 13 Oct 2009 13:20:45 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? Message-ID: <200910131320.45629.marc.schiffbauer@mightycare.de> Hello all, I am running a rsyslog server version 4.5.2 I tested the message queuing capabilities recently and noticed that queue files do not get cleaned up. Evertime rsyslog has to spool something (e.g. when the database server ist stopped) it creates new spoolfiles so the spool area will grow over time. Is this a bug or how can I handle those old queue files? This is the relevant part from my config: # enable message spooling $WorkDirectory /data/syslog-spool # default location for work (spool) files $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName dbq # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure TIA -Marc Schiffbauer -- Senior Consultant :: Solution Architect IT-Security :: Free Software :: GNU/Linux MightyCare Solutions GmbH http://www.mightycare.de Firmenangaben unter http://www.mightycare.de/#/de/impressum/ From rgerhards at hq.adiscon.com Tue Oct 13 14:16:47 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 13 Oct 2009 14:16:47 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com> This looks like a dirty shutdown. In general, rsyslog cleans up when processing completes. Do you have a .qi file inside that directory? Absence of a .qi file - when the instance is not running - is indication of an issue during shutdown. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, October 13, 2009 1:21 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Hello all, > > I am running a rsyslog server version 4.5.2 > > I tested the message queuing capabilities recently and noticed that > queue > files do not get cleaned up. Evertime rsyslog has to spool something > (e.g. > when the database server ist stopped) it creates new spoolfiles so the > spool > area will grow over time. > > Is this a bug or how can I handle those old queue files? > > This is the relevant part from my config: > > # enable message spooling > $WorkDirectory /data/syslog-spool # default location for work (spool) > files > $ActionQueueType LinkedList # use asynchronous processing > $ActionQueueFileName dbq # set file name, also enables disk > mode > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > TIA > -Marc Schiffbauer > -- > Senior Consultant :: Solution Architect > IT-Security :: Free Software :: GNU/Linux > MightyCare Solutions GmbH http://www.mightycare.de > Firmenangaben unter http://www.mightycare.de/#/de/impressum/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Oct 13 15:31:44 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 13 Oct 2009 15:31:44 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com> Message-ID: <200910131531.44817.marc.schiffbauer@mightycare.de> Hello Rainer, Am Dienstag, 13. Oktober 2009 14:16:47 schrieb Rainer Gerhards: > This looks like a dirty shutdown. I am not aware of any... > In general, rsyslog cleans up when > processing completes. OK. This is good to know. So something went wrong here: I produced a lot of logging traffic using the logger shell command. Then I stopped that postgres server to test that spooling works. After a while I started it again. After a while rsyslog reconnected to the db and logging to the DB was working again. > Do you have a .qi file inside that directory? Absence > of a .qi file - when the instance is not running - is indication of an > issue during shutdown. There is a .qi file when rsyslog is NOT running and ist being removed when I start it again. I will remove stop rsyslog now, then remove *everything* from the spool- directory and start it up again. Then I will repeat the spool-test. Thanks for helping -Marc > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > > Sent: Tuesday, October 13, 2009 1:21 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > > Hello all, > > > > I am running a rsyslog server version 4.5.2 > > > > I tested the message queuing capabilities recently and noticed that > > queue > > files do not get cleaned up. Evertime rsyslog has to spool something > > (e.g. > > when the database server ist stopped) it creates new spoolfiles so the > > spool > > area will grow over time. > > > > Is this a bug or how can I handle those old queue files? > > > > This is the relevant part from my config: > > > > # enable message spooling > > $WorkDirectory /data/syslog-spool # default location for work (spool) > > files > > $ActionQueueType LinkedList # use asynchronous processing > > $ActionQueueFileName dbq # set file name, also enables disk > > mode > > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > > > > TIA > > -Marc Schiffbauer -- Senior Consultant :: Solution Architect IT-Security :: Free Software :: GNU/Linux Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 MightyCare Solutions GmbH http://www.mightycare.de Firmenangaben unter http://www.mightycare.de/#/de/impressum/ From marc.schiffbauer at mightycare.de Tue Oct 13 15:37:52 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 13 Oct 2009 15:37:52 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <200910131531.44817.marc.schiffbauer@mightycare.de> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com> <200910131531.44817.marc.schiffbauer@mightycare.de> Message-ID: <200910131537.52917.marc.schiffbauer@mightycare.de> Am Dienstag, 13. Oktober 2009 15:31:44 schrieb Marc Schiffbauer: > > Do you have a .qi file inside that directory? Absence > > of a .qi file - when the instance is not running - is indication of an > > issue during shutdown. > > There is a .qi file when rsyslog is NOT running and ist being removed when > I start it again. > Now this is strange: To prepare the new test I stopped rsyslog again, and now it did NOT create the .qi file. Another restart: same result. This rsyslog is idle ATM BTW. -Marc -- Senior Consultant :: Solution Architect IT-Security :: Free Software :: GNU/Linux Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 MightyCare Solutions GmbH http://www.mightycare.de Firmenangaben unter http://www.mightycare.de/#/de/impressum/ From rgerhards at hq.adiscon.com Tue Oct 13 15:40:07 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 13 Oct 2009 15:40:07 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com><200910131531.44817.marc.schiffbauer@mightycare.de> <200910131537.52917.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031E6@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, October 13, 2009 3:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Am Dienstag, 13. Oktober 2009 15:31:44 schrieb Marc Schiffbauer: > > > Do you have a .qi file inside that directory? Absence > > > of a .qi file - when the instance is not running - is indication of > an > > > issue during shutdown. > > > > There is a .qi file when rsyslog is NOT running and ist being removed > when > > I start it again. > > > > Now this is strange: To prepare the new test I stopped rsyslog again, > and now > it did NOT create the .qi file. Another restart: same result. > > This rsyslog is idle ATM BTW. Both sounds like there simply was nothing in the queue... > > -Marc > -- > Senior Consultant :: Solution Architect > IT-Security :: Free Software :: GNU/Linux > Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 > MightyCare Solutions GmbH http://www.mightycare.de > Firmenangaben unter http://www.mightycare.de/#/de/impressum/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Oct 13 15:41:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 13 Oct 2009 15:41:17 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA71031E5@GRFEXC.intern.adiscon.com> <200910131531.44817.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031E7@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, October 13, 2009 3:32 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > Am Dienstag, 13. Oktober 2009 14:16:47 schrieb Rainer Gerhards: > > This looks like a dirty shutdown. > > I am not aware of any... > > > In general, rsyslog cleans up when > > processing completes. > > OK. This is good to know. So something went wrong here: I produced a > lot of > logging traffic using the logger shell command. Then I stopped that > postgres > server to test that spooling works. After a while I started it again. > After a while rsyslog reconnected to the db and logging to the DB was > working > again. > > > > Do you have a .qi file inside that directory? Absence > > of a .qi file - when the instance is not running - is indication of > an > > issue during shutdown. > > There is a .qi file when rsyslog is NOT running and ist being removed > when I > start it again. That is OK, because then rsyslog keeps the information in memory once it runs, to safe i/o time. You can modify that by specifying a different queue configuration, but you need to understand how the queue system works in order to do so. A good read is the queue doc. Rainer > > I will remove stop rsyslog now, then remove *everything* from the > spool- > directory and start it up again. > > Then I will repeat the spool-test. > > Thanks for helping > -Marc > > > > > Rainer > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > > > Sent: Tuesday, October 13, 2009 1:21 PM > > > To: rsyslog at lists.adiscon.com > > > Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > > > > Hello all, > > > > > > I am running a rsyslog server version 4.5.2 > > > > > > I tested the message queuing capabilities recently and noticed that > > > queue > > > files do not get cleaned up. Evertime rsyslog has to spool > something > > > (e.g. > > > when the database server ist stopped) it creates new spoolfiles so > the > > > spool > > > area will grow over time. > > > > > > Is this a bug or how can I handle those old queue files? > > > > > > This is the relevant part from my config: > > > > > > # enable message spooling > > > $WorkDirectory /data/syslog-spool # default location for work > (spool) > > > files > > > $ActionQueueType LinkedList # use asynchronous processing > > > $ActionQueueFileName dbq # set file name, also enables > disk > > > mode > > > $ActionResumeRetryCount -1 # infinite retries on insert > failure > > > > > > > > > TIA > > > -Marc Schiffbauer > > -- > Senior Consultant :: Solution Architect > IT-Security :: Free Software :: GNU/Linux > Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 > MightyCare Solutions GmbH http://www.mightycare.de > Firmenangaben unter http://www.mightycare.de/#/de/impressum/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Oct 13 16:27:44 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 13 Oct 2009 16:27:44 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031E7@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <200910131531.44817.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA71031E7@GRFEXC.intern.adiscon.com> Message-ID: <200910131627.44897.marc.schiffbauer@mightycare.de> Am Dienstag, 13. Oktober 2009 15:41:17 schrieb Rainer Gerhards: > > > > There is a .qi file when rsyslog is NOT running and ist being removed > > when I > > start it again. > > That is OK, because then rsyslog keeps the information in memory once it > runs, to safe i/o time. You can modify that by specifying a different queue > configuration, but you need to understand how the queue system works in > order to do so. A good read is the queue doc. > > Rainer > > > I will remove stop rsyslog now, then remove *everything* from the > > spool- > > directory and start it up again. > > > > Then I will repeat the spool-test. > > As it seems, rsyslog will not write a .qi file in all cases. New tests were not all successful (with rsyslog under load): OK => spooling while DB is offline OK => reconnect to DB OK => despooling while still under load and spooling to disk Now the following produced "stale" queue files and a loss of messages I guess: NOT OK => despooling while under load and while spooling to disk, then stopping rsyslogd (stopped via /etc/init.d/syslog stop) -> no .qi file has been created! after making sure there are no more rsyslog processes I started it again. The spool files will not be cleared (no load anymore and DB started of course) bug? TIA -Marc -- Senior Consultant :: Solution Architect IT-Security :: Free Software :: GNU/Linux Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 MightyCare Solutions GmbH http://www.mightycare.de Firmenangaben unter http://www.mightycare.de/#/de/impressum/ From rgerhards at hq.adiscon.com Tue Oct 13 16:35:49 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 13 Oct 2009 16:35:49 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><200910131531.44817.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA71031E7@GRFEXC.intern.adiscon.com> <200910131627.44897.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031EA@GRFEXC.intern.adiscon.com> > As it seems, rsyslog will not write a .qi file in all cases. not always, but it should always write them when necessary ;) > > New tests were not all successful (with rsyslog under load): > OK => spooling while DB is offline > OK => reconnect to DB > OK => despooling while still under load and spooling to disk > > Now the following produced "stale" queue files and a loss of messages I > guess: > NOT OK => despooling while under load and while spooling to disk, > then stopping rsyslogd > (stopped via /etc/init.d/syslog stop) > -> no .qi file has been created! > after making sure there are no more rsyslog processes I started it > again. > The spool files will not be cleared (no load anymore and DB started of > course) > > bug? Smells like one. I re-checked your config, I think it does not include a directive to tell the engine to persist messages on shutdown. Even if it does not do that, it should clean up the files. A debug log would be most useful. Note that the v4 engine, and more so the v5 engine, has had a number of important changes, and people only gradually begin to utilize it in practice. The past couple of month, I had comparatively few bug reports, but the past three weeks or so people tend to adopt the new features and consequently the "bug rate" goes up. This is good, as it helps iron out things, but it is also somewhat bad, because I need to prioritize work and for some bugs that means I have not to touch too many things at once. Looking forward for additional information. Rainer PS: please all keep contributing bug reports! It is really useful to have them (even better if they are timely ;)) as my lab can not cover everything practice does ;) > > TIA > -Marc > -- > Senior Consultant :: Solution Architect > IT-Security :: Free Software :: GNU/Linux > Mobile +49_151_16227402 :: Fon +49_6187_9058695 :: Fax +49_6187_900157 > MightyCare Solutions GmbH http://www.mightycare.de > Firmenangaben unter http://www.mightycare.de/#/de/impressum/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Oct 14 00:40:14 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 13 Oct 2009 15:40:14 -0700 (PDT) Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031DA@GRFEXC.intern.adiscon.com> References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com><4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71031DA@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 12 Oct 2009, Rainer Gerhards wrote: > David, All, > >> even the 4.x and 5.x do not populate the fromhost-ip if you have -x on >> the >> command line. >> >> I think this is a bug, but there are bigger bugs that are being worked >> on. > > Does anybody have a debug log where this happens? I do not see it in my lab > (but I did not try hard). For the DNS cache, I am shuffling code in v5, and > that would be a good time to address the bug. you should have it in the debug logs I've made available for the segfault problems. Those instances are running with -x David Lang From rgerhards at hq.adiscon.com Wed Oct 14 07:14:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 14 Oct 2009 07:14:44 +0200 Subject: [rsyslog] Wrongly formatted messages from kernel/netconsole References: <4255c2570910081228i39d44560n3d7a6ffb2574c0fd@mail.gmail.com><4255c2570910090719m2e7b9c61t999ad92add962c8c@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA71031DA@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71031F1@GRFEXC.intern.adiscon.com> thanks, then I know where to look at :) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, October 14, 2009 12:40 AM > To: rsyslog-users > Subject: Re: [rsyslog] Wrongly formatted messages from > kernel/netconsole > > On Mon, 12 Oct 2009, Rainer Gerhards wrote: > > > David, All, > > > >> even the 4.x and 5.x do not populate the fromhost-ip if you have -x > on > >> the > >> command line. > >> > >> I think this is a bug, but there are bigger bugs that are being > worked > >> on. > > > > Does anybody have a debug log where this happens? I do not see it in > my lab > > (but I did not try hard). For the DNS cache, I am shuffling code in > v5, and > > that would be a good time to address the bug. > > you should have it in the debug logs I've made available for the > segfault > problems. Those instances are running with -x > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From tbergfeld at hq.adiscon.com Thu Oct 15 10:59:17 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Thu, 15 Oct 2009 10:59:17 +0200 Subject: [rsyslog] rsyslog 5.1.6 (v5-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103210@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.1.6, a member of the v5-beta branch. This is a bug-fixing release containing some important fixes. It is a recommended update for all users of the v5-beta. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article413.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-180.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From marc.schiffbauer at mightycare.de Thu Oct 15 14:14:33 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Thu, 15 Oct 2009 14:14:33 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? Message-ID: <200910151414.33860.marc.schiffbauer@mightycare.de> [ sorry, attachement in first try was too big, next try with shortened debug log ] Am Dienstag, 13. Oktober 2009 16:35:49 schrieb Rainer Gerhards: > > As it seems, rsyslog will not write a .qi file in all cases. > > not always, but it should always write them when necessary ;) > > > New tests were not all successful (with rsyslog under load): > > OK => spooling while DB is offline > > OK => reconnect to DB > > OK => despooling while still under load and spooling to disk > > > > Now the following produced "stale" queue files and a loss of messages I > > guess: > > NOT OK => despooling while under load and while spooling to disk, > > then stopping rsyslogd > > (stopped via /etc/init.d/syslog stop) > > -> no .qi file has been created! > > after making sure there are no more rsyslog processes I started it > > again. > > The spool files will not be cleared (no load anymore and DB started of > > course) > > > > bug? > > Smells like one. I re-checked your config, I think it does not include a > directive to tell the engine to persist messages on shutdown. Oh, I did not know that there is a directive for it.. good to know. > Even if it > does not do that, it should clean up the files. A debug log would be most > useful. Trying to produce one it seems a difficult task to me because rsyslogd seems to behave completely different when in debug mode... In one console I started: /sbin/rsyslogd -c 4 -f /etc/rsyslog.conf -d &> rsyslog-debug.log Then I stopped postgres: /etc/init.d/postgresql stop Now I started a logger loop: while true; do logger -t spool-test "no real message here... (PID=$$)"; done After the first spool file is being created I stop the loop again. Now I want to stop rsyslogd by calling "killall rsyslogd" I had to call this several times before rsyslogd really did exit. After the first attemots it seemed that it did try to reach the DB in a loop and did not attempt to prepare for exit. See the log attached. Is this helpful? -Marc > > Note that the v4 engine, and more so the v5 engine, has had a number of > important changes, and people only gradually begin to utilize it in > practice. The past couple of month, I had comparatively few bug reports, > but the past three weeks or so people tend to adopt the new features and > consequently the "bug rate" goes up. This is good, as it helps iron out > things, but it is also somewhat bad, because I need to prioritize work and > for some bugs that means I have not to touch too many things at once. > > Looking forward for additional information. > > Rainer > PS: please all keep contributing bug reports! It is really useful to have > them (even better if they are timely ;)) as my lab can not cover everything > practice does ;) > -- Senior Consultant :: Solution Architect IT-Security :: Free Software :: GNU/Linux From rgerhards at hq.adiscon.com Thu Oct 15 14:21:42 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 15 Oct 2009 14:21:42 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910151414.33860.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710321F@GRFEXC.intern.adiscon.com> Hi Marc, the mailing list is set to be very restrictive in what it accepts. So the attachment did not went through ;) You can mail it to me at rgerhards at gmail.com - but please let me know when you have done, I do not regularly check that mailbox. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Thursday, October 15, 2009 2:15 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > [ sorry, attachement in first try was too big, next try with shortened > debug > log ] > > Am Dienstag, 13. Oktober 2009 16:35:49 schrieb Rainer Gerhards: > > > As it seems, rsyslog will not write a .qi file in all cases. > > > > not always, but it should always write them when necessary ;) > > > > > New tests were not all successful (with rsyslog under load): > > > OK => spooling while DB is offline > > > OK => reconnect to DB > > > OK => despooling while still under load and spooling to disk > > > > > > Now the following produced "stale" queue files and a loss of > messages I > > > guess: > > > NOT OK => despooling while under load and while spooling to disk, > > > then stopping rsyslogd > > > (stopped via /etc/init.d/syslog stop) > > > -> no .qi file has been created! > > > after making sure there are no more rsyslog processes I started it > > > again. > > > The spool files will not be cleared (no load anymore and DB started > of > > > course) > > > > > > bug? > > > > Smells like one. I re-checked your config, I think it does not > include a > > directive to tell the engine to persist messages on shutdown. > > Oh, I did not know that there is a directive for it.. good to know. > > > Even if it > > does not do that, it should clean up the files. A debug log would be > most > > useful. > > Trying to produce one it seems a difficult task to me because rsyslogd > seems > to behave completely different when in debug mode... > > In one console I started: > /sbin/rsyslogd -c 4 -f /etc/rsyslog.conf -d &> rsyslog-debug.log > > Then I stopped postgres: > /etc/init.d/postgresql stop > > Now I started a logger loop: > while true; do logger -t spool-test "no real message here... > (PID=$$)"; > done > > After the first spool file is being created I stop the loop again. > > Now I want to stop rsyslogd by calling "killall rsyslogd" > > I had to call this several times before rsyslogd really did exit. After > the > first attemots it seemed that it did try to reach the DB in a loop and > did not > attempt to prepare for exit. > > See the log attached. Is this helpful? > > -Marc > > > > > Note that the v4 engine, and more so the v5 engine, has had a number > of > > important changes, and people only gradually begin to utilize it in > > practice. The past couple of month, I had comparatively few bug > reports, > > but the past three weeks or so people tend to adopt the new features > and > > consequently the "bug rate" goes up. This is good, as it helps iron > out > > things, but it is also somewhat bad, because I need to prioritize > work and > > for some bugs that means I have not to touch too many things at > once. > > > > Looking forward for additional information. > > > > Rainer > > PS: please all keep contributing bug reports! It is really useful to > have > > them (even better if they are timely ;)) as my lab can not cover > everything > > practice does ;) > > > -- > Senior Consultant :: Solution Architect > IT-Security :: Free Software :: GNU/Linux From marc.schiffbauer at mightycare.de Thu Oct 15 19:33:11 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Thu, 15 Oct 2009 19:33:11 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710321F@GRFEXC.intern.adiscon.com> References: <200910151414.33860.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA710321F@GRFEXC.intern.adiscon.com> Message-ID: <200910151933.11296.marc.schiffbauer@mightycare.de> Am Donnerstag, 15. Oktober 2009 14:21:42 schrieb Rainer Gerhards: > Hi Marc, > > the mailing list is set to be very restrictive in what it accepts. So the > attachment did not went through ;) You can mail it to me at > rgerhards at gmail.com - but please let me know when you have done, I do not > regularly check that mailbox. Hi Rainer, mail sent! TIA -Marc From marc.schiffbauer at mightycare.de Fri Oct 16 12:52:26 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Fri, 16 Oct 2009 12:52:26 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71031EA@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <200910131627.44897.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA71031EA@GRFEXC.intern.adiscon.com> Message-ID: <200910161252.26765.marc.schiffbauer@mightycare.de> Hi Rainer, Am Dienstag, 13. Oktober 2009 16:35:49 schrieb Rainer Gerhards: > > > > bug? > > Smells like one. I re-checked your config, I think it does not include a > directive to tell the engine to persist messages on shutdown. Even if it > does not do that, it should clean up the files. A debug log would be most > useful. I now added "$ActionQueueSaveOnShutdown on" to the configuration. But I still get queuefiles without a .qi file after rsyslog has quit by SIGTERM -Marc From rgerhards at hq.adiscon.com Fri Oct 16 14:07:23 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 16 Oct 2009 14:07:23 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><200910131627.44897.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA71031EA@GRFEXC.intern.adiscon.com> <200910161252.26765.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103231@GRFEXC.intern.adiscon.com> busy day today... I had a quick look at the debug log, it looks like there indeed is a shutdown issue. Out of curiosity, would you be willing to test a current (v5) devel version? I am currently working on an enhanced queue engine, which has been greatly simplified and speeded up. During that work, I notice that there exist some potential issue in v4, that I had thought to not surface in practice. Queue shutdown is very, very complicated. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Friday, October 16, 2009 12:52 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > Hi Rainer, > > Am Dienstag, 13. Oktober 2009 16:35:49 schrieb Rainer Gerhards: > > > > > > bug? > > > > Smells like one. I re-checked your config, I think it does not > include a > > directive to tell the engine to persist messages on shutdown. Even if > it > > does not do that, it should clean up the files. A debug log would be > most > > useful. > > I now added "$ActionQueueSaveOnShutdown on" to the configuration. But I > still > get queuefiles without a .qi file after rsyslog has quit by SIGTERM > > -Marc > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Fri Oct 16 15:55:40 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Fri, 16 Oct 2009 15:55:40 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103231@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <200910161252.26765.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103231@GRFEXC.intern.adiscon.com> Message-ID: <200910161555.40812.marc.schiffbauer@mightycare.de> Am Freitag, 16. Oktober 2009 14:07:23 schrieb Rainer Gerhards: > busy day today... I had a quick look at the debug log, it looks like there > indeed is a shutdown issue. > Do you think you can get it fixed in 4.5.4? I would be pleased to test patches... > Out of curiosity, would you be willing to test a current (v5) devel > version? As this is at a customers system I can do only a short test to see whether it works or not. > I am currently working on an enhanced queue engine, which has > been greatly simplified and speeded up. During that work, I notice that > there exist some potential issue in v4, that I had thought to not surface > in practice. Queue shutdown is very, very complicated. > I see. Sounds good for future versions.... Are these DA-queuing features are present in the 3.xx version already? Maybe I should consider using 3.xx the instead if its more robust. Thanks -Marc > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > > Sent: Friday, October 16, 2009 12:52 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > > > > Hi Rainer, > > > > Am Dienstag, 13. Oktober 2009 16:35:49 schrieb Rainer Gerhards: > > > > bug? > > > > > > Smells like one. I re-checked your config, I think it does not > > > > include a > > > > > directive to tell the engine to persist messages on shutdown. Even if > > > > it > > > > > does not do that, it should clean up the files. A debug log would be > > > > most > > > > > useful. > > > > I now added "$ActionQueueSaveOnShutdown on" to the configuration. But I > > still > > get queuefiles without a .qi file after rsyslog has quit by SIGTERM > > > > -Marc > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Oct 16 15:59:18 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 16 Oct 2009 15:59:18 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><200910161252.26765.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA7103231@GRFEXC.intern.adiscon.com> <200910161555.40812.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103234@GRFEXC.intern.adiscon.com> just quickly > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Friday, October 16, 2009 3:56 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Am Freitag, 16. Oktober 2009 14:07:23 schrieb Rainer Gerhards: > > busy day today... I had a quick look at the debug log, it looks like > there > > indeed is a shutdown issue. > > > > Do you think you can get it fixed in 4.5.4? I would be pleased to test > patches... > > > > Out of curiosity, would you be willing to test a current (v5) devel > > version? > > As this is at a customers system I can do only a short test to see > whether it > works or not. > > > > I am currently working on an enhanced queue engine, which has > > been greatly simplified and speeded up. During that work, I notice > that > > there exist some potential issue in v4, that I had thought to not > surface > > in practice. Queue shutdown is very, very complicated. > > > > I see. Sounds good for future versions.... > > > Are these DA-queuing features are present in the 3.xx version already? > Maybe I should consider using 3.xx the instead if its more robust. Indeed, they are. Raienr From marc.schiffbauer at mightycare.de Fri Oct 16 16:20:50 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Fri, 16 Oct 2009 16:20:50 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <200910161555.40812.marc.schiffbauer@mightycare.de> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103231@GRFEXC.intern.adiscon.com> <200910161555.40812.marc.schiffbauer@mightycare.de> Message-ID: <200910161620.50460.marc.schiffbauer@mightycare.de> Am Freitag, 16. Oktober 2009 15:55:40 schrieb Marc Schiffbauer: > Am Freitag, 16. Oktober 2009 14:07:23 schrieb Rainer Gerhards: > > busy day today... I had a quick look at the debug log, it looks like > > there indeed is a shutdown issue. > > Do you think you can get it fixed in 4.5.4? I would be pleased to test > patches... I meant 4.4.3 of course ;-) From marc.schiffbauer at mightycare.de Fri Oct 16 16:32:20 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Fri, 16 Oct 2009 16:32:20 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103234@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <200910161555.40812.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103234@GRFEXC.intern.adiscon.com> Message-ID: <200910161632.20392.marc.schiffbauer@mightycare.de> Am Freitag, 16. Oktober 2009 15:59:18 schrieb Rainer Gerhards: > just quickly > > > > > Are these DA-queuing features are present in the 3.xx version already? > > Maybe I should consider using 3.xx the instead if its more robust. > > Indeed, they are. But 3.22.1 seems to have the same issues with DA-spooling :-( I will do some more testing on monday. -Marc From rgerhards at hq.adiscon.com Fri Oct 16 16:39:16 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 16 Oct 2009 16:39:16 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><200910161555.40812.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA7103234@GRFEXC.intern.adiscon.com> <200910161632.20392.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103235@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Friday, October 16, 2009 4:32 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Am Freitag, 16. Oktober 2009 15:59:18 schrieb Rainer Gerhards: > > just quickly > > > > > > > > Are these DA-queuing features are present in the 3.xx version > already? > > > Maybe I should consider using 3.xx the instead if its more robust. > > > > Indeed, they are. > > But 3.22.1 seems to have the same issues with DA-spooling :-( > I will do some more testing on monday. One problem is that most people didn't use the enhanced queing modes until lately. So even while the versions were out for quite some while, and there were no bug reports, there seem to be some hidden issues. I have been receiving bug reports the past three month or so and I am working to get these things straight. I consider these reports to be a good sign, as it indicates the functionality gradually is adopted in practice. Right now, I am working on a much more streamlined queue engine in v5. Queue shutdown is a very complex operation, and I have been able to greatly simplify processing. Also, there has been a tremendous amount of work gone into performance optimization, where the simplified engine will also set new marks, I think. Of course, that doesn't help you immediately. I just thought I try to convey at least some important bits of "the big picture". Rainer From david at lang.hm Fri Oct 16 22:05:14 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 16 Oct 2009 13:05:14 -0700 (PDT) Subject: [rsyslog] disabling doc installation Message-ID: is there a configure option that will skip populating everything under the DATAROOT directory? while I see the value of having everything there by default, on my server builds I don't want to waste the space of having everything there, so I end up having to create the package, and then manually modify it. David Lang From sovrevage at gmail.com Mon Oct 19 12:46:17 2009 From: sovrevage at gmail.com (=?ISO-8859-1?Q?Stian_=D8vrev=E5ge?=) Date: Mon, 19 Oct 2009 12:46:17 +0200 Subject: [rsyslog] timegenerated and timereported Message-ID: timegenerated - timestamp when the message was RECEIVED. Always in high resolution timereported - timestamp from the message. Resolution depends on what was provided in the message (in most cases, only seconds) Neither of these fit into either a mysql datetime nor unixtime field type! They come in the format "Oct 19 12:32:54", exacly what is high resolution about that I don't know. So, does rsyslog provide a way to easily fit these into a time-aware mysql field type? My template looks like this: $template SyslogAllInsert,"INSERT INTO `Syslog`.`AllMessages` (`ID` ,`timeGenerated` ,`timeReported` ,`source` ,`programName` ,`syslogTag` ,`syslogFacility` ,`syslogSeverity` ,`message`)VALUES ( NULL , '%timegenerated%', '%timereported%', '%source%', '%programname%', '%syslogtag%', '%syslogfacility%', '%syslogseverity%', '%msg%');",sql TIA Cheers, Stian ?vrev?ge From dirk.schulz at kinzesberg.de Mon Oct 19 13:51:15 2009 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Mon, 19 Oct 2009 13:51:15 +0200 Subject: [rsyslog] Getting Facility from Synlog-NG In-Reply-To: <87a1ae540910080951g6c7106fby53979808642cb537@mail.gmail.com> References: <4ACCA646.8050209@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA7103196@GRFEXC.intern.adiscon.com> <4ACE0C3B.7090207@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA71031AC@GRFEXC.intern.adiscon.com> <4ACE11D9.1020806@kinzesberg.de> <87a1ae540910080951g6c7106fby53979808642cb537@mail.gmail.com> Message-ID: <4ADC52B3.4050806@kinzesberg.de> I fould the problem - I have to admit it was me. :-( Had a configuration error (cleverly combined with a research error). It is working fine now. Thanks a lot for you help. Dirk Sayan Chowdhury schrieb: > Hello Dirk, I think Rainer wants you to add this to your rsyslog.conf file, > and send him the file that gets saved because of adding this statement. So > as to debug the issue. > You will need to restart rsyslog to pick up the config change > Hope this helps. > Sayan > > On Thu, Oct 8, 2009 at 9:52 PM, Dirk H. Schulz wrote: > > >> Rainer Gerhards schrieb: >> >>>>> check doc for exact syntax): >>>>> >>>>> $template raw,"%rawmsg%\n" >>>>> *.* /path/to/file;raw >>>>> >>>>> >>>>> >>>> I did not find these terms, neither in rsyslog.conf on the central >>>> logserver servers nor in syslog-ng.conf on the log client servers. >>>> Rsyslog is 3.18.6-4 on Debian 5. >>>> >>>> >>> Well, these are the config statements for the central rsyslog instance... >>> >>> >> Does that mean that I am lacking a template statement in the central >> rsyslog config? >> >> Dirk >> >>> Rainer >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Oct 20 11:32:55 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 11:32:55 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <200910131320.45629.marc.schiffbauer@mightycare.de><200910161555.40812.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA7103234@GRFEXC.intern.adiscon.com> <200910161632.20392.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103259@GRFEXC.intern.adiscon.com> Marc, I finally had some time to look into the issue. The problem actually is rooted in the shutdown sequence, which is rather complex in v3/v4. I am right now working on great simplification in v5, which will lead to more robust and even better performing code. Thanks to that work, I think I was able to quickly find the culprit and also to develop a patch. In my lab it works, and in theory it works as well, but practice is always another beast... So I would appreciate if you could give it a try. It is a pre-release of what once will become 4.4.3 and is available at: http://download.rsyslog.com/rsyslog/pre/rsyslog-4.4.3.tar.gz FYI, the actual patch is small: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=8e45fd5c8e7df45532c5d239 676cd92107c9e942 Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Friday, October 16, 2009 4:32 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Am Freitag, 16. Oktober 2009 15:59:18 schrieb Rainer Gerhards: > > just quickly > > > > > > > > Are these DA-queuing features are present in the 3.xx version > already? > > > Maybe I should consider using 3.xx the instead if its more robust. > > > > Indeed, they are. > > But 3.22.1 seems to have the same issues with DA-spooling :-( > I will do some more testing on monday. > > -Marc > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Oct 20 12:50:45 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 20 Oct 2009 12:50:45 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103259@GRFEXC.intern.adiscon.com> References: <200910131320.45629.marc.schiffbauer@mightycare.de> <200910161632.20392.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103259@GRFEXC.intern.adiscon.com> Message-ID: <200910201250.45820.marc.schiffbauer@mightycare.de> Am Dienstag, 20. Oktober 2009 11:32:55 schrieb Rainer Gerhards: > Marc, Hi Rainer, > > I finally had some time to look into the issue. The problem actually is > rooted in the shutdown sequence, which is rather complex in v3/v4. I am > right now working on great simplification in v5, which will lead to more > robust and even better performing code. > > Thanks to that work, I think I was able to quickly find the culprit and > also to develop a patch. In my lab it works, and in theory it works as > well, but practice is always another beast... > > So I would appreciate if you could give it a try. It is a pre-release of > what once will become 4.4.3 and is available at: > > http://download.rsyslog.com/rsyslog/pre/rsyslog-4.4.3.tar.gz Thanks very much! Now it works as expected with one exception: In my test that I did now I put an ID into each log-message to see if no messages get lost. That way I found that one message always gets lost when I do the following: * DB and rsyslogd started * Start logger loop with ID (slow message rate): X=1; while true; do logger -t spool-test "no real message here... (ID=$X)"; X=$((X+1)); sleep 3; done & * stop DB server, spooling to mem starts * start another logger loop to force DA spooling: while true; do logger -t spool-test "no real message here..."; done & * watch spool dir: as soon as a queue file is being created: stop both loops ("fg" followeg by ctrl-c, two times) * stop rsyslogd: qi is file being created now! now there should be no lost message, right? * start DB server * start rsyslogd * check the log: *one* of those ID=$X messages is lost. I did that test two times. -Marc From rgerhards at hq.adiscon.com Tue Oct 20 13:36:48 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 13:36:48 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? Message-ID: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> Can you mail me a debug log where the message loss can be seen? rainer ----- Urspr?ngliche Nachricht ----- Von: "Marc Schiffbauer" An: "rsyslog-users" Gesendet: 20.10.09 12:51 Betreff: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? Am Dienstag, 20. Oktober 2009 11:32:55 schrieb Rainer Gerhards: > Marc, Hi Rainer, > > I finally had some time to look into the issue. The problem actually is > rooted in the shutdown sequence, which is rather complex in v3/v4. I am > right now working on great simplification in v5, which will lead to more > robust and even better performing code. > > Thanks to that work, I think I was able to quickly find the culprit and > also to develop a patch. In my lab it works, and in theory it works as > well, but practice is always another beast... > > So I would appreciate if you could give it a try. It is a pre-release of > what once will become 4.4.3 and is available at: > > http://download.rsyslog.com/rsyslog/pre/rsyslog-4.4.3.tar.gz Thanks very much! Now it works as expected with one exception: In my test that I did now I put an ID into each log-message to see if no messages get lost. That way I found that one message always gets lost when I do the following: * DB and rsyslogd started * Start logger loop with ID (slow message rate): X=1; while true; do logger -t spool-test "no real message here... (ID=$X)"; X=$((X+1)); sleep 3; done & * stop DB server, spooling to mem starts * start another logger loop to force DA spooling: while true; do logger -t spool-test "no real message here..."; done & * watch spool dir: as soon as a queue file is being created: stop both loops ("fg" followeg by ctrl-c, two times) * stop rsyslogd: qi is file being created now! now there should be no lost message, right? * start DB server * start rsyslogd * check the log: *one* of those ID=$X messages is lost. I did that test two times. -Marc _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Oct 20 14:48:16 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 20 Oct 2009 14:48:16 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> Message-ID: <200910201448.16203.marc.schiffbauer@mightycare.de> Am Dienstag, 20. Oktober 2009 13:36:48 schrieb Rainer Gerhards: > Can you mail me a debug log where the message loss can be seen? Hi Rainer, I sent the logs to your gmail address. The lost message contains "ID=12". TIA -Marc From rgerhards at hq.adiscon.com Tue Oct 20 15:23:31 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 15:23:31 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200910201448.16203.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> interesting... the log looks like all works ok (of course, without the missing message ;)). Can you send me the contents of the .qi file after the shutdown? I am asking because it should contain the missing message (with that version of the queue engine). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, October 20, 2009 2:48 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > Am Dienstag, 20. Oktober 2009 13:36:48 schrieb Rainer Gerhards: > > Can you mail me a debug log where the message loss can be seen? > > Hi Rainer, > > I sent the logs to your gmail address. > > The lost message contains "ID=12". > > > TIA > -Marc > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marc.schiffbauer at mightycare.de Tue Oct 20 16:12:59 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 20 Oct 2009 16:12:59 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200910201448.16203.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> Message-ID: <200910201612.59316.marc.schiffbauer@mightycare.de> Am Dienstag, 20. Oktober 2009 15:23:31 schrieb Rainer Gerhards: > interesting... the log looks like all works ok (of course, without the > missing message ;)). Can you send me the contents of the .qi file after the > shutdown? I am asking because it should contain the missing message (with > that version of the queue engine). > > Rainer > Hi Rainer, I sent you another gmail. I repeated the test and this time I saved everything from the queue-Directory (one queue file + the dbq.qi file) before restarting rsyslogd The missing log entry in this try: ID=7 TIA -Marc From marc.schiffbauer at mightycare.de Tue Oct 20 16:39:42 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Tue, 20 Oct 2009 16:39:42 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200910201448.16203.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> Message-ID: <200910201639.43143.marc.schiffbauer@mightycare.de> Hi Rainer, new mail on gmail, now with attachment ;) TIA -Marc From anichols at trumped.org Tue Oct 20 17:32:07 2009 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 20 Oct 2009 09:32:07 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase in write volume (bytes/sec) to NAS - can someone help shed light? Message-ID: Hello, We have two rsyslog "writer" instances which are receiving syslog messages and writing them to an NFS mount on a NAS. The NAS is connected via 10gb ethernet but has a number of other applications writing to it as well. Typical write volume for our log writers is 2-3Mbps to the NAS. Yesterday I attempted to resolve some complaints with logs being "bursty" - events appear to be written in batches which sounded consistent with buffering on the rsyslog daemon or not syncing the file with every message. These bursts could come as infrequently as every 30 seconds however, which seems excessive. I took a number of steps to remedy this but ultimately had to reverse the changes because write volume (measured at the disks) to the NAS went from 2Mbps up to 9Mbps and iops went from 30 up to 900 at the peak. I'm sure there is an explanation for this - but there are some subtle configuration changes which had an impact and I don't understand exactly why. First, the main change was to set "$ActionFileEnableSync on" in each logging servers configuration. From what I can tell this globally enables the ability to turn on file sync and it defaults to "off". This change, combined with removing the "-" from all the actions in my configuration should have resulted in all files being synced all the time - probably the worst case scenario and I'm not all that suprised this caused problems. Second, before rolling the change back 100% - I removed the "$ActionFileEnableSync on" configuration from the servers but left the action configurations without the "-" - assuming that the global configuration option would completely disable file sync and using a "-" would have no effect. This did not seem to be the case - it did dramatically decrease the volume of traffic to the NAS but still the volume was about double (5Mbps) the normal volume. Third, I added "-" before all the action lines in the configuration and this brought volume back down to the 2-3Mbps we are used to. This leaves me with a few questions: 1) I would expect file sync to increase transactions with the NAS but not increase the volume of data being written to the NAS as measured at the disk. Can someone shed light on why this would so profoundly impact write volume? Load on the rsyslog servers also went down substantially after making this change - presumably because there were fewer queued transactions - not sure. 2) Is there some difference between what "$ActionFileEnableSync on" does vs. adding a "-" before an action other than one is global and one is per-action? I thought these were just two different levels of granularity for configuring the same thing but that doesn't appear to be the case. 3) Is it expected to have rsyslog take up to 30 seconds to flush messages to a logfile? These are BUSY logs - being written to multiple times per second, so it's not a delay on the client. I can observe these same clients logging messages to another logging server (the old server) running syslog-ng and messages are observable in near real-time. Ultimate - #3 is the problem I am trying to solve, but I'm very curious why these changes had the impact they did and why globally disabling file sync using "$ActionFileEnableSync" would not entire decrease log volume until I add "-" to all the actions. Let me know if I can add info that would help shed light on this. Thanks, Aaron From rgerhards at hq.adiscon.com Tue Oct 20 18:32:30 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 18:32:30 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase inwrite volume (bytes/sec) to NAS - can someone help shed light? References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> Hi Aaron, first and very important question: which version of rsyslog are you using? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Tuesday, October 20, 2009 5:32 PM > To: rsyslog-users > Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive > increase inwrite volume (bytes/sec) to NAS - can someone help shed > light? > > Hello, > We have two rsyslog "writer" instances which are receiving syslog > messages and writing them to an NFS mount on a NAS. The NAS is > connected via > 10gb ethernet but has a number of other applications writing to it as > well. > Typical write volume for our log writers is 2-3Mbps to the NAS. > Yesterday I > attempted to resolve some complaints with logs being "bursty" - events > appear to be written in batches which sounded consistent with buffering > on > the rsyslog daemon or not syncing the file with every message. These > bursts > could come as infrequently as every 30 seconds however, which seems > excessive. I took a number of steps to remedy this but ultimately had > to > reverse the changes because write volume (measured at the disks) to the > NAS > went from 2Mbps up to 9Mbps and iops went from 30 up to 900 at the > peak. I'm > sure there is an explanation for this - but there are some subtle > configuration changes which had an impact and I don't understand > exactly > why. > > First, the main change was to set "$ActionFileEnableSync on" in each > logging > servers configuration. From what I can tell this globally enables the > ability to turn on file sync and it defaults to "off". This change, > combined > with removing the "-" from all the actions in my configuration should > have > resulted in all files being synced all the time - probably the worst > case > scenario and I'm not all that suprised this caused problems. > > Second, before rolling the change back 100% - I removed the > "$ActionFileEnableSync on" configuration from the servers but left the > action configurations without the "-" - assuming that the global > configuration option would completely disable file sync and using a "-" > would have no effect. This did not seem to be the case - it did > dramatically > decrease the volume of traffic to the NAS but still the volume was > about > double (5Mbps) the normal volume. > > Third, I added "-" before all the action lines in the configuration and > this > brought volume back down to the 2-3Mbps we are used to. > > This leaves me with a few questions: > > 1) I would expect file sync to increase transactions with the NAS but > not > increase the volume of data being written to the NAS as measured at the > disk. Can someone shed light on why this would so profoundly impact > write > volume? Load on the rsyslog servers also went down substantially after > making this change - presumably because there were fewer queued > transactions > - not sure. > > 2) Is there some difference between what "$ActionFileEnableSync on" > does vs. > adding a "-" before an action other than one is global and one is > per-action? I thought these were just two different levels of > granularity > for configuring the same thing but that doesn't appear to be the case. > > 3) Is it expected to have rsyslog take up to 30 seconds to flush > messages to > a logfile? These are BUSY logs - being written to multiple times per > second, > so it's not a delay on the client. I can observe these same clients > logging > messages to another logging server (the old server) running syslog-ng > and > messages are observable in near real-time. > > Ultimate - #3 is the problem I am trying to solve, but I'm very curious > why > these changes had the impact they did and why globally disabling file > sync > using "$ActionFileEnableSync" would not entire decrease log volume > until I > add "-" to all the actions. > > Let me know if I can add info that would help shed light on this. > > Thanks, > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Oct 20 18:38:16 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 18:38:16 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com><200910201448.16203.marc.schiffbauer@mightycare.de><9B6E2A8877C38245BFB15CC491A11DA710325D@GRFEXC.intern.adiscon.com> <200910201639.43143.marc.schiffbauer@mightycare.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> thanks! Interesting, I see that only one of the messages that should be in the .qi are actually present. I wonder if this is related to the fact that ompgsql emits an error message itself while being down (the others don't do this AFIK). Will try to dig down to this (but I have to do so as time permits). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marc Schiffbauer > Sent: Tuesday, October 20, 2009 4:40 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.5.x queue file cleanup? > > > Hi Rainer, > > new mail on gmail, now with attachment ;) > > TIA > -Marc > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From anichols at trumped.org Tue Oct 20 18:54:02 2009 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 20 Oct 2009 10:54:02 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase inwrite volume (bytes/sec) to NAS - can someone help shed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> Message-ID: On Tue, Oct 20, 2009 at 10:32 AM, Rainer Gerhards wrote: > Hi Aaron, > > first and very important question: which version of rsyslog are you using? > > Of course - I forgot to include that, sorry - 4.4.1. Running on RHEL 5.3. From rgerhards at hq.adiscon.com Tue Oct 20 18:58:19 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 20 Oct 2009 18:58:19 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincrease inwrite volume (bytes/sec) to NAS - can someone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Tuesday, October 20, 2009 6:54 PM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling $ActionFileEnableSync caused > massiveincrease inwrite volume (bytes/sec) to NAS - can someone > helpshed light? > > On Tue, Oct 20, 2009 at 10:32 AM, Rainer Gerhards > wrote: > > > Hi Aaron, > > > > first and very important question: which version of rsyslog are you > using? > > > > > Of course - I forgot to include that, sorry - 4.4.1. Running on RHEL > 5.3. Hmmm... OK, so this is a version without output batching support (v5+) and without background writer (current v4-beta). So I do not really see how rsyslog could request this writing (other, of course, than by using the sync calls). Could you create a debug log? I'd try to dig through it. But I'd actually think that it has something to do with the way NFS parameters are set. Rainer From anichols at trumped.org Tue Oct 20 20:41:44 2009 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 20 Oct 2009 12:41:44 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincrease inwrite volume (bytes/sec) to NAS - can someone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> Message-ID: On Tue, Oct 20, 2009 at 10:58 AM, Rainer Gerhards wrote: > Hmmm... OK, so this is a version without output batching support (v5+) and > without background writer (current v4-beta). So I do not really see how > rsyslog could request this writing (other, of course, than by using the > sync > calls). Could you create a debug log? I'd try to dig through it. But I'd > actually think that it has something to do with the way NFS parameters are > set. > Here are the NFS mount options for the nodes writing to the NFS volume. The NFS server is an EMC Celerra - I can get any specific parameters there are questions about but can't easily provide a dump of the configured options: hostname:/log/syslog /log/syslog nfs rw,vers=3,rsize=32768,wsize=32768,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=hostname 0 0 I will work on a debug log but as these are production systems it'll take a little time. Thanks, Aaron From anichols at trumped.org Tue Oct 20 22:37:43 2009 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 20 Oct 2009 14:37:43 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincrease inwrite volume (bytes/sec) to NAS - can someone helpshed light? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> Message-ID: Also - can someone just confirm that my assumption is correct about the expected behavior? Should the "$ActionFileEnableSync [on/off]" option enable the ABILITY to turn on file sync per action using "-" or does it do something different? If I have "$ActionFileEnableSync off" set - should I expect adding or removing "-" before an action to have any impact at all? The documentation just associates these two options with enabling/disabling "file sync" but it seems like there are some additional subtleties that aren't clear. Thanks, Aaron On Tue, Oct 20, 2009 at 12:41 PM, Aaron Nichols wrote: > > > On Tue, Oct 20, 2009 at 10:58 AM, Rainer Gerhards < > rgerhards at hq.adiscon.com> wrote: > >> Hmmm... OK, so this is a version without output batching support (v5+) >> and >> without background writer (current v4-beta). So I do not really see how >> rsyslog could request this writing (other, of course, than by using the >> sync >> calls). Could you create a debug log? I'd try to dig through it. But I'd >> actually think that it has something to do with the way NFS parameters are >> set. >> > > From david at lang.hm Wed Oct 21 07:12:39 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 20 Oct 2009 22:12:39 -0700 (PDT) Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase in write volume (bytes/sec) to NAS - can someone help shed light? In-Reply-To: References: Message-ID: hmm, how large are the log entries? I'm wondering if the fact that every log entry then updates the metadata (size, time modified, etc) could account for most, if not all, of your extra traffic. the other thing that you could be seeing is if the NFS server records write traffic based on # of blocks written * block size. when doing a sync after each write you will do a lot of blocks (but re-write the same block many times), if it counts each of these writes as full block that would greatly magnify the write totals. I think you will really like the message batching in v5. you configure it to output every X messages or Y ms, whichever comes first. doing a sync after every write will have _far_ less effect. David Lang On Tue, 20 Oct 2009, Aaron Nichols wrote: > Date: Tue, 20 Oct 2009 09:32:07 -0600 > From: Aaron Nichols > Reply-To: rsyslog-users > To: rsyslog-users > Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase in > write volume (bytes/sec) to NAS - can someone help shed light? > > Hello, > We have two rsyslog "writer" instances which are receiving syslog > messages and writing them to an NFS mount on a NAS. The NAS is connected via > 10gb ethernet but has a number of other applications writing to it as well. > Typical write volume for our log writers is 2-3Mbps to the NAS. Yesterday I > attempted to resolve some complaints with logs being "bursty" - events > appear to be written in batches which sounded consistent with buffering on > the rsyslog daemon or not syncing the file with every message. These bursts > could come as infrequently as every 30 seconds however, which seems > excessive. I took a number of steps to remedy this but ultimately had to > reverse the changes because write volume (measured at the disks) to the NAS > went from 2Mbps up to 9Mbps and iops went from 30 up to 900 at the peak. I'm > sure there is an explanation for this - but there are some subtle > configuration changes which had an impact and I don't understand exactly > why. > > First, the main change was to set "$ActionFileEnableSync on" in each logging > servers configuration. From what I can tell this globally enables the > ability to turn on file sync and it defaults to "off". This change, combined > with removing the "-" from all the actions in my configuration should have > resulted in all files being synced all the time - probably the worst case > scenario and I'm not all that suprised this caused problems. > > Second, before rolling the change back 100% - I removed the > "$ActionFileEnableSync on" configuration from the servers but left the > action configurations without the "-" - assuming that the global > configuration option would completely disable file sync and using a "-" > would have no effect. This did not seem to be the case - it did dramatically > decrease the volume of traffic to the NAS but still the volume was about > double (5Mbps) the normal volume. > > Third, I added "-" before all the action lines in the configuration and this > brought volume back down to the 2-3Mbps we are used to. > > This leaves me with a few questions: > > 1) I would expect file sync to increase transactions with the NAS but not > increase the volume of data being written to the NAS as measured at the > disk. Can someone shed light on why this would so profoundly impact write > volume? Load on the rsyslog servers also went down substantially after > making this change - presumably because there were fewer queued transactions > - not sure. > > 2) Is there some difference between what "$ActionFileEnableSync on" does vs. > adding a "-" before an action other than one is global and one is > per-action? I thought these were just two different levels of granularity > for configuring the same thing but that doesn't appear to be the case. > > 3) Is it expected to have rsyslog take up to 30 seconds to flush messages to > a logfile? These are BUSY logs - being written to multiple times per second, > so it's not a delay on the client. I can observe these same clients logging > messages to another logging server (the old server) running syslog-ng and > messages are observable in near real-time. > > Ultimate - #3 is the problem I am trying to solve, but I'm very curious why > these changes had the impact they did and why globally disabling file sync > using "$ActionFileEnableSync" would not entire decrease log volume until I > add "-" to all the actions. > > Let me know if I can add info that would help shed light on this. > > Thanks, > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From anichols at trumped.org Wed Oct 21 07:28:48 2009 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 20 Oct 2009 23:28:48 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase in write volume (bytes/sec) to NAS - can someone help shed light? In-Reply-To: References: Message-ID: On Tue, Oct 20, 2009 at 11:12 PM, wrote: > hmm, how large are the log entries? I'm wondering if the fact that every > log entry then updates the metadata (size, time modified, etc) could > account for most, if not all, of your extra traffic. > Most log entries are small (under 2k) but there is one log destination which produces very large log entries. We have had to increase the max message size to 64k to accommodate these messages. This is only one destination but comes from multiple sources (collated into a single file) - so with a better understanding of how the "$ActionFileEnableSync" option vs. using "-" on a particular action works, I may be able to enable sync for log destinations other than that one high volume, large message size destination. > the other thing that you could be seeing is if the NFS server records > write traffic based on # of blocks written * block size. > > when doing a sync after each write you will do a lot of blocks (but > re-write the same block many times), if it counts each of these writes as > full block that would greatly magnify the write totals. > > I think you will really like the message batching in v5. you configure it > to output every X messages or Y ms, whichever comes first. doing a sync > after every write will have _far_ less effect. > > I probably will enjoy v5 but am not yet ready to run it in production. I need to find a good balance between log update frequency and keeping NAS performance sane. I did pass this feedback along to our storage guys to see if there is any tuning we can do around this. I will also be gathering a debug log soon - awaiting approvals to run the test. I assume the debug output would show some indication of where this might be causing problems? Thanks again for the feedback. I'm pushing for a support contract as I get the feeling it'll come in handy as well and I want to support the project, but I appreciate the feedback in this forum, I know everyone is hard at work. Thanks, Aaron From rgerhards at hq.adiscon.com Wed Oct 21 08:59:48 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 21 Oct 2009 08:59:48 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massive increase in write volume (bytes/sec) to NAS - can someone help shed light? References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103267@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, October 21, 2009 7:13 AM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling $ActionFileEnableSync caused massive > increase in write volume (bytes/sec) to NAS - can someone help shed > light? > > hmm, how large are the log entries? I'm wondering if the fact that > every > log entry then updates the metadata (size, time modified, etc) could > account for most, if not all, of your extra traffic. rsyslog does an fdatasync() to avoid this problem. However, that API is not available on all platforms... Rainer From rgerhards at hq.adiscon.com Wed Oct 21 09:33:22 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 21 Oct 2009 09:33:22 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincrease in write volume (bytes/sec) to NAS - can someonehelp shed light? References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710326A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Wednesday, October 21, 2009 7:29 AM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling $ActionFileEnableSync caused > massiveincrease in write volume (bytes/sec) to NAS - can someonehelp > shed light? > > On Tue, Oct 20, 2009 at 11:12 PM, wrote: > > > hmm, how large are the log entries? I'm wondering if the fact that > every > > log entry then updates the metadata (size, time modified, etc) could > > account for most, if not all, of your extra traffic. > > > > Most log entries are small (under 2k) but there is one log destination > which > produces very large log entries. We have had to increase the max > message > size to 64k to accommodate these messages. This is only one destination > but > comes from multiple sources (collated into a single file) - so with a > better > understanding of how the "$ActionFileEnableSync" option vs. using "-" > on a > particular action works, I may be able to enable sync for log > destinations > other than that one high volume, large message size destination. Sorry, I forgot to answer that in my posting yesterday. $ActionFileEnableSync is position dependent. So I assume that it is giving right at the top of rsyslog.conf, before any action. If set to on, it *enables* the sync facility. So you can use the dash in front of the file to specify whether or not a sync should happen. If it is set to "off" (the default), the "dash-rule" is always ignored and a sync never happens. This was done because many default configs have the sync option set just out of habit and the performance toll, as you have seen, is immense. > > > > the other thing that you could be seeing is if the NFS server records > > write traffic based on # of blocks written * block size. > > > > when doing a sync after each write you will do a lot of blocks (but > > re-write the same block many times), if it counts each of these > writes as > > full block that would greatly magnify the write totals. > > > > I think you will really like the message batching in v5. you > configure it > > to output every X messages or Y ms, whichever comes first. doing a > sync > > after every write will have _far_ less effect. > > > > > I probably will enjoy v5 but am not yet ready to run it in production. > I > need to find a good balance between log update frequency and keeping > NAS > performance sane. I did pass this feedback along to our storage guys to > see > if there is any tuning we can do around this. > > I will also be gathering a debug log soon - awaiting approvals to run > the > test. I assume the debug output would show some indication of where > this > might be causing problems? My primary interest in the debug log is to see if something works other than I would expect it. If that is not the case, you need to dig down into what NFS does to cause this behavior. Doing an strace may also be a good idea, depending on the output of the debug log. The core problem from my POV is that in 4.4.1 there is no code that could create the behavior you describe (at least not to the best of my knowledge, and this is what the debug log helps verify ;)). Rainer From rgerhards at hq.adiscon.com Wed Oct 21 09:38:19 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 21 Oct 2009 09:38:19 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincreaseinwrite volume (bytes/sec) to NAS - can someone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Tuesday, October 20, 2009 10:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling $ActionFileEnableSync caused > massiveincreaseinwrite volume (bytes/sec) to NAS - can someone helpshed > light? > > Also - can someone just confirm that my assumption is correct about the > expected behavior? Should the "$ActionFileEnableSync [on/off]" option > enable > the ABILITY to turn on file sync per action using "-" or does it do > something different? See my other note, but there may be a subtle misunderstanding: The presence of "-" turns OFF file syncing. Absence of "-" causes a sync! So in *.* /path/to/fileWithSync *.* -/path/to/fileWithoutSync fileWithSync is being synced and fileWithoutSync is not! This is traditional config file syntax and probably the main reason why *a lot* of real-world config files do syncs... Rainer From tbergfeld at hq.adiscon.com Wed Oct 21 11:57:33 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 21 Oct 2009 11:57:33 +0200 Subject: [rsyslog] rsyslog 4.5.5 (v4-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103272@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.5.5, a member of the v4-beta branch. This release offers a new config directive $InputTCPServerNotifyOnConnectionClose. Also there are also some bug fixes. See Changelog for more details. This is a recommended update for all users of the devel branch. ChangeLog: http://www.rsyslog.com/Article416.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-182.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com _______________________________________________ From tbergfeld at hq.adiscon.com Wed Oct 21 11:56:58 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 21 Oct 2009 11:56:58 +0200 Subject: [rsyslog] rsyslog 5.3.2 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103271@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.3.2, a member of the devel branch. This release offers a lot of enhancement like the enhanced omfile to support transactional interface. Also there are also some bug fixes. See Changelog for more details. This is a recommended update for all users of the devel branch. ChangeLog: http://www.rsyslog.com/Article415.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-181.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com _______________________________________________ From anichols at trumped.org Wed Oct 21 15:17:49 2009 From: anichols at trumped.org (Aaron Nichols) Date: Wed, 21 Oct 2009 07:17:49 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync caused massiveincreaseinwrite volume (bytes/sec) to NAS - can someone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> Message-ID: On Wed, Oct 21, 2009 at 1:38 AM, Rainer Gerhards wrote: > See my other note, but there may be a subtle misunderstanding: The presence > of "-" turns OFF file syncing. Absence of "-" causes a sync! So in > > *.* /path/to/fileWithSync > *.* -/path/to/fileWithoutSync > > fileWithSync is being synced and fileWithoutSync is not! > > This is traditional config file syntax and probably the main reason why *a > lot* of real-world config files do syncs... > > I understood correctly and now I'm certain I know how it should work. I need to do some more testing and I think I have enough information to do that and will come back here when I have something interesting to share. An observation I did make when this was happening was that if $ActionFileEnableSync was disabled (commented out) but the "-" was missing before the actions in the configuration, write volume to the NAS was higher than when we had $ActionFileEnableSync disabled and had the "-" added before the action (write volume was 100% higher if $ActionFileEnableSync was OFF and the "-" option was missing from all log actions than if $ActionFileEnableSync was OFF and "-" was included in all log actions.) This led to my original suspicion that the two options somehow change sync behavior independently. If they shouldn't, then I need to get the debug log and do some analysis. I appreciate the guidance - I'll gather some more details. Thanks, Aaron From rgerhards at hq.adiscon.com Wed Oct 21 15:22:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 21 Oct 2009 15:22:29 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSync causedmassiveincreaseinwrite volume (bytes/sec) to NAS - cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Wednesday, October 21, 2009 3:18 PM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling $ActionFileEnableSync > causedmassiveincreaseinwrite volume (bytes/sec) to NAS - cansomeone > helpshed light? > > On Wed, Oct 21, 2009 at 1:38 AM, Rainer Gerhards > wrote: > > > See my other note, but there may be a subtle misunderstanding: The > presence > > of "-" turns OFF file syncing. Absence of "-" causes a sync! So in > > > > *.* /path/to/fileWithSync > > *.* -/path/to/fileWithoutSync > > > > fileWithSync is being synced and fileWithoutSync is not! > > > > This is traditional config file syntax and probably the main reason > why *a > > lot* of real-world config files do syncs... > > > > > I understood correctly and now I'm certain I know how it should work. I > need > to do some more testing and I think I have enough information to do > that and > will come back here when I have something interesting to share. An > observation I did make when this was happening was that if > $ActionFileEnableSync was disabled (commented out) but the "-" was > missing > before the actions in the configuration, write volume to the NAS was > higher > than when we had $ActionFileEnableSync disabled and had the "-" added > before > the action (write volume was 100% higher if $ActionFileEnableSync was > OFF > and the "-" option was missing from all log actions than if > $ActionFileEnableSync was OFF and "-" was included in all log actions.) That indeed sounds odd. Are you sure the $ActionFileEnableSync is in front of all write statements? On some systems, write statements may be included via files and if these are in front of the directive, those would be unaffected by it... Rainer From anichols at trumped.org Wed Oct 21 16:25:21 2009 From: anichols at trumped.org (Aaron Nichols) Date: Wed, 21 Oct 2009 08:25:21 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSync causedmassiveincreaseinwrite volume (bytes/sec) to NAS - cansomeone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> Message-ID: On Wed, Oct 21, 2009 at 7:22 AM, Rainer Gerhards wrote: > That indeed sounds odd. Are you sure the $ActionFileEnableSync is in front > of > all write statements? On some systems, write statements may be included via > files and if these are in front of the directive, those would be unaffected > by it... > We do also use includes but in this case $ActionFileEnableSync was at the top of the included file with all write statements below it. I will also note that the write volume changed significantly with two different config changes. To uber-simplify my original email: "normal" volume - $ActionFileEnableSync off & all actions include "-" in front : 2-3mbps ~30 iops $ActionFileEnableSync off & all actions exclude "-" in front : 5mbps (~100% increase) (iops unknown) $ActionFileEnableSync on & all actions exclude "-" in front : 9mbps (~300-400% increase) ~900 iops I am using a mix of static and dynamic filename actions and some actions use format templates as well. I suppose there could be one of those combinations which is not honoring the global $ActionFileEnableSync configuration. Again, I need to do some testing because at this point I'm just making guesses without much information. Thanks, Aaron From rgerhards at hq.adiscon.com Wed Oct 21 17:32:59 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 21 Oct 2009 17:32:59 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103262@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Wednesday, October 21, 2009 4:25 PM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling > $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to > NAS -cansomeone helpshed light? > > On Wed, Oct 21, 2009 at 7:22 AM, Rainer Gerhards > wrote: > > > That indeed sounds odd. Are you sure the $ActionFileEnableSync is in > front > > of > > all write statements? On some systems, write statements may be > included via > > files and if these are in front of the directive, those would be > unaffected > > by it... > > > > We do also use includes but in this case $ActionFileEnableSync was at > the > top of the included file with all write statements below it. I will > also > note that the write volume changed significantly with two different > config > changes. To uber-simplify my original email: > > "normal" volume - $ActionFileEnableSync off & all actions include "-" > in > front : 2-3mbps ~30 iops > $ActionFileEnableSync off & all actions exclude "-" in front : 5mbps > (~100% > increase) (iops unknown) > $ActionFileEnableSync on & all actions exclude "-" in front : 9mbps > (~300-400% increase) ~900 iops > > I am using a mix of static and dynamic filename actions and some > actions use > format templates as well. I suppose there could be one of those > combinations > which is not honoring the global $ActionFileEnableSync configuration. I may be overlooking something here, but the check is very straightforward and I don't see why some should be affected but others not. This is the file (and spot) in question: http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/omfile.c;h=bb12b4b61dacb 84167b9b0d17e49a597342dddc0;hb=07b076ddcfed4ea9b447d0be574d1dcdb799bc2f#l623 Do a search for "bEnableSync" to see all occurences. But the debug log will show... Rainer > Again, > I need to do some testing because at this point I'm just making guesses > without much information. > > Thanks, > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From wirelessdreamer at gmail.com Wed Oct 21 18:09:36 2009 From: wirelessdreamer at gmail.com (Nathanael Anderson) Date: Wed, 21 Oct 2009 11:09:36 -0500 Subject: [rsyslog] Regex matching problems Message-ID: I've been trying to get a basic regex to work in the regex tester at: http://www.rsyslog.com/user-regex.php \.[0-9]{1,3} I'm using this regex to match one octet in this message: Killing attempted connection: tcp (192.168.129.13:48351 - 192.168.145.91:35500) This should return 6 matches that can be accessed by changing the position fied from 0 to the field I want. in both rsyslog and the testing link, this is not the case. %msg:R,ERE,0,DFLT:\.[0-9]{1,3}--end% Is this a bug, or am I misunderstanding how this should behave? Nathanael From david at lang.hm Wed Oct 21 20:04:27 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 21 Oct 2009 11:04:27 -0700 (PDT) Subject: [rsyslog] makefile tweak Message-ID: when doing 'make clean' it should not need to run configure first. David Lang From mbiebl at gmail.com Wed Oct 21 20:11:49 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 21 Oct 2009 20:11:49 +0200 Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: 2009/10/21 : > when doing 'make clean' it should not need to run configure first. Ist that from a release tarball or a git snapshot? Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From david at lang.hm Wed Oct 21 22:19:41 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 21 Oct 2009 13:19:41 -0700 (PDT) Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: On Wed, 21 Oct 2009, Michael Biebl wrote: > 2009/10/21 : >> when doing 'make clean' it should not need to run configure first. > > Ist that from a release tarball or a git snapshot? git snapshot, especially after doing the autoconf -fvi. the system spends several min doing lots of checks before it can remove the files, then I do a configure with the options that I need and it goes through and does all those checks again. David Lang From mbiebl at gmail.com Wed Oct 21 22:26:09 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 21 Oct 2009 22:26:09 +0200 Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: 2009/10/21 : > On Wed, 21 Oct 2009, Michael Biebl wrote: > >> 2009/10/21 ?: >>> when doing 'make clean' it should not need to run configure first. >> >> Ist that from a release tarball or a git snapshot? > > git snapshot, especially after doing the autoconf -fvi. the system spends That's the wrong command. You should use autoreconf -vfi in a fresh git checkout. After that ./configure && make clean works as expected for me. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From david at lang.hm Wed Oct 21 22:30:22 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 21 Oct 2009 13:30:22 -0700 (PDT) Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: On Wed, 21 Oct 2009, Michael Biebl wrote: > Date: Wed, 21 Oct 2009 22:26:09 +0200 > From: Michael Biebl > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] makefile tweak > > 2009/10/21 : >> On Wed, 21 Oct 2009, Michael Biebl wrote: >> >>> 2009/10/21 ?: >>>> when doing 'make clean' it should not need to run configure first. >>> >>> Ist that from a release tarball or a git snapshot? >> >> git snapshot, especially after doing the autoconf -fvi. the system spends > > That's the wrong command. > > You should use autoreconf -vfi in a fresh git checkout. sorry, I typed the wrong thing in the e-mail > After that ./configure && make clean works as expected for me. I do autoreconf -fvi make clean ./configure --enable-imfile make the make clean forces a configure David Lang From mbiebl at gmail.com Wed Oct 21 22:39:36 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 21 Oct 2009 22:39:36 +0200 Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: 2009/10/21 : > On Wed, 21 Oct 2009, Michael Biebl wrote: > >> Date: Wed, 21 Oct 2009 22:26:09 +0200 >> From: Michael Biebl >> Reply-To: rsyslog-users >> To: rsyslog-users >> Subject: Re: [rsyslog] makefile tweak >> >> 2009/10/21 ?: >>> >>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>> >>>> 2009/10/21 ?: >>>>> >>>>> when doing 'make clean' it should not need to run configure first. >>>> >>>> Ist that from a release tarball or a git snapshot? >>> >>> git snapshot, especially after doing the autoconf -fvi. the system spends >> >> That's the wrong command. >> >> You should use autoreconf -vfi in a fresh git checkout. > > sorry, I typed the wrong thing in the e-mail > >> After that ./configure && make clean works as expected for me. > > I do > autoreconf -fvi > make clean How do you expect "make" to work without running ./configure before? If you have a Makefile's from a previous ./configure run, it's absolutely correct behaviour that after a autoreconf -vfi, a configure run is first done before you can run make. To sum up, everything works as it should. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From david at lang.hm Wed Oct 21 22:52:01 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 21 Oct 2009 13:52:01 -0700 (PDT) Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: On Wed, 21 Oct 2009, Michael Biebl wrote: > 2009/10/21 : >> On Wed, 21 Oct 2009, Michael Biebl wrote: >> >>> 2009/10/21 ?: >>>> >>>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>>> >>>>> 2009/10/21 ?: >>>>>> >>>>>> when doing 'make clean' it should not need to run configure first. >>>>> >>>>> Ist that from a release tarball or a git snapshot? >>>> >>>> git snapshot, especially after doing the autoconf -fvi. the system spends >>> >>> That's the wrong command. >>> >>> You should use autoreconf -vfi in a fresh git checkout. >> >> sorry, I typed the wrong thing in the e-mail >> >>> After that ./configure && make clean works as expected for me. >> >> I do >> autoreconf -fvi >> make clean > > How do you expect "make" to work without running ./configure before? > If you have a Makefile's from a previous ./configure run, it's > absolutely correct behaviour that after a autoreconf -vfi, a configure > run is first done before you can run make. > > To sum up, everything works as it should. I would not expect to be able to do a normal make, but since a make clean just deletes files (and the same files, not matter what the config options are) I would expect it to work. David Lang From mbiebl at gmail.com Wed Oct 21 23:02:17 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 21 Oct 2009 23:02:17 +0200 Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: 2009/10/21 : > On Wed, 21 Oct 2009, Michael Biebl wrote: > >> 2009/10/21 ?: >>> >>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>> >>>> 2009/10/21 ?: >>>>> >>>>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>>>> >>>>>> 2009/10/21 ?: >>>>>>> >>>>>>> when doing 'make clean' it should not need to run configure first. >>>>>> >>>>>> Ist that from a release tarball or a git snapshot? >>>>> >>>>> git snapshot, especially after doing the autoconf -fvi. the system >>>>> spends >>>> >>>> That's the wrong command. >>>> >>>> You should use autoreconf -vfi in a fresh git checkout. >>> >>> sorry, I typed the wrong thing in the e-mail >>> >>>> After that ./configure && make clean works as expected for me. >>> >>> I do >>> autoreconf -fvi >>> make clean >> >> How do you expect "make" to work without running ./configure before? >> If you have a Makefile's from a previous ./configure run, it's >> absolutely correct behaviour that after a autoreconf -vfi, a configure >> run is first done before you can run make. >> >> To sum up, everything works as it should. > > I would not expect to be able to do a normal make, but since a make clean > just deletes files (and the same files, not matter what the config options > are) I would expect it to work. Well, that's how automake/autoconf works. If the timestamps of build input files change, the build system is automatically re-generated. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Wed Oct 21 23:09:35 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 21 Oct 2009 23:09:35 +0200 Subject: [rsyslog] makefile tweak In-Reply-To: References: Message-ID: 2009/10/21 Michael Biebl : > 2009/10/21 ?: >> On Wed, 21 Oct 2009, Michael Biebl wrote: >> >>> 2009/10/21 ?: >>>> >>>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>>> >>>>> 2009/10/21 ?: >>>>>> >>>>>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>>>>> >>>>>>> 2009/10/21 ?: >>>>>>>> >>>>>>>> when doing 'make clean' it should not need to run configure first. >>>>>>> >>>>>>> Ist that from a release tarball or a git snapshot? >>>>>> >>>>>> git snapshot, especially after doing the autoconf -fvi. the system >>>>>> spends >>>>> >>>>> That's the wrong command. >>>>> >>>>> You should use autoreconf -vfi in a fresh git checkout. >>>> >>>> sorry, I typed the wrong thing in the e-mail >>>> >>>>> After that ./configure && make clean works as expected for me. >>>> >>>> I do >>>> autoreconf -fvi >>>> make clean >>> >>> How do you expect "make" to work without running ./configure before? >>> If you have a Makefile's from a previous ./configure run, it's >>> absolutely correct behaviour that after a autoreconf -vfi, a configure >>> run is first done before you can run make. >>> >>> To sum up, everything works as it should. >> >> I would not expect to be able to do a normal make, but since a make clean >> just deletes files (and the same files, not matter what the config options >> are) I would expect it to work. > > Well, that's how automake/autoconf works. > If the timestamps of build input files change, the build system is > automatically re-generated. Btw, since you run autoreconf anyway there is a much nicer (and quicker way to get rid of the old build files): git clean -x -d -f autoreconf -vfi Will give you a clean working state ready to run ./configure && make -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Oct 22 06:40:52 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 22 Oct 2009 06:40:52 +0200 Subject: [rsyslog] makefile tweak Message-ID: <000101ca52d1$dea24313$100013ac@intern.adiscon.com> The point is that configure decides which files are to be created at all. So without a configure run, make does not even know what to delete. rainer ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 21.10.09 22:52 Betreff: Re: [rsyslog] makefile tweak On Wed, 21 Oct 2009, Michael Biebl wrote: > 2009/10/21 : >> On Wed, 21 Oct 2009, Michael Biebl wrote: >> >>> 2009/10/21 : >>>> >>>> On Wed, 21 Oct 2009, Michael Biebl wrote: >>>> >>>>> 2009/10/21 : >>>>>> >>>>>> when doing 'make clean' it should not need to run configure first. >>>>> >>>>> Ist that from a release tarball or a git snapshot? >>>> >>>> git snapshot, especially after doing the autoconf -fvi. the system spends >>> >>> That's the wrong command. >>> >>> You should use autoreconf -vfi in a fresh git checkout. >> >> sorry, I typed the wrong thing in the e-mail >> >>> After that ./configure && make clean works as expected for me. >> >> I do >> autoreconf -fvi >> make clean > > How do you expect "make" to work without running ./configure before? > If you have a Makefile's from a previous ./configure run, it's > absolutely correct behaviour that after a autoreconf -vfi, a configure > run is first done before you can run make. > > To sum up, everything works as it should. I would not expect to be able to do a normal make, but since a make clean just deletes files (and the same files, not matter what the config options are) I would expect it to work. David Lang From marc.schiffbauer at mightycare.de Thu Oct 22 15:06:29 2009 From: marc.schiffbauer at mightycare.de (Marc Schiffbauer) Date: Thu, 22 Oct 2009 15:06:29 +0200 Subject: [rsyslog] rsyslog 4.5.x queue file cleanup? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> References: <000001ca5179$a3469ab3$100013ac@intern.adiscon.com> <200910201639.43143.marc.schiffbauer@mightycare.de> <9B6E2A8877C38245BFB15CC491A11DA7103264@GRFEXC.intern.adiscon.com> Message-ID: <200910221506.29850.marc.schiffbauer@mightycare.de> Am Dienstag, 20. Oktober 2009 18:38:16 schrieb Rainer Gerhards: > thanks! > > Interesting, I see that only one of the messages that should be in the .qi > are actually present. I wonder if this is related to the fact that ompgsql > emits an error message itself while being down (the others don't do this > AFIK). Will try to dig down to this (but I have to do so as time permits). > Thanks, please tell me if I can assist in any way. -Marc From david at lang.hm Thu Oct 22 21:29:48 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 22 Oct 2009 12:29:48 -0700 (PDT) Subject: [rsyslog] version numbers from git repository Message-ID: I've been doing a lot of testing and can have a dozen different copies of rsyslog that all claim to be the same version number. the ability of git to give more precise versions (as described in the message from the git mailing list below) would be handy in eliminating confusion. the nice thing is that if a person is using a tagged version, this just reports the tag with nothing extra, so changing the makefile to use this would not affect most people. David Lang ---------- Forwarded message ---------- Date: Thu, 22 Oct 2009 10:01:25 +0200 From: Martin Langhoff To: Robin Rosenberg Cc: newsgroups at catchall.shelter13.net, git at vger.kernel.org Subject: Re: Deciding between Git/Mercurial On Sun, Sep 27, 2009 at 8:01 PM, Robin Rosenberg wrote: > s?ndag 27 september 2009 14:24:32 skrev Anteru : >> Mercurial's revision number system: With git, I get an SHA1 hash for >> every commit, but it's not possible to see whether Hash1 is newer than >> Hash2, while Mecurial also adds a running number to each commit. What's > > But those numbers cannot be communicated since they are local to your > clone. You can use git-describe, which will look for the latest tag, and make a combo of latest tag, commits since the tag, short form of sha1. So you get "v1.6.3-33-g1234" - 33 commits after 1.6.3. Works very well to integrate in versioning. The git project itself uses it in the Makefile to set the versions, same as the kernel folk do -- I use it to version even RPM/DEBs. hth, m -- martin.langhoff at gmail.com martin at laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majordomo at vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html From rgerhards at hq.adiscon.com Thu Oct 22 21:32:58 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 22 Oct 2009 21:32:58 +0200 Subject: [rsyslog] version numbers from git repository References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> David, I like this idea, but I have to admit I have not even the slightest clue on how I should implement it... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, October 22, 2009 9:30 PM > To: rsyslog-users > Subject: [rsyslog] version numbers from git repository > > I've been doing a lot of testing and can have a dozen > different copies of > rsyslog that all claim to be the same version number. > > the ability of git to give more precise versions (as described in the > message from the git mailing list below) would be handy in > eliminating > confusion. > > the nice thing is that if a person is using a tagged version, > this just > reports the tag with nothing extra, so changing the makefile > to use this > would not affect most people. > > David Lang > > ---------- Forwarded message ---------- > Date: Thu, 22 Oct 2009 10:01:25 +0200 > From: Martin Langhoff > To: Robin Rosenberg > Cc: newsgroups at catchall.shelter13.net, git at vger.kernel.org > Subject: Re: Deciding between Git/Mercurial > > On Sun, Sep 27, 2009 at 8:01 PM, Robin Rosenberg > wrote: > > s?ndag 27 september 2009 14:24:32 skrev Anteru > : > >> Mercurial's revision number system: With git, I get an > SHA1 hash for > >> every commit, but it's not possible to see whether Hash1 > is newer than > >> Hash2, while Mecurial also adds a running number to each > commit. What's > > > > But those numbers cannot be communicated since they are > local to your > > clone. > > You can use git-describe, which will look for the latest tag, and make > a combo of latest tag, commits since the tag, short form of sha1. So > you get "v1.6.3-33-g1234" - 33 commits after 1.6.3. Works very well to > integrate in versioning. The git project itself uses it in the > Makefile to set the versions, same as the kernel folk do -- I use it > to version even RPM/DEBs. > > hth, > > > > m > -- > martin.langhoff at gmail.com > martin at laptop.org -- School Server Architect > - ask interesting questions > - don't get distracted with shiny stuff - working code first > - http://wiki.laptop.org/go/User:Martinlanghoff > -- > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo at vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Thu Oct 22 21:36:49 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 22 Oct 2009 12:36:49 -0700 (PDT) Subject: [rsyslog] version numbers from git repository In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 22 Oct 2009, Rainer Gerhards wrote: > David, > > I like this idea, but I have to admit I have not even the slightest clue on > how I should implement it... I don't know either, but I do know a couple projects that do implement it. git implements it the linux kernel implements it if LOCAL_VERSION_AUTO is enabled in the config if I get a chance I'll try to dig into their makefiles to find out how. David Lang P.S. I don't know what rsyslog does with the version number, but when I use checkinstall to create a package from the source it ends up detecting the version number many times instead of once so instead of version='5.3.2' I get version='5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2 5.3.2' > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Thursday, October 22, 2009 9:30 PM >> To: rsyslog-users >> Subject: [rsyslog] version numbers from git repository >> >> I've been doing a lot of testing and can have a dozen >> different copies of >> rsyslog that all claim to be the same version number. >> >> the ability of git to give more precise versions (as described in the >> message from the git mailing list below) would be handy in >> eliminating >> confusion. >> >> the nice thing is that if a person is using a tagged version, >> this just >> reports the tag with nothing extra, so changing the makefile >> to use this >> would not affect most people. >> >> David Lang >> >> ---------- Forwarded message ---------- >> Date: Thu, 22 Oct 2009 10:01:25 +0200 >> From: Martin Langhoff >> To: Robin Rosenberg >> Cc: newsgroups at catchall.shelter13.net, git at vger.kernel.org >> Subject: Re: Deciding between Git/Mercurial >> >> On Sun, Sep 27, 2009 at 8:01 PM, Robin Rosenberg >> wrote: >>> s?ndag 27 september 2009 14:24:32 skrev Anteru >> : >>>> Mercurial's revision number system: With git, I get an >> SHA1 hash for >>>> every commit, but it's not possible to see whether Hash1 >> is newer than >>>> Hash2, while Mecurial also adds a running number to each >> commit. What's >>> >>> But those numbers cannot be communicated since they are >> local to your >>> clone. >> >> You can use git-describe, which will look for the latest tag, and make >> a combo of latest tag, commits since the tag, short form of sha1. So >> you get "v1.6.3-33-g1234" - 33 commits after 1.6.3. Works very well to >> integrate in versioning. The git project itself uses it in the >> Makefile to set the versions, same as the kernel folk do -- I use it >> to version even RPM/DEBs. >> >> hth, >> >> >> >> m >> -- >> martin.langhoff at gmail.com >> martin at laptop.org -- School Server Architect >> - ask interesting questions >> - don't get distracted with shiny stuff - working code first >> - http://wiki.laptop.org/go/User:Martinlanghoff >> -- >> To unsubscribe from this list: send the line "unsubscribe git" in >> the body of a message to majordomo at vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Oct 23 07:54:28 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 07:54:28 +0200 Subject: [rsyslog] version numbers from git repository References: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103292@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, October 22, 2009 9:37 PM > To: rsyslog-users > Subject: Re: [rsyslog] version numbers from git repository > > On Thu, 22 Oct 2009, Rainer Gerhards wrote: > > > David, > > > > I like this idea, but I have to admit I have not even the slightest > clue on > > how I should implement it... > > I don't know either, but I do know a couple projects that do implement > it. > > git implements it > > the linux kernel implements it if LOCAL_VERSION_AUTO is enabled in the > config > > if I get a chance I'll try to dig into their makefiles to find out how. I've looked a bit into the issue. Neither of them use the GNU build system. In essence, it boils down to running "git describe" as part of the build process, generating a file with it. However, I do not know how I may achieve that. It should probably be run during the ./configure stage. Maybe someone can provide some advise ;) > > David Lang > > P.S. I don't know what rsyslog does with the version number, but when I > use checkinstall to create a package from the source it ends up > detecting > the version number many times instead of once > > so instead of > version='5.3.2' > I get > version='5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2 > 5.3.2' I have no real idea what may cause that other than the fact that the version number is probably present more than once in the constant pool because of rsyslog's modular strucutre. Rainer From rgerhards at hq.adiscon.com Fri Oct 23 08:21:01 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 08:21:01 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> Aaron, I am replying on-list so that others can follow (and I hope they have some advise). I have stripped the sensitive part of your message. In short for all others: I got a debug log and have now reviewed it. > During this debug run we saw NAS write volume increase from a "normal" > 2-3mbps and 35 iops up to 500 iops and 5.6mbps toward the end of the > test. I do understand how enabling file sync will increase iops but > I'm unclear on why the volume of data being written would increase so > much. If rsyslog has file sync "disabled" then I would expect bursts > of data and if file sync is "enabled" I would expect the same amount > of data to be written in much more frequent, but also much smaller > chunks. NFS could be a culprit here, but I'm curious what NFS > parameters would impact this behavior. The debug log does not show anything that looks specifically suspect. The incoming traffic is intense and the default queue size is overrun quickly, resulting in voluntary delays of the sender (via tcp). Looking at the traffic, I think this is perfectly legal and increasing the queue size would not bring any benefit (I think the end result would just be a larger memory footprint without any advantage). I do not see any unusual write behavior. Of course, the debug log could be more verbose as it does not specifically tell about the syncs (usually, the log is missing some information you'd like to see for a given instant, but you can't include everything...). But looking at the code base plus the log really does not reveal anything suspicious. One thing you might want to try is doing an strace so that we can look at it and see which exact OS calls are made and how long they last. > For the record, changing the sync behavior of rsyslog DOES fix the > original complaint. When we have file sync disabled we see up to 30 > second pauses between writes to files which get multiple messages per > second delivered to the rsyslog host. There is nothing in the log that points to rsyslog doing these delays. So my conclusion really is that you need to look at the NFS layer. To me, it looks like NFS does some local caching, and maybe 30 seconds is the timeout at which NFS writes data from files that are constantly being written in short intervals. With sync, NFS probably writes and rewrites each block as David has said, thus resulting in the far larger i/o footprint. But I am far from being a NFS expert, so it would probably make sense to ask one ;) > I have looked at the option > $OMFileFlushOnTXEnd and wonder if it would be a preferred method to > resolve this in the 4.1.1 version (or if it even works there). This makes no sense for v4 (I think it is not even available). Sorry I have no better answer. If get a strace, I'd gladly look at it. Rainer From david at lang.hm Fri Oct 23 08:32:05 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 22 Oct 2009 23:32:05 -0700 (PDT) Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 23 Oct 2009, Rainer Gerhards wrote: >> For the record, changing the sync behavior of rsyslog DOES fix the >> original complaint. When we have file sync disabled we see up to 30 >> second pauses between writes to files which get multiple messages per >> second delivered to the rsyslog host. > > There is nothing in the log that points to rsyslog doing these delays. > > So my conclusion really is that you need to look at the NFS layer. To me, it > looks like NFS does some local caching, and maybe 30 seconds is the timeout > at which NFS writes data from files that are constantly being written in > short intervals. With sync, NFS probably writes and rewrites each block as > David has said, thus resulting in the far larger i/o footprint. But I am far > from being a NFS expert, so it would probably make sense to ask one ;) this reminded me of something. a standard complient NFS implementation would require that any write to a NFS volume go all the way to the destination and be safe on disk before the write call returns to the calling program. this doesn't mean when you do a fsync, it means each individual write call. for the program to think the data has been written and it not be available on the NFS drive to other systems there has to be some non-standard caching in place here. note that unless you have a high-end NFS server that includes some battery-backed ram on it, standard complient behavior also means a very low write transaction rate. note that the caching could be on the reader machines instead of on the writer, and it could be something like caching of the metadata (including size and last modified timestamp) David Lang From aland at freeradius.org Fri Oct 23 08:39:35 2009 From: aland at freeradius.org (Alan T DeKok) Date: Fri, 23 Oct 2009 08:39:35 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> Message-ID: <4AE14FA7.4010008@freeradius.org> Rainer Gerhards wrote: > So my conclusion really is that you need to look at the NFS layer. Logs should NEVER be written to an NFS mount. David gave a good explanation as to why. But the above prohibition should be made clear in the docs, if it isn't already. If the NFS server goes away, then the write transaction rate will go to zero. This likely isn't what people want from a syslog server. There is already a way to get syslog data from one system to another: the syslog protocol. Using NFS as a replacement for syslog is wrong on many levels. Alan DeKok. From rgerhards at hq.adiscon.com Fri Oct 23 09:03:05 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 09:03:05 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> <4AE14FA7.4010008@freeradius.org> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> I think the main point of the OP was that he logs to a NAS device, so I do not see way to write to it without going through a network share. Other than that, I agree to your statement. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Alan T DeKok > Sent: Friday, October 23, 2009 8:40 AM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling > $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to > NAS -cansomeone helpshed light? > > Rainer Gerhards wrote: > > So my conclusion really is that you need to look at the NFS layer. > > Logs should NEVER be written to an NFS mount. > > David gave a good explanation as to why. But the above prohibition > should be made clear in the docs, if it isn't already. > > If the NFS server goes away, then the write transaction rate will go > to zero. This likely isn't what people want from a syslog server. > > There is already a way to get syslog data from one system to another: > the syslog protocol. Using NFS as a replacement for syslog is wrong on > many levels. > > Alan DeKok. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Oct 23 09:17:43 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 23 Oct 2009 00:17:43 -0700 (PDT) Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> <4AE14FA7.4010008@freeradius.org> <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 23 Oct 2009, Rainer Gerhards wrote: > I think the main point of the OP was that he logs to a NAS device, so I do > not see way to write to it without going through a network share. ahh, have we been assuming that this is NFS when all he said was NAS? if so, please clarify what protocol you are using to talk to the NAS. it could make a huge difference here. David Lang > Other than that, I agree to your statement. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Alan T DeKok >> Sent: Friday, October 23, 2009 8:40 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Enabling >> $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to >> NAS -cansomeone helpshed light? >> >> Rainer Gerhards wrote: >>> So my conclusion really is that you need to look at the NFS layer. >> >> Logs should NEVER be written to an NFS mount. >> >> David gave a good explanation as to why. But the above prohibition >> should be made clear in the docs, if it isn't already. >> >> If the NFS server goes away, then the write transaction rate will go >> to zero. This likely isn't what people want from a syslog server. >> >> There is already a way to get syslog data from one system to another: >> the syslog protocol. Using NFS as a replacement for syslog is wrong on >> many levels. >> >> Alan DeKok. >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From aland at freeradius.org Fri Oct 23 09:42:22 2009 From: aland at freeradius.org (Alan T DeKok) Date: Fri, 23 Oct 2009 09:42:22 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> <4AE14FA7.4010008@freeradius.org> <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> Message-ID: <4AE15E5E.2020100@freeradius.org> Rainer Gerhards wrote: > I think the main point of the OP was that he logs to a NAS device, so I do > not see way to write to it without going through a network share. Configure rsyslog to write to local disk, and rsync periodically to the network share. Or, use another instance of rsyslog to copy from local disk to the network share. Any other method will have problems. Even NetApp systems have issues from time to time. Alan DeKok. From rgerhards at hq.adiscon.com Fri Oct 23 09:53:32 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 09:53:32 +0200 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com><4AE14FA7.4010008@freeradius.org><9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103298@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 23, 2009 9:18 AM > To: rsyslog-users > Subject: Re: [rsyslog] Enabling > $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to > NAS -cansomeone helpshed light? > > On Fri, 23 Oct 2009, Rainer Gerhards wrote: > > > I think the main point of the OP was that he logs to a NAS device, so > I do > > not see way to write to it without going through a network share. > > ahh, have we been assuming that this is NFS when all he said was NAS? No, he said NFS, at least in the last message (not sure about the beginning of this story...) Rainer > > if so, please clarify what protocol you are using to talk to the NAS. > it > could make a huge difference here. > > David Lang > > > Other than that, I agree to your statement. > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Alan T DeKok > >> Sent: Friday, October 23, 2009 8:40 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Enabling > >> $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) > to > >> NAS -cansomeone helpshed light? > >> > >> Rainer Gerhards wrote: > >>> So my conclusion really is that you need to look at the NFS layer. > >> > >> Logs should NEVER be written to an NFS mount. > >> > >> David gave a good explanation as to why. But the above > prohibition > >> should be made clear in the docs, if it isn't already. > >> > >> If the NFS server goes away, then the write transaction rate will > go > >> to zero. This likely isn't what people want from a syslog server. > >> > >> There is already a way to get syslog data from one system to > another: > >> the syslog protocol. Using NFS as a replacement for syslog is wrong > on > >> many levels. > >> > >> Alan DeKok. > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Oct 23 10:52:02 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 23 Oct 2009 01:52:02 -0700 (PDT) Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: <4AE15E5E.2020100@freeradius.org> References: <9B6E2A8877C38245BFB15CC491A11DA7103265@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710326B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> <4AE14FA7.4010008@freeradius.org> <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> <4AE15E5E.2020100@freeradius.org> Message-ID: On Fri, 23 Oct 2009, Alan T DeKok wrote: > Rainer Gerhards wrote: >> I think the main point of the OP was that he logs to a NAS device, so I do >> not see way to write to it without going through a network share. > > Configure rsyslog to write to local disk, and rsync periodically to > the network share. Or, use another instance of rsyslog to copy from > local disk to the network share. > > Any other method will have problems. Even NetApp systems have issues > from time to time. I've found that for every situation I've ever heard someone (including me) say 'never do that' that there is _some_ situaton where 'that' is exactly the right thing to do ;-) I agree that in almost all cases, writing logs to NFS is not a good idea. you really will need a high-end NFS server to have any chance of it working (and the load that the logging will put on the server is going to be significant, unless something does caching, which is what is causing your problems) so this is a _very_ expensive way to go to make logging work. if you are putting the logs on NFS to make them readable by a bunch of machines, consider running rsyslog on a linux system and let that linux system export the log directory via NFS. for read-only access from many machines, this is probably going to rivel the speed of a very expensive NFS server, while being much simpler, cheaper, and more reliable (if you need redundant hardware things get more complicated, but it's still not hard to make it reliable and it will definantly be much cheaper) David Lang From anichols at trumped.org Fri Oct 23 15:22:24 2009 From: anichols at trumped.org (Aaron Nichols) Date: Fri, 23 Oct 2009 07:22:24 -0600 Subject: [rsyslog] Enabling $ActionFileEnableSynccausedmassiveincreaseinwrite volume (bytes/sec) to NAS -cansomeone helpshed light? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com> <4AE14FA7.4010008@freeradius.org> <9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com> <4AE15E5E.2020100@freeradius.org> Message-ID: Thank you everyone for your responses - rather than reply to each post, here is some feedback. The storage system - by all measure we have a "high end" system - It's an EMC Clarion CX3-80 attached to a Celerra NS80 NAS head with 10gb interfaces. The network file system is NFS, the Celerra can also present CIFS shares but I doubt those would work any better. This is not easily replaced with a linux box so that I can do rsync. The logs must be made available to multiple hosts and our legacy implementation *is* a Linux box writing to direct attached disk sharing NFS... it is crumbling under our write/read load so we had to come up with another solution. This solution is understood to be a bit of a compromise - the ideal situation would be direct attached fiberchannel disk and a cluster filesystem but that wasn't an option - mostly for cost. For all the complaints about NFS it performs just fine - if we enable file sync the write load just goes through the roof, but the NAS keeps up. Client bufferring (or throttling) is still possible however because we are transitioning to this new system many of these systems are dual-logging to two destinations so their logging behavior can be observed on two different systems. On the old system running syslog-ng these messages are written to disk and visible near real-time and they don't seem to be doing any rate limiting or otherwise. The debug log that was taken however had file sync enable - and when that's enabled this delay is not seen so I'm not suprised it wasn't visible. I can produce a debug log with file sync disabled for comparison, that's probably something I should have done in the first place. I have tried an strace on the rsyslog process(es) when running normally and haven't been able to get much out of them but that could be my own limited knowledge of strace. `strace -p ` yields a line of information and then just stops, even though the rsyslog process is still working fine and is obviously doing something. I've tried attaching to the different threads as well with the same result. I will see if I can get a comparison debug log with file sync disabled & if there is some guidance on getting a good strace I'm happy to provide that output. Yesterday we had to move the server from RHEL 5.3 to RHEL 5.1 due to some discovered limits in tcp connections - either intentionally or unintentionally different in our installation of 5.3. The server is now performing (with file sync disabled) at an even higher capacity on RH5.1 however I haven't had a chance to review whether the write latency is still observed - hopefully today I can look at that. Thanks again for all the information. Aaron From rgerhards at hq.adiscon.com Fri Oct 23 15:31:18 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 15:31:18 +0200 Subject: [rsyslog] Enabling$ActionFileEnableSynccausedmassiveincreaseinwrite volume(bytes/sec) to NAS -cansomeone helpshed light? References: <9B6E2A8877C38245BFB15CC491A11DA710327A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710327D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103293@GRFEXC.intern.adiscon.com><4AE14FA7.4010008@freeradius.org><9B6E2A8877C38245BFB15CC491A11DA7103295@GRFEXC.intern.adiscon.com><4AE15E5E.2020100@freeradius.org> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032A4@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Friday, October 23, 2009 3:22 PM > To: rsyslog-users > Subject: Re: [rsyslog] > Enabling$ActionFileEnableSynccausedmassiveincreaseinwrite > volume(bytes/sec) to NAS -cansomeone helpshed light? > > Thank you everyone for your responses - rather than reply to each > post, here is some feedback. > > The storage system - by all measure we have a "high end" system - It's > an EMC Clarion CX3-80 attached to a Celerra NS80 NAS head with 10gb > interfaces. The network file system is NFS, the Celerra can also > present CIFS shares but I doubt those would work any better. This is > not easily replaced with a linux box so that I can do rsync. The logs > must be made available to multiple hosts and our legacy implementation > *is* a Linux box writing to direct attached disk sharing NFS... it is > crumbling under our write/read load so we had to come up with another > solution. This solution is understood to be a bit of a compromise - > the ideal situation would be direct attached fiberchannel disk and a > cluster filesystem but that wasn't an option - mostly for cost. For > all the complaints about NFS it performs just fine - if we enable file > sync the write load just goes through the roof, but the NAS keeps up. > > Client bufferring (or throttling) is still possible however because we > are transitioning to this new system many of these systems are > dual-logging to two destinations so their logging behavior can be > observed on two different systems. On the old system running syslog-ng > these messages are written to disk and visible near real-time and they > don't seem to be doing any rate limiting or otherwise. Note that the debug run is *much* slower than the actual run. It's the usual problem, if you instrument the system you also affect it. Performance-wise, we have seen that a debug build can be up to ten times slower (sometimes more, depending on config), but even the regular debug log alone, if turned on, heavily changes performance and order of execution. The root cause is the many debug outputs, which require at least some mutex snyc and that obviously affects the overall system... > The debug log > that was taken however had file sync enable - and when that's enabled > this delay is not seen so I'm not suprised it wasn't visible. I can > produce a debug log with file sync disabled for comparison, that's > probably something I should have done in the first place. >From my POV, that actually doesn't help at all. The reason is that I don't see any unusual in the log, and that means all I can see is that data is being continously written... > > I have tried an strace on the rsyslog process(es) when running > normally and haven't been able to get much out of them but that could > be my own limited knowledge of strace. `strace -p ` yields a line > of information and then just stops, even though the rsyslog process is > still working fine and is obviously doing something. I've tried > attaching to the different threads as well with the same result. ... the strace will probably provide much better info, because here we see the API calls. One thing I forgot to mention is that you should NOT use it together with the debug log. The debug output api calls will make the strace output unreadable. For example, I have these line in one of my test scripts: strace -T -tt -f -F tools/rsyslogd -c..... I don't know out of my head what each option does, but that line may give you a starting point. The end result is not a single line but rather a complete log of each API called PLUS how much time the processing took. That, too, is a quite enormous log... Rainer > > I will see if I can get a comparison debug log with file sync disabled > & if there is some guidance on getting a good strace I'm happy to > provide that output. Yesterday we had to move the server from RHEL 5.3 > to RHEL 5.1 due to some discovered limits in tcp connections - either > intentionally or unintentionally different in our installation of 5.3. > The server is now performing (with file sync disabled) at an even > higher capacity on RH5.1 however I haven't had a chance to review > whether the write latency is still observed - hopefully today I can > look at that. > > Thanks again for all the information. > > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Oct 23 22:22:52 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 23 Oct 2009 13:22:52 -0700 (PDT) Subject: [rsyslog] queue configuration Message-ID: I know that you can create an action queue for a specific output. is there any way to create an action queue for multiple outputs? for example, in my configuration I have :fromhost, !isequal, "127.0.0.1" /var/log/messages;TraditionalFormat :fromhost, isequal, "127.0.0.1" @192.168.1.1;TraditionalForwardFormat *.* @192.168.1.115 *.* @192.168.1.241 *.* @192.168.1.242 *.* @192.168.1.6 *.* @192.168.1.7 *.* @192.168.1.122 :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 & @192.168.1.1;fixsnareForwardFormat2 & ~ :syslogtag, startswith, "MSWinEventLog#011" /var/log/messages;fixsnareFormat & @192.168.1.1;fixsnareForwardFormat & ~ *.* /var/log/messages;TraditionalFormat *.* @192.168.1.1 it would be nice to move the outbound relays off to a different queue, and to put the list of rules that have order dependancies (because they output in different formats to fix up formatting problem) to a seperate queue to spread the work across multiple processes and not slow down the basic writing to file. but as far as I can tell I would have to create a queue for each individual item, and I can't create a queue for the part of the config where I need to discard. am I missing something (including a better way to do everything ;-) David Lang From rgerhards at hq.adiscon.com Fri Oct 23 22:27:06 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 22:27:06 +0200 Subject: [rsyslog] queue configuration References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032AF@GRFEXC.intern.adiscon.com> In short: this can not currently be done and it would require considerable modification to the engine to do so. Part of the work I plan in moving queues to rule sets may actually help, but it is some way to there. Icluding that I first want to get a stable v5 and there are lots of smaller, but important things in front of it (like the DNS resolution issue. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 23, 2009 10:23 PM > To: rsyslog-users > Subject: [rsyslog] queue configuration > > I know that you can create an action queue for a specific output. > > is there any way to create an action queue for multiple outputs? > > for example, in my configuration I have > > :fromhost, !isequal, "127.0.0.1" > /var/log/messages;TraditionalFormat > :fromhost, isequal, "127.0.0.1" > @192.168.1.1;TraditionalForwardFormat > *.* @192.168.1.115 > *.* @192.168.1.241 > *.* @192.168.1.242 > *.* @192.168.1.6 > *.* @192.168.1.7 > *.* @192.168.1.122 > :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 > & @192.168.1.1;fixsnareForwardFormat2 > & ~ > :syslogtag, startswith, "MSWinEventLog#011" > /var/log/messages;fixsnareFormat > & @192.168.1.1;fixsnareForwardFormat > & ~ > *.* /var/log/messages;TraditionalFormat > *.* @192.168.1.1 > > it would be nice to move the outbound relays off to a > different queue, and > to put the list of rules that have order dependancies > (because they output > in different formats to fix up formatting problem) to a > seperate queue to > spread the work across multiple processes and not slow down the basic > writing to file. > > but as far as I can tell I would have to create a queue for each > individual item, and I can't create a queue for the part of > the config > where I need to discard. > > am I missing something (including a better way to do everything ;-) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Oct 23 22:33:35 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 23 Oct 2009 13:33:35 -0700 (PDT) Subject: [rsyslog] queue configuration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032AF@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71032AF@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 23 Oct 2009, Rainer Gerhards wrote: > In short: this can not currently be done and it would require considerable > modification to the engine to do so. Part of the work I plan in moving queues > to rule sets may actually help, but it is some way to there. Icluding that I > first want to get a stable v5 and there are lots of smaller, but important > things in front of it (like the DNS resolution issue. that's what I suspected, I just wanted to make sure. with the new queue engine can you have multiple worker threads? (I seem to remember that initially you couldn't with batch mode, and in almost every case there is no reason to as a single thread in batch mode can do so much more than it could before) David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Friday, October 23, 2009 10:23 PM >> To: rsyslog-users >> Subject: [rsyslog] queue configuration >> >> I know that you can create an action queue for a specific output. >> >> is there any way to create an action queue for multiple outputs? >> >> for example, in my configuration I have >> >> :fromhost, !isequal, "127.0.0.1" >> /var/log/messages;TraditionalFormat >> :fromhost, isequal, "127.0.0.1" >> @192.168.1.1;TraditionalForwardFormat >> *.* @192.168.1.115 >> *.* @192.168.1.241 >> *.* @192.168.1.242 >> *.* @192.168.1.6 >> *.* @192.168.1.7 >> *.* @192.168.1.122 >> :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 >> & @192.168.1.1;fixsnareForwardFormat2 >> & ~ >> :syslogtag, startswith, "MSWinEventLog#011" >> /var/log/messages;fixsnareFormat >> & @192.168.1.1;fixsnareForwardFormat >> & ~ >> *.* /var/log/messages;TraditionalFormat >> *.* @192.168.1.1 >> >> it would be nice to move the outbound relays off to a >> different queue, and >> to put the list of rules that have order dependancies >> (because they output >> in different formats to fix up formatting problem) to a >> seperate queue to >> spread the work across multiple processes and not slow down the basic >> writing to file. >> >> but as far as I can tell I would have to create a queue for each >> individual item, and I can't create a queue for the part of >> the config >> where I need to discard. >> >> am I missing something (including a better way to do everything ;-) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Oct 23 22:35:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 23 Oct 2009 22:35:56 +0200 Subject: [rsyslog] queue configuration References: <9B6E2A8877C38245BFB15CC491A11DA71032AF@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032B1@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 23, 2009 10:34 PM > To: rsyslog-users > Subject: Re: [rsyslog] queue configuration > > On Fri, 23 Oct 2009, Rainer Gerhards wrote: > > > In short: this can not currently be done and it would > require considerable > > modification to the engine to do so. Part of the work I > plan in moving queues > > to rule sets may actually help, but it is some way to > there. Icluding that I > > first want to get a stable v5 and there are lots of > smaller, but important > > things in front of it (like the DNS resolution issue. > > that's what I suspected, I just wanted to make sure. > > with the new queue engine can you have multiple worker > threads? (I seem to > remember that initially you couldn't with batch mode, and in > almost every > case there is no reason to as a single thread in batch mode > can do so much > more than it could before) Yes, you can do that BUT not for disk queues. This was the restriction. It may be that one early stage of batching support limited all workers to one. But we can do multiple workers. Rainer > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > >> Sent: Friday, October 23, 2009 10:23 PM > >> To: rsyslog-users > >> Subject: [rsyslog] queue configuration > >> > >> I know that you can create an action queue for a specific output. > >> > >> is there any way to create an action queue for multiple outputs? > >> > >> for example, in my configuration I have > >> > >> :fromhost, !isequal, "127.0.0.1" > >> /var/log/messages;TraditionalFormat > >> :fromhost, isequal, "127.0.0.1" > >> @192.168.1.1;TraditionalForwardFormat > >> *.* @192.168.1.115 > >> *.* @192.168.1.241 > >> *.* @192.168.1.242 > >> *.* @192.168.1.6 > >> *.* @192.168.1.7 > >> *.* @192.168.1.122 > >> :hostname, contains ,"MSWinEventLog" > /var/log/messages;fixsnareFormat2 > >> & @192.168.1.1;fixsnareForwardFormat2 > >> & ~ > >> :syslogtag, startswith, "MSWinEventLog#011" > >> /var/log/messages;fixsnareFormat > >> & @192.168.1.1;fixsnareForwardFormat > >> & ~ > >> *.* /var/log/messages;TraditionalFormat > >> *.* @192.168.1.1 > >> > >> it would be nice to move the outbound relays off to a > >> different queue, and > >> to put the list of rules that have order dependancies > >> (because they output > >> in different formats to fix up formatting problem) to a > >> seperate queue to > >> spread the work across multiple processes and not slow > down the basic > >> writing to file. > >> > >> but as far as I can tell I would have to create a queue for each > >> individual item, and I can't create a queue for the part of > >> the config > >> where I need to discard. > >> > >> am I missing something (including a better way to do everything ;-) > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Sat Oct 24 21:10:32 2009 From: david at lang.hm (david at lang.hm) Date: Sat, 24 Oct 2009 12:10:32 -0700 (PDT) Subject: [rsyslog] is there a way to find currnt queue status? Message-ID: thinking back to the testing of the v5 code that I have been doing this last week, one thing that would be very useful is if there is a way to find out what the current queue lengths are. one possible way to do this would be to have rsyslog watch for a signal and when it receives it, spit out the queue stats. for the stats definantly the length of each queue, possibly timestamps of oldest and newest message in each queue when spitting them out, some possible approaches are to stderr pro: simple con: stderr may not appear if rsyslog is running as a daemon from startup add a log message to the main queue containing the data pro: still fairly simple con: if there is a long queue, or it is blocked the data may not be visable for a while have each queue walker thread generate a message internally and process it pro: the data will show up fast, through the normal log mechanism con: this is probably the most work to imeplement this is not a critical issue, but it could be something that would speed up development in the long run by making it easier to figure out the state of the system. or I could have just missed how something like this is already supported ;-) David Lang From rgerhards at hq.adiscon.com Sat Oct 24 21:14:57 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 24 Oct 2009 21:14:57 +0200 Subject: [rsyslog] is there a way to find currnt queue status? References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032B2@GRFEXC.intern.adiscon.com> It's very partially being supported right now via imdiag. That module enables to query the main message queue length (and only for the main message queue). There is also a very, very rudamentary Java app in the v5. This was created as part of the debugging/performance testing effort earlier this year, but unfortunately more important things got into the way. An interace needs to be designed (and implemented) for performance counters. IT's on the agenda, but unfortunately not on the top spot. In concur that it would be very useful in many cases. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Saturday, October 24, 2009 9:11 PM > To: rsyslog-users > Subject: [rsyslog] is there a way to find currnt queue status? > > thinking back to the testing of the v5 code that I have been > doing this > last week, one thing that would be very useful is if there is > a way to > find out what the current queue lengths are. > > one possible way to do this would be to have rsyslog watch > for a signal > and when it receives it, spit out the queue stats. > > for the stats definantly the length of each queue, possibly > timestamps of > oldest and newest message in each queue > > when spitting them out, some possible approaches are > > to stderr > pro: simple > con: stderr may not appear if rsyslog is running as a > daemon from startup > > add a log message to the main queue containing the data > pro: still fairly simple > con: if there is a long queue, or it is blocked the data > may not be > visable for a while > > have each queue walker thread generate a message internally > and process it > pro: the data will show up fast, through the normal log mechanism > con: this is probably the most work to imeplement > > this is not a critical issue, but it could be something that > would speed > up development in the long run by making it easier to figure > out the state > of the system. > > or I could have just missed how something like this is > already supported > ;-) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Mon Oct 26 08:17:23 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 26 Oct 2009 08:17:23 +0100 Subject: [rsyslog] queue configuration References: <9B6E2A8877C38245BFB15CC491A11DA71032AF@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032B5@GRFEXC.intern.adiscon.com> David, all, I have thought a bit more about this. Actually, I think almost everything is already present by now, just some glue is missing. The most important thing, ruleset-specific "main" queues was due to implement soon (I worked towards that the past two month). So, in short, I'll have a look at it this week while running the stability lab tests (they always take considerable time). I don't want to promise anything, but it may be possible that this feature could become available shortly. That would also build that basis for some considerabe performance improvement (for some environments), so it seems to be very relevant to my short-term goals. Thanks for bringing this up and making me think about it ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 23, 2009 10:34 PM > To: rsyslog-users > Subject: Re: [rsyslog] queue configuration > > On Fri, 23 Oct 2009, Rainer Gerhards wrote: > > > In short: this can not currently be done and it would > require considerable > > modification to the engine to do so. Part of the work I > plan in moving queues > > to rule sets may actually help, but it is some way to > there. Icluding that I > > first want to get a stable v5 and there are lots of > smaller, but important > > things in front of it (like the DNS resolution issue. > > that's what I suspected, I just wanted to make sure. > > with the new queue engine can you have multiple worker > threads? (I seem to > remember that initially you couldn't with batch mode, and in > almost every > case there is no reason to as a single thread in batch mode > can do so much > more than it could before) > > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > >> Sent: Friday, October 23, 2009 10:23 PM > >> To: rsyslog-users > >> Subject: [rsyslog] queue configuration > >> > >> I know that you can create an action queue for a specific output. > >> > >> is there any way to create an action queue for multiple outputs? > >> > >> for example, in my configuration I have > >> > >> :fromhost, !isequal, "127.0.0.1" > >> /var/log/messages;TraditionalFormat > >> :fromhost, isequal, "127.0.0.1" > >> @192.168.1.1;TraditionalForwardFormat > >> *.* @192.168.1.115 > >> *.* @192.168.1.241 > >> *.* @192.168.1.242 > >> *.* @192.168.1.6 > >> *.* @192.168.1.7 > >> *.* @192.168.1.122 > >> :hostname, contains ,"MSWinEventLog" > /var/log/messages;fixsnareFormat2 > >> & @192.168.1.1;fixsnareForwardFormat2 > >> & ~ > >> :syslogtag, startswith, "MSWinEventLog#011" > >> /var/log/messages;fixsnareFormat > >> & @192.168.1.1;fixsnareForwardFormat > >> & ~ > >> *.* /var/log/messages;TraditionalFormat > >> *.* @192.168.1.1 > >> > >> it would be nice to move the outbound relays off to a > >> different queue, and > >> to put the list of rules that have order dependancies > >> (because they output > >> in different formats to fix up formatting problem) to a > >> seperate queue to > >> spread the work across multiple processes and not slow > down the basic > >> writing to file. > >> > >> but as far as I can tell I would have to create a queue for each > >> individual item, and I can't create a queue for the part of > >> the config > >> where I need to discard. > >> > >> am I missing something (including a better way to do everything ;-) > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From tbergfeld at hq.adiscon.com Tue Oct 27 11:12:18 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Tue, 27 Oct 2009 11:12:18 +0100 Subject: [rsyslog] rsyslog 5.3.3 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032CC@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.3.3, a member of the devel branch. This version offers a partially rewritten queue engine. The functionality was streamlined and simplified, resulting in not only fewer bug potential but also greater speed. This engine sets an important basis on which all further work can build. Note that due to the magnitude of changes, there is some increased bug potential. But in general, the new engine will be able to better handle some situations where the previous engine ran into problems. Also there are some bug fixes. This is a recommended update for all users of the devel branch. ChangeLog: http://www.rsyslog.com/Article419.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-183.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From martinmie at PartyGaming.com Tue Oct 27 11:56:52 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Tue, 27 Oct 2009 11:56:52 +0100 Subject: [rsyslog] rsyslog in uninterruptable sleep Message-ID: <0B1B9163138571439EAF804646F3F6AA0892EE19@GIBSVWIN004X.partygaming.local> Hi, I just did some small changes to the rsyslog.conf and wanted to restart the process: # /etc/init.d/rsyslog restart Shutting down system logger: [FAILED] Starting system logger: Already running. [FAILED] # ps -elf | grep rsyslog 5 D root 2037 1 1 76 0 - 12040 sync_b Oct18 ? 03:32:10 rsyslogd -c 4 -x -f /etc/rsyslog.conf -i /var/run/syslogd.pid This is rsyslog 4.2.0 running on RHEL5 and it's not the first time it happens. Any ideas on why this might happened and how to solve this w/o rebooting the system?? Thanks, Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From martinmie at PartyGaming.com Tue Oct 27 11:59:19 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Tue, 27 Oct 2009 11:59:19 +0100 Subject: [rsyslog] rsyslog and snare Message-ID: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> Hi, We recently started to receive logs from a Snare client. I applied the configuration changes proposed here: http://wiki.rsyslog.com/index.php/Using_Snare_as_a_client_on_Windows After restarting rsyslog, I get the following error: -- 2009-10-27T06:45:39.067279-04:00 logserger rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="29021" x-info="http://www.rsyslog.com"] (re)start 2009-10-27T06:45:39.063213-04:00 logserger rsyslogd-3000: unknown priority name "" [try http://www.rsyslog.com/e/3000 ] 2009-10-27T06:45:39.063571-04:00 logserger rsyslogd: the last error occured in /etc/rsyslog.conf, line 18 2009-10-27T06:45:39.064044-04:00 logserger rsyslogd: warning: selector line without actions will be discarded 2009-10-27T06:45:39.064213-04:00 logserger rsyslogd-2123: CONFIG ERROR: could not interpret master config file '/etc/rsyslog.conf'. [try http://www.rsyslog.com/e/2123 ] -- If I comment out the "%msg:::space-cc%" part it starts up cleanly but I assume that the filtering won't work. Suggestions? Thanks, Martin This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. From rgerhards at hq.adiscon.com Tue Oct 27 16:02:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 27 Oct 2009 16:02:17 +0100 Subject: [rsyslog] rsyslog in uninterruptable sleep References: <0B1B9163138571439EAF804646F3F6AA0892EE19@GIBSVWIN004X.partygaming.local> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032D3@GRFEXC.intern.adiscon.com> I think we just recently had this on the mailing list or bug tracker and I made a fix for it. I guess the fix is included in 4.4.2. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Martin Mielke > Sent: Tuesday, October 27, 2009 11:57 AM > To: rsyslog-users > Subject: [rsyslog] rsyslog in uninterruptable sleep > > > Hi, > > I just did some small changes to the rsyslog.conf and wanted to restart > the process: > > # /etc/init.d/rsyslog restart > Shutting down system logger: [FAILED] > Starting system logger: Already running. > [FAILED] > > # ps -elf | grep rsyslog > 5 D root 2037 1 1 76 0 - 12040 sync_b Oct18 ? > 03:32:10 rsyslogd -c 4 -x -f /etc/rsyslog.conf -i /var/run/syslogd.pid > > > This is rsyslog 4.2.0 running on RHEL5 and it's not the first time it > happens. > > > Any ideas on why this might happened and how to solve this w/o > rebooting > the system?? > > > Thanks, > Martin > > > This email and any attachments are confidential, and may be legally > privileged and protected by copyright. If you are not the intended > recipient dissemination or copying of this email is prohibited. If you > have received this in error, please notify the sender by replying by > email and then delete the email completely from your system. > > Any views or opinions are solely those of the sender. This > communication is not intended to form a binding contract unless > expressly indicated to the contrary and properly authorised. Any > actions taken on the basis of this email are at the recipient's own > risk. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Oct 27 17:53:20 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 27 Oct 2009 17:53:20 +0100 Subject: [rsyslog] important enhancement in rsyslog upcoming Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032D6@GRFEXC.intern.adiscon.com> Hi all, I was finally able to add some new featueres to rsyslog (stability is still my priority), and I think it is a very exciting one. In short: we can now partition input message into different queues, dramatically decreasing lock contention potential. Please read yourself: http://blog.gerhards.net/2009/10/next-round-of-performance-enhancement.html The change is committed, but I'll probably be busy a day or two with lab testing and testbench enhancement before I continue to add more enhancements. Rainer From epiphani at gmail.com Tue Oct 27 17:59:27 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Tue, 27 Oct 2009 12:59:27 -0400 Subject: [rsyslog] important enhancement in rsyslog upcoming In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032D6@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71032D6@GRFEXC.intern.adiscon.com> Message-ID: This looks like it has a lot of potential! I look forward to trying it. -Aaron On Tue, Oct 27, 2009 at 12:53 PM, Rainer Gerhards wrote: > Hi all, > > I was finally able to add some new featueres to rsyslog (stability is still > my priority), and I think it is a very exciting one. In short: we can now > partition input message into different queues, dramatically decreasing lock > contention potential. > > Please read yourself: > > http://blog.gerhards.net/2009/10/next-round-of-performance-enhancement.html > > The change is committed, but I'll probably be busy a day or two with lab > testing and testbench enhancement before I continue to add more enhancements. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Oct 27 18:09:06 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 27 Oct 2009 18:09:06 +0100 Subject: [rsyslog] important enhancement in rsyslog upcoming References: <9B6E2A8877C38245BFB15CC491A11DA71032D6@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032D7@GRFEXC.intern.adiscon.com> As a side-note, this does not yet mean that the input module itself runs on multiple threads. So the new functionality probably works best across different inputs. Massiv-parallel inputs are on my agenda as well, but some other things need to be done first so that I will have the necessary basis to build on (I'll not use a simple "one thread per connection" model, as this does not scale well). I'll probably go for multiplexed io with potentially one queue per worker, and quick worker engagement into that queue where no wait is necessary ... but more later ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Tuesday, October 27, 2009 5:59 PM > To: rsyslog-users > Subject: Re: [rsyslog] important enhancement in rsyslog upcoming > > This looks like it has a lot of potential! I look forward to trying > it. > > -Aaron > > On Tue, Oct 27, 2009 at 12:53 PM, Rainer Gerhards > wrote: > > Hi all, > > > > I was finally able to add some new featueres to rsyslog (stability is > still > > my priority), and I think it is a very exciting one. In short: we can > now > > partition input message into different queues, dramatically > decreasing lock > > contention potential. > > > > Please read yourself: > > > > http://blog.gerhards.net/2009/10/next-round-of-performance- > enhancement.html > > > > The change is committed, but I'll probably be busy a day or two with > lab > > testing and testbench enhancement before I continue to add more > enhancements. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jbondc at openmv.com Tue Oct 27 20:40:42 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Tue, 27 Oct 2009 15:40:42 -0400 Subject: [rsyslog] Property-Based Filters, not working? Message-ID: <002101ca573d$5f417f70$1dc47e50$@com> I tried adding the following to my config file: :msg,contains,"MSWinEventLog" *.* /var/log/windows.log :msg, !contains, "MSWinEventLog" *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err /var/log/messages security.* /var/log/security auth.info;authpriv.info /var/log/auth.log However the filter doesn't seem to apply, when debugging the parse sequence goes like this: 5250.008750501:main thread: Action 0x80a8600: queue 0x8094c00 created 5250.008823471:main thread: cfline: ':msg,contains,"MSWinEventLog"' 5250.008884805:main thread: selector line successfully processed 5250.008943799:main thread: - property-based filter 5250.009059262:main thread: tried selector action for builtin-file: -2001 5250.009130893:main thread: tried selector action for builtin-fwd: -2001 5250.009195089:main thread: tried selector action for builtin-shell: -2001 5250.009259497:main thread: tried selector action for builtin-discard: -2001 5250.009324602:main thread: tried selector action for builtin-usrmsg: -2001 5250.009394008:main thread: tried selector action for ompgsql.so: -2001 5250.009454057:main thread: config line NOT successfully processed Am I missing something? rsyslogd 4.4.2, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: Yes FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: No FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: No Runtime Instrumentation (slow code): No From david at lang.hm Tue Oct 27 21:03:01 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 27 Oct 2009 13:03:01 -0700 (PDT) Subject: [rsyslog] Property-Based Filters, not working? In-Reply-To: <002101ca573d$5f417f70$1dc47e50$@com> References: <002101ca573d$5f417f70$1dc47e50$@com> Message-ID: On Tue, 27 Oct 2009, Jonathan Bond-Caron wrote: > Date: Tue, 27 Oct 2009 15:40:42 -0400 > From: Jonathan Bond-Caron > Reply-To: rsyslog-users > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Property-Based Filters, not working? > > I tried adding the following to my config file: > > > > :msg,contains,"MSWinEventLog" > > > > *.* /var/log/windows.log > > > > :msg, !contains, "MSWinEventLog" > > > > *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err > /var/log/messages this is not how the property based filters work. I made the same mistake instead of :msg,contains,"MSWinEventLog" *.* /var/log/windows.log it should be :msg,contains,"MSWinEventLog" /var/log/windows.log when you do filter rules those filters are everything, you don't also do facility/priority filters as well (or if you need to do so, you need to make a compound if (() and ()) type rule) FYI, what I do with snare is $template fixsnareFormat,"%timereported% %HOSTNAME% MSWinEventLog %syslogtag:18:$%%msg:::drop-last-lf%\n" $template fixsnareForwardFormat,"<%pri%>%timereported% %HOSTNAME% MSWinEventLog %syslogtag:18:$:%%msg:::drop-last-lf%\n" $template fixsnareFormat2,"%timereported% %fromhost-ip% broken-MSWinEventLog %HOSTNAME% %syslogtag:18:$%%msg:::drop-last-lf%\n" $template fixsnareForwardFormat2,"<%pri%>%timereported% %fromhost-ip% broken-MSWinEventLog %HOSTNAME% %syslogtag:18:$:%%msg:::drop-last-lf%\n" :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 & @192.168.210.8;fixsnareForwardFormat2 & ~ :syslogtag, startswith, "MSWinEventLog\011" /var/log/messages;fixsnareFormat & @192.168.210.8;fixsnareForwardFormat & ~ *.* /var/log/messages;TraditionalFormat *.* @192.168.210.8 the fixsnare*2 formats are to handle cases where I haven't tracked down why snare is sending things incorrectly (8 of a couple hundred servers) David Lang > security.* /var/log/security > > auth.info;authpriv.info /var/log/auth.log > > > > > > However the filter doesn't seem to apply, when debugging the parse sequence > goes like this: > > > > 5250.008750501:main thread: Action 0x80a8600: queue 0x8094c00 created > > 5250.008823471:main thread: cfline: ':msg,contains,"MSWinEventLog"' > > 5250.008884805:main thread: selector line successfully processed > > 5250.008943799:main thread: - property-based filter > > 5250.009059262:main thread: tried selector action for builtin-file: -2001 > > 5250.009130893:main thread: tried selector action for builtin-fwd: -2001 > > 5250.009195089:main thread: tried selector action for builtin-shell: -2001 > > 5250.009259497:main thread: tried selector action for builtin-discard: -2001 > > 5250.009324602:main thread: tried selector action for builtin-usrmsg: -2001 > > 5250.009394008:main thread: tried selector action for ompgsql.so: -2001 > > 5250.009454057:main thread: config line NOT successfully processed > > > > Am I missing something? > > > > rsyslogd 4.4.2, compiled with: > > FEATURE_REGEXP: Yes > > FEATURE_LARGEFILE: Yes > > FEATURE_NETZIP (message compression): Yes > > GSSAPI Kerberos 5 support: No > > FEATURE_DEBUG (debug build, slow code): No > > Atomic operations supported: No > > Runtime Instrumentation (slow code): No > > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Tue Oct 27 21:05:47 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 27 Oct 2009 13:05:47 -0700 (PDT) Subject: [rsyslog] control character escaping change? Message-ID: it appears that the 5.3 development series recently changed the default for escaping control characters from # to \ so a tab changed from #011 to \011 was this a deliberate change going forward, or a mistake? David Lang From david at lang.hm Tue Oct 27 21:32:39 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 27 Oct 2009 13:32:39 -0700 (PDT) Subject: [rsyslog] rsyslog and snare In-Reply-To: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> Message-ID: On Tue, 27 Oct 2009, Martin Mielke wrote: > Hi, > > We recently started to receive logs from a Snare client. > I applied the configuration changes proposed here: > http://wiki.rsyslog.com/index.php/Using_Snare_as_a_client_on_Windows I just got a chance to read this, and I disagree with it's recommendation the #011 is a tab. snare uses it to seperate fields from the windows event log. if you just replace it with a space you loose the ability to do things like look for field 10 because field 8 may contain spaces. In addition there are two formats of logs that snare can output to syslog 1. the snare format 2. 'syslog' format the difference seems to be that in snare format it does Jan 1 01:01:01 mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri while in syslog format it does Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri the biggest change is the space between the host and MSWinEventLog there is also sometime a "1#011" before the field with Security in it in this example and sometimes not. I have not figured out the pattern here yet (it may be the format difference, my parsing logic just looks for a '1' in the first field, and if it finds it, drops that field) what I do is to setup the following rules to reformat the syslog formatted messages to look like Jan 1 01:01:01 mail.abc.com MSWinEventLog 1#011Security#0114169#011Fri I can then filter on the program name MSWinEventLog and later parsing tools can examine specific fields of the log message I have not fully figured out the snare formatted message, but the *2 formats at lease prevent lots of garbage from ending up in the host and program fields. $template fixsnareFormat,"%timereported% %HOSTNAME% MSWinEventLog %syslogtag:18:$%%msg:::drop-last-lf%\n" $template fixsnareForwardFormat,"<%pri%>%timereported% %HOSTNAME% MSWinEventLog %syslogtag:18:$:%%msg:::drop-last-lf%\n" $template fixsnareFormat2,"%timereported% %fromhost-ip% broken-MSWinEventLog %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" $template fixsnareForwardFormat2,"<%pri%>%timereported% %fromhost-ip% broken-MSWinEventLog %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" :hostname, contains ,"MSWinEventLog" /var/log/messages;fixsnareFormat2 & @192.168.1.8;fixsnareForwardFormat2 & ~ :syslogtag, startswith, "MSWinEventLog#011" /var/log/messages;fixsnareFormat & @192.168.1.8;fixsnareForwardFormat & ~ *.* /var/log/messages;TraditionalFormat *.* @192.168.1.8 I updated to wiki to show this option > After restarting rsyslog, I get the following error: > -- > 2009-10-27T06:45:39.067279-04:00 logserger rsyslogd: [origin > software="rsyslogd" swVersion="4.2.0" x-pid="29021" > x-info="http://www.rsyslog.com"] (re)start > 2009-10-27T06:45:39.063213-04:00 logserger rsyslogd-3000: unknown > priority name "" [try http://www.rsyslog.com/e/3000 ] > 2009-10-27T06:45:39.063571-04:00 logserger rsyslogd: the last error > occured in /etc/rsyslog.conf, line 18 > 2009-10-27T06:45:39.064044-04:00 logserger rsyslogd: warning: selector > line without actions will be discarded > 2009-10-27T06:45:39.064213-04:00 logserger rsyslogd-2123: CONFIG ERROR: > could not interpret master config file '/etc/rsyslog.conf'. [try > http://www.rsyslog.com/e/2123 ] > -- > > If I comment out the "%msg:::space-cc%" part it starts up cleanly but I > assume that the filtering won't work. the %msg:::space-cc% shouldn't have been listed as a seperate line, that would replace %msg% in a format string. David Lang > > Suggestions? > > > Thanks, > Martin > > > > > > This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. > > Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Oct 27 22:09:58 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 27 Oct 2009 22:09:58 +0100 Subject: [rsyslog] control character escaping change? References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032D8@GRFEXC.intern.adiscon.com> This smells like a mistake. At least I don't remember changing that, but I'll look at the changelog tomorrow... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, October 27, 2009 9:06 PM > To: rsyslog-users > Subject: [rsyslog] control character escaping change? > > it appears that the 5.3 development series recently changed > the default > for escaping control characters from # to \ so a tab changed > from #011 to > \011 > > was this a deliberate change going forward, or a mistake? > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From r.bhatia at ipax.at Wed Oct 28 11:40:09 2009 From: r.bhatia at ipax.at (Raoul Bhatia [IPAX]) Date: Wed, 28 Oct 2009 11:40:09 +0100 Subject: [rsyslog] version numbers from git repository In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103292@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103292@GRFEXC.intern.adiscon.com> Message-ID: <4AE81F89.3070409@ipax.at> On 10/23/2009 07:54 AM, Rainer Gerhards wrote: > I've looked a bit into the issue. Neither of them use the GNU build system. > In essence, it boils down to running "git describe" as part of the build > process, generating a file with it. However, I do not know how I may achieve > that. It should probably be run during the ./configure stage. Maybe someone > can provide some advise ;) i like the way drbd includes the git-hash in its version output: > # cat /proc/drbd > version: 8.3.2 (api:88/proto:86-90) > GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by phil at fat-tyre, 2009-07-03 15:35:39 > ... take a look at line 112ff at (especially line 125) http://git.drbd.org/?p=drbd-8.3.git;a=blob;f=drbd/Makefile;hb=HEAD there, a drbd_buildtag.c file is created with the help of: > git rev-parse HEAD the output of this Makefile section is: > /* automatically generated. DO NOT EDIT. */ > #include > const char * drbd_buildtag(void) > { > return "GIT-hash: 9ce425f51860f1205395bf7127cad3c427070837" > " build by raoul at dev.ipax.at, 2008-09-04 17:00:24"; > } one can use this, or something similar using e.g. git-describe. cheers, raoul -- ____________________________________________________________________ DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at Technischer Leiter IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at Barawitzkagasse 10/2/2/11 email. office at ipax.at 1190 Wien tel. +43 1 3670030 FN 277995t HG Wien fax. +43 1 3670030 15 ____________________________________________________________________ From rgerhards at hq.adiscon.com Wed Oct 28 12:06:32 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 12:06:32 +0100 Subject: [rsyslog] version numbers from git repository References: <9B6E2A8877C38245BFB15CC491A11DA710328C@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103292@GRFEXC.intern.adiscon.com> <4AE81F89.3070409@ipax.at> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032DC@GRFEXC.intern.adiscon.com> Hi Raoul, this is useful, but unfortunately it does not tell how to do it with GNU autotools. I'll keep this on my mind when I try to tackle it, but so far it looks like I need too much time to find out the details. If someone could suggest a solution within the GNU build system, I'd love to include that ASAP. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX] > Sent: Wednesday, October 28, 2009 11:40 AM > To: rsyslog-users > Subject: Re: [rsyslog] version numbers from git repository > > On 10/23/2009 07:54 AM, Rainer Gerhards wrote: > > I've looked a bit into the issue. Neither of them use the GNU build > system. > > In essence, it boils down to running "git describe" as part of the > build > > process, generating a file with it. However, I do not know how I may > achieve > > that. It should probably be run during the ./configure stage. Maybe > someone > > can provide some advise ;) > > i like the way drbd includes the git-hash in its version output: > > # cat /proc/drbd > > version: 8.3.2 (api:88/proto:86-90) > > GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by phil at fat- > tyre, 2009-07-03 15:35:39 > > ... > > take a look at line 112ff at (especially line 125) > http://git.drbd.org/?p=drbd-8.3.git;a=blob;f=drbd/Makefile;hb=HEAD > > there, a drbd_buildtag.c file is created with the help of: > > git rev-parse HEAD > > the output of this Makefile section is: > > /* automatically generated. DO NOT EDIT. */ > > #include > > const char * drbd_buildtag(void) > > { > > return "GIT-hash: 9ce425f51860f1205395bf7127cad3c427070837" > > " build by raoul at dev.ipax.at, 2008-09-04 17:00:24"; > > } > > one can use this, or something similar using e.g. git-describe. > > cheers, > raoul > -- > ____________________________________________________________________ > DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at > Technischer Leiter > > IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at > Barawitzkagasse 10/2/2/11 email. office at ipax.at > 1190 Wien tel. +43 1 3670030 > FN 277995t HG Wien fax. +43 1 3670030 15 > ____________________________________________________________________ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jbondc at openmv.com Wed Oct 28 12:32:35 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Wed, 28 Oct 2009 07:32:35 -0400 Subject: [rsyslog] Property-Based Filters, not working? In-Reply-To: References: <002101ca573d$5f417f70$1dc47e50$@com> Message-ID: <001b01ca57c2$593248a0$0b96d9e0$@com> On Tue Oct 27 04:03 PM, david at lang.hm wrote: > On Tue, 27 Oct 2009, Jonathan Bond-Caron wrote: > > > Date: Tue, 27 Oct 2009 15:40:42 -0400 > > From: Jonathan Bond-Caron > > Reply-To: rsyslog-users > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] Property-Based Filters, not working? > > > > I tried adding the following to my config file: > > > > > > :msg,contains,"MSWinEventLog" > > > > *.* /var/log/windows.log > > > > :msg, !contains, "MSWinEventLog" > > > > *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err > > /var/log/messages > > this is not how the property based filters work. > > I made the same mistake > > instead of > > :msg,contains,"MSWinEventLog" *.* /var/log/windows.log > > it should be > > :msg,contains,"MSWinEventLog" /var/log/windows.log > > when you do filter rules those filters are everything, you don't also > do facility/priority filters as well (or if you need to do so, you > need to make a compound if (() and ()) type rule) > Thanks a lot, that works well. The docs are bit misleading: http://www.rsyslog.com/doc-rsyslog_conf_filter.html It could show something like this: :msg, contains, "error" /var/log/error.log :msg, !contains, "error" ~ All messages that contain the word "error" are logged to /var/log/error.log All messages that do not contain the word "error" are thrown away. From rgerhards at hq.adiscon.com Wed Oct 28 12:35:25 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 12:35:25 +0100 Subject: [rsyslog] Property-Based Filters, not working? References: <002101ca573d$5f417f70$1dc47e50$@com> <001b01ca57c2$593248a0$0b96d9e0$@com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032DD@GRFEXC.intern.adiscon.com> > Thanks a lot, that works well. > > The docs are bit misleading: > http://www.rsyslog.com/doc-rsyslog_conf_filter.html > > It could show something like this: > :msg, contains, "error" /var/log/error.log > :msg, !contains, "error" ~ > > All messages that contain the word "error" are logged to > /var/log/error.log > All messages that do not contain the word "error" are thrown away. The doc in general could be much improved (any volunteers?), but here I think it is right. The link you quote defines what filters are, but what you post is not only a filter but a full selector line, consisting from a filter and the associated action. I agree that it would be useful to have more scenario-based cases which contain all the pieces put together (again, any volunteers?). Rainer From jbondc at openmv.com Wed Oct 28 13:25:40 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Wed, 28 Oct 2009 08:25:40 -0400 Subject: [rsyslog] Property-Based Filters, not working? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032DD@GRFEXC.intern.adiscon.com> References: <002101ca573d$5f417f70$1dc47e50$@com> <001b01ca57c2$593248a0$0b96d9e0$@com> <9B6E2A8877C38245BFB15CC491A11DA71032DD@GRFEXC.intern.adiscon.com> Message-ID: <001c01ca57c9$c3870810$4a951830$@com> On Wed Oct 28 07:35 AM, Rainer Gerhards wrote: > > Thanks a lot, that works well. > > > > The docs are bit misleading: > > http://www.rsyslog.com/doc-rsyslog_conf_filter.html > > > > It could show something like this: > > :msg, contains, "error" /var/log/error.log > > :msg, !contains, "error" ~ > > > > All messages that contain the word "error" are logged to > > /var/log/error.log All messages that do not contain the word "error" > > are thrown away. > > The doc in general could be much improved (any volunteers?), but here > I think it is right. The link you quote defines what filters are, but > what you post is not only a filter but a full selector line, > consisting from a filter and the associated action. > > I agree that it would be useful to have more scenario-based cases > which contain all the pieces put together (again, any volunteers?). Noted ;) I'll prepare a patch for 4.4.2 this week, will also fix an issue with the syslog 'tag' parsing. It should terminate when it sees a tab or escape control character. Makes it a little more RFC3164 compliant. From a.smith at ukgrid.net Wed Oct 28 13:53:21 2009 From: a.smith at ukgrid.net (Andy Smith) Date: Wed, 28 Oct 2009 12:53:21 +0000 Subject: [rsyslog] config prob with mail.info messages Message-ID: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> Hi, I?m finding rsyslog is logging all routine (I assume mail.info) mail messages to the main messages file (FreeBSD 7.0). Can someone suggest why based on the config: *.err;kern.warning;auth.notice;mail.crit;local7.none /dev/console *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err;local7.none /var/log/messages security.* /var/log/security auth.info;authpriv.info /var/log/auth.log mail.info /var/log/maillog lpr.info /var/log/lpd-errs ftp.info /var/log/xferlog cron.* /var/log/cron *.=debug;local7.none /var/log/debug.log *.emerg * local7.* :ommysql:127.0.0.1,Syslog,syslog,lisboa08 !startslip *.* /var/log/slip.log !ppp *.* /var/log/ppp.log The messages in the messages file dont include a facility or criticality value so makes it tough to work out. For example: 2009-10-28T12:21:36.782834+00:00 ourmailserver tpop3d[90983]: connections_post_select: client [3]dave at somedomain.co.uk(1.8.1.1): disconnected; 93/602717 bytes read/written Can anyone give me a clue?? thanks, Andy. From rgerhards at hq.adiscon.com Wed Oct 28 13:56:08 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 13:56:08 +0100 Subject: [rsyslog] config prob with mail.info messages References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032E2@GRFEXC.intern.adiscon.com> I have to admit I do not really understand the question, but to record severity and facility, you can follow this guide: http://www.rsyslog.com/doc-rsyslog_recording_pri.html HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Andy Smith > Sent: Wednesday, October 28, 2009 1:53 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] config prob with mail.info messages > > Hi, > > I?m finding rsyslog is logging all routine (I assume mail.info) > mail messages to the main messages file (FreeBSD 7.0). Can someone > suggest why based on the config: > > *.err;kern.warning;auth.notice;mail.crit;local7.none > /dev/console > *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err;local7.no > ne > /var/log/messages > security.* /var/log/security > auth.info;authpriv.info /var/log/auth.log > mail.info /var/log/maillog > lpr.info /var/log/lpd-errs > ftp.info /var/log/xferlog > cron.* /var/log/cron > *.=debug;local7.none > /var/log/debug.log > *.emerg * > local7.* :ommysql:127.0.0.1,Syslog,syslog,lisboa08 > !startslip > *.* /var/log/slip.log > !ppp > *.* /var/log/ppp.log > > The messages in the messages file dont include a facility or > criticality value so makes it tough to work out. For example: > > 2009-10-28T12:21:36.782834+00:00 ourmailserver tpop3d[90983]: > connections_post_select: client [3]dave at somedomain.co.uk(1.8.1.1): > disconnected; 93/602717 bytes read/written > > > Can anyone give me a clue?? > > thanks, Andy. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jbondc at openmv.com Wed Oct 28 14:04:24 2009 From: jbondc at openmv.com (Jonathan Bond-Caron) Date: Wed, 28 Oct 2009 09:04:24 -0400 Subject: [rsyslog] rsyslog and snare In-Reply-To: References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> Message-ID: <002301ca57cf$2cece590$86c6b0b0$@com> On Tue Oct 27 04:32 PM, david at lang.hm wrote: > On Tue, 27 Oct 2009, Martin Mielke wrote: > > > Hi, > > > > We recently started to receive logs from a Snare client. > > I applied the configuration changes proposed here: > > http://wiki.rsyslog.com/index.php/Using_Snare_as_a_client_on_Windows > > I just got a chance to read this, and I disagree with it's > recommendation > > the #011 is a tab. > > snare uses it to seperate fields from the windows event log. > > if you just replace it with a space you loose the ability to do things > like look for field 10 because field 8 may contain spaces. > > In addition there are two formats of logs that snare can output to > syslog > > 1. the snare format > 2. 'syslog' format > > the difference seems to be that in snare format it does > > Jan 1 01:01:01 > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > > while in syslog format it does > > Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri > > Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric character will terminate the TAG field and will be assumed to be the starting character of the CONTENT field.) The result would be: TAG: '' MSG: '#011MSWinEventLog#0111#011Security#0114169#011Fri...' At least I think, any thoughts Rainer? The following works for me: // If first character not alpha-numeric, skip tag parsing if( !isalphanum((int)*p2parse) ) bTAGCharDetected = 1; Is this bad for other loggers? From rgerhards at hq.adiscon.com Wed Oct 28 14:12:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 14:12:53 +0100 Subject: [rsyslog] rsyslog and snare References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> <002301ca57cf$2cece590$86c6b0b0$@com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> > > Jan 1 01:01:01 > > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > > > > while in syslog format it does > > > > Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri > > > > First things first: I really don't like to dig into snare, as I do not think it is a really good solution (but how could the designer of http://www.eventreporter.com say otherwise? ;)). so I assume you are talking about this message: mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric > character > will terminate the TAG field and will be assumed to be the starting > character of the CONTENT field.) Another thing to put straight: RFC3164 is NOT a standard, it is an informational document that describes things seen on the wire. And it does not even describe all *popular* cases. If it all, it gains some standard function via RFC3195, but this is questionable. Just for the records ;) So the key point here is that we have nothing to "respect" or "obey to", but rather something that describes things seen in practice. For any decent syslog receiver, this means it must try to work equally well with everything that comes in via legacy syslog. But, granted, RFC3164 is useful if we have nothing else to look at. So let's do that. We'll immediately see that snare is broken, because no control characters are allowed in the hostname. So rsyslog does the right thing and escapes these characters. So everthing belongs to the hostname and thus you have problems with tag. Damn... I didn't want to say somethign about snare. Now it has happened again. My conclusion: fix snare or use something that works ;) Rainer > > The result would be: > TAG: '' > MSG: '#011MSWinEventLog#0111#011Security#0114169#011Fri...' > > At least I think, any thoughts Rainer? > > The following works for me: > // If first character not alpha-numeric, skip tag parsing > if( !isalphanum((int)*p2parse) ) > bTAGCharDetected = 1; > > Is this bad for other loggers? > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From a.smith at ukgrid.net Wed Oct 28 14:17:41 2009 From: a.smith at ukgrid.net (Andy Smith) Date: Wed, 28 Oct 2009 13:17:41 +0000 Subject: [rsyslog] config prob with mail.info messages In-Reply-To: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> Message-ID: <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> Hi Rainer, ok, the question is because based on my config I do not expect or want routine mail messages logged to my messages file, it is over run by messages from courier IMAP and TPOP3 daemons. When using the built in FreeBSD syslog these types of messages are written to the maillog log file, and I dont understand why the same isnt true of rsyslog given the config file I am using, thanks Andy. Quoting Rainer Gerhards: > I have to admit I do not really understand the question From rgerhards at hq.adiscon.com Wed Oct 28 14:43:43 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 14:43:43 +0100 Subject: [rsyslog] config prob with mail.info messages References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032E5@GRFEXC.intern.adiscon.com> Ah, I see. It would probably be useful to see the priority values, as specified in the other link I posted. Maybe the problem is rooted there. Ultimately, a debug log could bring the information how selectors are processed. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Andy Smith > Sent: Wednesday, October 28, 2009 2:18 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] config prob with mail.info messages > > Hi Rainer, > > ok, the question is because based on my config I do not expect or > want routine mail messages logged to my messages file, it is over run > by messages from courier IMAP and TPOP3 daemons. When using the built > in FreeBSD syslog these types of messages are written to the maillog > log file, and I dont understand why the same isnt true of rsyslog > given the config file I am using, > > thanks Andy. > > Quoting Rainer Gerhards: > > > I have to admit I do not really understand the question > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Oct 28 15:50:29 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 07:50:29 -0700 (PDT) Subject: [rsyslog] Property-Based Filters, not working? In-Reply-To: <001c01ca57c9$c3870810$4a951830$@com> References: <002101ca573d$5f417f70$1dc47e50$@com> <001b01ca57c2$593248a0$0b96d9e0$@com> <9B6E2A8877C38245BFB15CC491A11DA71032DD@GRFEXC.intern.adiscon.com> <001c01ca57c9$c3870810$4a951830$@com> Message-ID: On Wed, 28 Oct 2009, Jonathan Bond-Caron wrote: > On Wed Oct 28 07:35 AM, Rainer Gerhards wrote: >>> Thanks a lot, that works well. >>> >>> The docs are bit misleading: >>> http://www.rsyslog.com/doc-rsyslog_conf_filter.html >>> >>> It could show something like this: >>> :msg, contains, "error" /var/log/error.log >>> :msg, !contains, "error" ~ >>> >>> All messages that contain the word "error" are logged to >>> /var/log/error.log All messages that do not contain the word "error" >>> are thrown away. >> >> The doc in general could be much improved (any volunteers?), but here >> I think it is right. The link you quote defines what filters are, but >> what you post is not only a filter but a full selector line, >> consisting from a filter and the associated action. >> >> I agree that it would be useful to have more scenario-based cases >> which contain all the pieces put together (again, any volunteers?). > > Noted ;) > > I'll prepare a patch for 4.4.2 this week, will also fix an issue with the > syslog 'tag' parsing. It should terminate when it sees a tab or escape > control character. > > Makes it a little more RFC3164 compliant. it should also terminate parsing a hostname when it hits a tab (the RFC says it should be a space, but I've got one case (snare) that sometimes uses a tab. David Lang From david at lang.hm Wed Oct 28 15:53:32 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 07:53:32 -0700 (PDT) Subject: [rsyslog] rsyslog and snare In-Reply-To: <002301ca57cf$2cece590$86c6b0b0$@com> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> <002301ca57cf$2cece590$86c6b0b0$@com> Message-ID: On Wed, 28 Oct 2009, Jonathan Bond-Caron wrote: > On Tue Oct 27 04:32 PM, david at lang.hm wrote: >> On Tue, 27 Oct 2009, Martin Mielke wrote: >> >>> Hi, >>> >>> We recently started to receive logs from a Snare client. >>> I applied the configuration changes proposed here: >>> http://wiki.rsyslog.com/index.php/Using_Snare_as_a_client_on_Windows >> >> I just got a chance to read this, and I disagree with it's >> recommendation >> >> the #011 is a tab. >> >> snare uses it to seperate fields from the windows event log. >> >> if you just replace it with a space you loose the ability to do things >> like look for field 10 because field 8 may contain spaces. >> >> In addition there are two formats of logs that snare can output to >> syslog >> >> 1. the snare format >> 2. 'syslog' format >> >> the difference seems to be that in snare format it does >> >> Jan 1 01:01:01 >> mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri >> >> while in syslog format it does >> >> Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri >> >> > > Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric character > will terminate the TAG field and will be assumed to be the starting > character of the CONTENT field.) > > The result would be: > TAG: '' > MSG: '#011MSWinEventLog#0111#011Security#0114169#011Fri...' > > At least I think, any thoughts Rainer? > > The following works for me: > // If first character not alpha-numeric, skip tag parsing > if( !isalphanum((int)*p2parse) ) > bTAGCharDetected = 1; > > Is this bad for other loggers? if it stops on the tab it should end up hostname =vmail.abc.com then it hits a tab, so stops putting things as part of the hostname tag =vMSWinEventLog then it hits a tab, so stops putting things as part of the tag message = 1#011Security#0114169#011Fri... David Lang From david at lang.hm Wed Oct 28 15:56:48 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 07:56:48 -0700 (PDT) Subject: [rsyslog] rsyslog and snare In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local> <002301ca57cf$2cece590$86c6b0b0$@com> <9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 28 Oct 2009, Rainer Gerhards wrote: >>> Jan 1 01:01:01 >>> mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri >>> >>> while in syslog format it does >>> >>> Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri >>> >>> > > First things first: I really don't like to dig into snare, as I do not think > it is a really good solution (but how could the designer of > http://www.eventreporter.com say otherwise? ;)). > > so I assume you are talking about this message: > > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > >> Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric >> character >> will terminate the TAG field and will be assumed to be the starting >> character of the CONTENT field.) > > Another thing to put straight: RFC3164 is NOT a standard, it is an > informational document that describes things seen on the wire. And it does > not even describe all *popular* cases. If it all, it gains some standard > function via RFC3195, but this is questionable. Just for the records ;) > > So the key point here is that we have nothing to "respect" or "obey to", but > rather something that describes things seen in practice. For any decent > syslog receiver, this means it must try to work equally well with everything > that comes in via legacy syslog. But, granted, RFC3164 is useful if we have > nothing else to look at. > > So let's do that. We'll immediately see that snare is broken, because no > control characters are allowed in the hostname. So rsyslog does the right > thing and escapes these characters. So everthing belongs to the hostname and > thus you have problems with tag. two other valid behaviors 1. it has a control character in the first text field, so that cannot be a hostname or a tag, so it must be part of the message (after doing the escaping) 2. treat running into a tab like running into a space I think #1 would be better than what we do today, but #2 would be the best for users. David Lang > Damn... I didn't want to say somethign about snare. Now it has happened > again. My conclusion: fix snare or use something that works ;) > > Rainer > >> >> The result would be: >> TAG: '' >> MSG: '#011MSWinEventLog#0111#011Security#0114169#011Fri...' >> >> At least I think, any thoughts Rainer? >> >> The following works for me: >> // If first character not alpha-numeric, skip tag parsing >> if( !isalphanum((int)*p2parse) ) >> bTAGCharDetected = 1; >> >> Is this bad for other loggers? >> >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 28 16:00:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 16:00:12 +0100 Subject: [rsyslog] rsyslog and snare References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local><002301ca57cf$2cece590$86c6b0b0$@com><9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032E8@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, October 28, 2009 3:57 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog and snare > > On Wed, 28 Oct 2009, Rainer Gerhards wrote: > > >>> Jan 1 01:01:01 > >>> mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > >>> > >>> while in syslog format it does > >>> > >>> Jan 1 01:01:01 mail.abc.com > MSWinEventLog#011Security#0114169#011Fri > >>> > >>> > > > > First things first: I really don't like to dig into snare, as I do > not think > > it is a really good solution (but how could the designer of > > http://www.eventreporter.com say otherwise? ;)). > > > > so I assume you are talking about this message: > > > > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > > > >> Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric > >> character > >> will terminate the TAG field and will be assumed to be the starting > >> character of the CONTENT field.) > > > > Another thing to put straight: RFC3164 is NOT a standard, it is an > > informational document that describes things seen on the wire. And it > does > > not even describe all *popular* cases. If it all, it gains some > standard > > function via RFC3195, but this is questionable. Just for the records > ;) > > > > So the key point here is that we have nothing to "respect" or "obey > to", but > > rather something that describes things seen in practice. For any > decent > > syslog receiver, this means it must try to work equally well with > everything > > that comes in via legacy syslog. But, granted, RFC3164 is useful if > we have > > nothing else to look at. > > > > So let's do that. We'll immediately see that snare is broken, because > no > > control characters are allowed in the hostname. So rsyslog does the > right > > thing and escapes these characters. So everthing belongs to the > hostname and > > thus you have problems with tag. > > two other valid behaviors > > 1. it has a control character in the first text field, so that cannot > be a > hostname or a tag, so it must be part of the message (after doing the > escaping) > > 2. treat running into a tab like running into a space > > I think #1 would be better than what we do today, but #2 would be the > best > for users. The problem with all that is that it breaks message sanitation. When a message hits rsyslog, it first is sanitized (which is important for security reasons, e.g. to prevent NUL characters to make parts of the messages unreadable and, later, to prevent the myriad of Unicode-based Vulnerabilites). So when it hits the parser, the is no such thing like a HT present anymore. What we could do, however, is add an option that tells the sanitizer to replace HT by SP in all cases. Rainer From david at lang.hm Wed Oct 28 16:08:22 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 08:08:22 -0700 (PDT) Subject: [rsyslog] rsyslog and snare In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032E8@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local><002301ca57cf$2cece590$86c6b0b0$@com><9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71032E8@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 28 Oct 2009, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >> two other valid behaviors >> >> 1. it has a control character in the first text field, so that cannot >> be a >> hostname or a tag, so it must be part of the message (after doing the >> escaping) >> >> 2. treat running into a tab like running into a space >> >> I think #1 would be better than what we do today, but #2 would be the >> best >> for users. > > The problem with all that is that it breaks message sanitation. When a > message hits rsyslog, it first is sanitized (which is important for security > reasons, e.g. to prevent NUL characters to make parts of the messages > unreadable and, later, to prevent the myriad of Unicode-based > Vulnerabilites). > > So when it hits the parser, the is no such thing like a HT present anymore. > What we could do, however, is add an option that tells the sanitizer to > replace HT by SP in all cases. this is not as good because the tabs are useful in the message itself, it acts as a field seperator for the different fields from the windows logs, and the fields themselves can contain spaces, so if you replace them with tabs you no longer have any way to identify individual fields. the parser could look for # (which is not a valid character in a hostname, and I don't think it's valid in a tag), then if it's #011 treat is as a space for seperating the hoatname/tag/message and if it's anything else, make that the start of the message. David Lang From david at lang.hm Wed Oct 28 16:08:55 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 08:08:55 -0700 (PDT) Subject: [rsyslog] control character escaping change? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032D8@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71032D8@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 27 Oct 2009, Rainer Gerhards wrote: > This smells like a mistake. At least I don't remember changing that, but I'll > look at the changelog tomorrow... did you get a chance to look at this? David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Tuesday, October 27, 2009 9:06 PM >> To: rsyslog-users >> Subject: [rsyslog] control character escaping change? >> >> it appears that the 5.3 development series recently changed >> the default >> for escaping control characters from # to \ so a tab changed >> from #011 to >> \011 >> >> was this a deliberate change going forward, or a mistake? >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From epiphani at gmail.com Wed Oct 28 16:14:03 2009 From: epiphani at gmail.com (Aaron Wiebe) Date: Wed, 28 Oct 2009 11:14:03 -0400 Subject: [rsyslog] Queue sizing Message-ID: Greetings, I'm trying to get a good handle on queue sizing and memory usage. In a higher-traffic environment, I have a dedicated rsyslog machine with a large amount of ram. I want to try and find the right balance between large queues for burst traffic, but I also want to make sure that the queue sizes are small enough that we don't run out of memory and crash. I'm using 4.5.5 currently. I'm currently using disk-backed memory queues, configured in fixedarray, like so: $MainMsgQueueSize 2000000 $MainMsgQueueMaxFileSize 50g $MainMsgQueueSaveOnShutdown on $MainMsgQueueHighWaterMark 15000000 $MainMsgQueueLowWaterMark 10000000 $MainMsgQueueType FixedArray $MainMsgQueueWorkerThreads 2 $ActionQueueSize 200000 $ActionQueueMaxFileSize 50g $ActionQueueSaveOnShutdown on $ActionQueueHighWaterMark 1500000 $ActionQueueLowWaterMark 1000000 $ActionQueueType FixedArray $ActionQueueWorkerThreads 1 I usually have upwards of a dozen action queue definitions, and my omfile outputs are based on hostname of the incoming connection. Is there some way I can calculate maximum memory use by the queuing system, or force the system to a set memory maximum where I -know- that it will simply start pushing back on the incoming connection instead of adding more data to the system? Thoughts? -Aaron From rgerhards at hq.adiscon.com Wed Oct 28 16:25:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 16:25:27 +0100 Subject: [rsyslog] control character escaping change? References: <9B6E2A8877C38245BFB15CC491A11DA71032D8@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032EB@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, October 28, 2009 4:09 PM > To: rsyslog-users > Subject: Re: [rsyslog] control character escaping change? > > On Tue, 27 Oct 2009, Rainer Gerhards wrote: > > > This smells like a mistake. At least I don't remember changing that, > but I'll > > look at the changelog tomorrow... > > did you get a chance to look at this? yes, sorry, this is a mistake. I'll fix it soon, but today (and maybe up to the end of the week), I am working on a journal article whom's deadline *is* end of the week ;) Rainer From rgerhards at hq.adiscon.com Wed Oct 28 16:28:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 16:28:53 +0100 Subject: [rsyslog] rsyslog and snare References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local><002301ca57cf$2cece590$86c6b0b0$@com><9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71032E8@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032EC@GRFEXC.intern.adiscon.com> > > So when it hits the parser, the is no such thing like a HT present > anymore. > > What we could do, however, is add an option that tells the sanitizer > to > > replace HT by SP in all cases. > > this is not as good because the tabs are useful in the message itself, > it > acts as a field seperator for the different fields from the windows > logs, > and the fields themselves can contain spaces, so if you replace them > with > tabs you no longer have any way to identify individual fields. > > the parser could look for # (which is not a valid character in a > hostname, > and I don't think it's valid in a tag), then if it's #011 treat is as a > space for seperating the hoatname/tag/message and if it's anything > else, > make that the start of the message. Yeah, but (not only because it is snare ;)) wouldn't it be the right thing to do to fix the broken sender? Of course, I can add another exception AND "#" may validly be inside a tag. I am all for trying to guess right, but keep in mind that all this also has a performance toll. One of my plans is to add a custom parser interface. With it, custom parsers could be added and they could take care of all the anomalies of the real world (and there *are* many!). But that is up for another increment, after the queue work. I don't like to change the parsers that have proven to work well in practice just because one application misbehaves... Rainer From david at lang.hm Wed Oct 28 17:15:17 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 28 Oct 2009 09:15:17 -0700 (PDT) Subject: [rsyslog] rsyslog and snare In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032EC@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local><002301ca57cf$2cece590$86c6b0b0$@com><9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71032E8@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71032EC@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 28 Oct 2009, Rainer Gerhards wrote: >>> So when it hits the parser, the is no such thing like a HT present >> anymore. >>> What we could do, however, is add an option that tells the sanitizer >> to >>> replace HT by SP in all cases. >> >> this is not as good because the tabs are useful in the message itself, >> it >> acts as a field seperator for the different fields from the windows >> logs, >> and the fields themselves can contain spaces, so if you replace them >> with >> tabs you no longer have any way to identify individual fields. >> >> the parser could look for # (which is not a valid character in a >> hostname, >> and I don't think it's valid in a tag), then if it's #011 treat is as a >> space for seperating the hoatname/tag/message and if it's anything >> else, >> make that the start of the message. > > Yeah, but (not only because it is snare ;)) wouldn't it be the right thing to > do to fix the broken sender? yes it would, but doing so is much harder than it should be :-( David Lang > Of course, I can add another exception AND "#" > may validly be inside a tag. I am all for trying to guess right, but keep in > mind that all this also has a performance toll. > > One of my plans is to add a custom parser interface. With it, custom parsers > could be added and they could take care of all the anomalies of the real > world (and there *are* many!). But that is up for another increment, after > the queue work. > > I don't like to change the parsers that have proven to work well in practice > just because one application misbehaves... > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From a.smith at ukgrid.net Wed Oct 28 19:30:59 2009 From: a.smith at ukgrid.net (Andy Smith) Date: Wed, 28 Oct 2009 18:30:59 +0000 Subject: [rsyslog] config prob with mail.info messages In-Reply-To: <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> Message-ID: <20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> Hi Rainer, ok thanks, I turned on the custom template as described in the link you posted and could immediately see that the events were being logged as mail.notice, which strangely isnt specifically configured in the FreeBSD built in syslog conf but these are none the less logged to maillog instead of messages. Anyway, thanks to knowing this I have been easily able to add an extra line in the config to direct these to the maillog as I want :) thanks again! Andy. Quoting Rainer Gerhards: > Ah, I see. It would probably be useful to see the priority values, as > specified in the other link I posted. From anichols at trumped.org Wed Oct 28 20:00:22 2009 From: anichols at trumped.org (Aaron Nichols) Date: Wed, 28 Oct 2009 13:00:22 -0600 Subject: [rsyslog] config prob with mail.info messages In-Reply-To: <20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> <20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> Message-ID: On Wed, Oct 28, 2009 at 12:30 PM, Andy Smith wrote: > ? ok thanks, I turned on the custom template as described in the link > you posted and could immediately see that the events were being logged > as mail.notice, which strangely isnt specifically configured in the > FreeBSD built in syslog conf but these are none the less logged to > maillog instead of messages. Your original configuration contained this: *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err;local7.none /var/log/messages If you have messages being delivered to mail.notice they will match the "*.notice" selector at the beginning of that line. It's easy enough to filter them out as you've found, but I figured I'd point out that the configuration did actually specify that these should be sent to /var/log/messages. Aaron From rgerhards at hq.adiscon.com Wed Oct 28 21:02:45 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 28 Oct 2009 21:02:45 +0100 Subject: [rsyslog] config prob with mail.info messages References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net><20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net><20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71032EF@GRFEXC.intern.adiscon.com> Aaron, Good catch - I overlooked that :) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Wednesday, October 28, 2009 8:00 PM > To: rsyslog-users > Subject: Re: [rsyslog] config prob with mail.info messages > > On Wed, Oct 28, 2009 at 12:30 PM, Andy Smith > wrote: > > ? ok thanks, I turned on the custom template as described > in the link > > you posted and could immediately see that the events were > being logged > > as mail.notice, which strangely isnt specifically configured in the > > FreeBSD built in syslog conf but these are none the less logged to > > maillog instead of messages. > > Your original configuration contained this: > *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err; > local7.none > /var/log/messages > > If you have messages being delivered to mail.notice they will match > the "*.notice" selector at the beginning of that line. It's easy > enough to filter them out as you've found, but I figured I'd point out > that the configuration did actually specify that these should be sent > to /var/log/messages. > > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From a.smith at ukgrid.net Wed Oct 28 21:51:44 2009 From: a.smith at ukgrid.net (Andy Smith) Date: Wed, 28 Oct 2009 20:51:44 +0000 Subject: [rsyslog] config prob with mail.info messages In-Reply-To: <20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> References: <20091028125321.12913anl3rb8alds@horde.ukgrid.net> <20091028131741.11051ylsrkf5ub1c@horde.ukgrid.net> <20091028183059.43742ineu7fnl4o4@horde.ukgrid.net> Message-ID: <20091028205144.17377b2djm644r8c@horde.ukgrid.net> Ah yep all is clear now, thanks Aaron, well spotted as Rainer said :P From martinmie at PartyGaming.com Sat Oct 31 02:28:22 2009 From: martinmie at PartyGaming.com (Martin Mielke) Date: Sat, 31 Oct 2009 02:28:22 +0100 Subject: [rsyslog] rsyslog and snare In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> References: <0B1B9163138571439EAF804646F3F6AA0892EE1D@GIBSVWIN004X.partygaming.local><002301ca57cf$2cece590$86c6b0b0$@com> <9B6E2A8877C38245BFB15CC491A11DA71032E3@GRFEXC.intern.adiscon.com> Message-ID: <0B1B9163138571439EAF804646F3F6AA0892F6F9@GIBSVWIN004X.partygaming.local> Semi off-topic: release EventReporter under the same license as rsyslog and you'll win a lot of happy friends ;-) Cheers > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: 28 October 2009 14:13 > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog and snare > > > > Jan 1 01:01:01 > > > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > > > > > > while in syslog format it does > > > > > > Jan 1 01:01:01 mail.abc.com MSWinEventLog#011Security#0114169#011Fri > > > > > > > > First things first: I really don't like to dig into snare, as I do not > think > it is a really good solution (but how could the designer of > http://www.eventreporter.com say otherwise? ;)). > > so I assume you are talking about this message: > > mail.abc.com#011MSWinEventLog#0111#011Security#0114169#011Fri > > > Basically rsyslogd should respect RFC 3164 (Any non-alphanumeric > > character > > will terminate the TAG field and will be assumed to be the starting > > character of the CONTENT field.) > > Another thing to put straight: RFC3164 is NOT a standard, it is an > informational document that describes things seen on the wire. And it does > not even describe all *popular* cases. If it all, it gains some standard > function via RFC3195, but this is questionable. Just for the records ;) > > So the key point here is that we have nothing to "respect" or "obey to", > but > rather something that describes things seen in practice. For any decent > syslog receiver, this means it must try to work equally well with > everything > that comes in via legacy syslog. But, granted, RFC3164 is useful if we > have > nothing else to look at. > > So let's do that. We'll immediately see that snare is broken, because no > control characters are allowed in the hostname. So rsyslog does the right > thing and escapes these characters. So everthing belongs to the hostname > and > thus you have problems with tag. > > Damn... I didn't want to say somethign about snare. Now it has happened > again. My conclusion: fix snare or use something that works ;) > > Rainer > > > > > The result would be: > > TAG: '' > > MSG: '#011MSWinEventLog#0111#011Security#0114169#011Fri...' > > > > At least I think, any thoughts Rainer? > > > > The following works for me: > > // If first character not alpha-numeric, skip tag parsing > > if( !isalphanum((int)*p2parse) ) > > bTAGCharDetected = 1; > > > > Is this bad for other loggers? > > > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > /rsyslog > http://www.rsyslog.com This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk.