From iamsayan at gmail.com  Tue Jun  2 23:42:11 2009
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Tue, 2 Jun 2009 17:42:11 -0400
Subject: [rsyslog] Property replacer manipulation question
Message-ID: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com>

Hello All,

This is my first post to this group, apologize if the question seems naive.

I am trying to change strings within the syslog message before forwarding it
to an external log server. For example, if my regular expression matches an
mac or an IP address present within the message, I want to replace it with a
string like this "<hidden ip>"
Another requirement is before forwarding the log to the external server, I
have to change the facility to a different number (say from local0 to
local7)

The approach I was taking is defining a template with the regular expression
in it, checking for the pattern and then replacing the mac/ip address.

However, I think this kind of pattern replacement is not supported with the
property replacer.
Is that correct?

I am using rsyslog version 3.21 to try this out and here is an example of
what I am trying to do

A sample log message

May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE|All elements Connected

I need to change it to

May 25 08:41:13. [Hidden MAC][12703]: 4|172|CORE|All elements Connected


Before it gets forwarded to the external machine.

Regards,
Sayan


From david at lang.hm  Tue Jun  2 23:50:40 2009
From: david at lang.hm (david at lang.hm)
Date: Tue, 2 Jun 2009 14:50:40 -0700 (PDT)
Subject: [rsyslog] Property replacer manipulation question
In-Reply-To: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com>
References: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906021445440.7953@asgard>

On Tue, 2 Jun 2009, Sayan Chowdhury wrote:

> Hello All,
>
> This is my first post to this group, apologize if the question seems naive.
>
> I am trying to change strings within the syslog message before forwarding it
> to an external log server. For example, if my regular expression matches an
> mac or an IP address present within the message, I want to replace it with a
> string like this "<hidden ip>"
> Another requirement is before forwarding the log to the external server, I
> have to change the facility to a different number (say from local0 to
> local7)
>
> The approach I was taking is defining a template with the regular expression
> in it, checking for the pattern and then replacing the mac/ip address.
>
> However, I think this kind of pattern replacement is not supported with the
> property replacer.
> Is that correct?
>
> I am using rsyslog version 3.21 to try this out and here is an example of
> what I am trying to do
>
> A sample log message
>
> May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE|All elements Connected
>
> I need to change it to
>
> May 25 08:41:13. [Hidden MAC][12703]: 4|172|CORE|All elements Connected
>
>
> Before it gets forwarded to the external machine.

in this case I believe that the MAC would show up in the programname (and 
syslog tag) fields.

so instead of the default format which looks something like
%timestamp% %hostname% %syslogtag%%msg%

you would do something like
%timestamp% %hostname% [Hidden MAC]%msg%

and setup the conditions so that if it matches your condition it sends it 
out with the second format, but if it doesn't it uses the first.

this isn't the search-and-replace functionality you were thinking of, but 
I think it will do the basic job (although the example I gave doesn't 
include the PID portion of the syslog tag, doing a substring on the syslog 
tag would let you add that back in if needed)

David Lang


From iamsayan at gmail.com  Wed Jun  3 01:56:37 2009
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Tue, 2 Jun 2009 19:56:37 -0400
Subject: [rsyslog] Property replacer manipulation question
In-Reply-To: <alpine.DEB.1.10.0906021445440.7953@asgard>
References: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com>
	<alpine.DEB.1.10.0906021445440.7953@asgard>
Message-ID: <87a1ae540906021656s34696amfc3f2e327a51b4be@mail.gmail.com>

Hello David,
Thanks for replying.

I think I should have made my template clearer as well ...
Actually these mac/ip addresses appear in the message portion. with
different kind of logging message that the system generates
so my rule should work for for a message like this as well


May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE| IP 10.0.0.87 assigned to
device DEVICE_0
should become something like


May 25 08:41:13. [HIDDEN MAC][12703]: 4|172|CORE| IP [HIDDEN IP] assigned to
device DEVICE_0


also, the other requirement I have is to change the facility Id in the
outgoing syslog message.

That's when I thought about a search and replace functionality with the
property replacer ...
Not sure whether it is possible though, so far I have been able to write
property replacers which can drop line feed, or replace / with an _ but not
something like this.

Regards,
Sayan


On Tue, Jun 2, 2009 at 5:50 PM, <david at lang.hm> wrote:

> On Tue, 2 Jun 2009, Sayan Chowdhury wrote:
>
> > Hello All,
> >
> > This is my first post to this group, apologize if the question seems
> naive.
> >
> > I am trying to change strings within the syslog message before forwarding
> it
> > to an external log server. For example, if my regular expression matches
> an
> > mac or an IP address present within the message, I want to replace it
> with a
> > string like this "<hidden ip>"
> > Another requirement is before forwarding the log to the external server,
> I
> > have to change the facility to a different number (say from local0 to
> > local7)
> >
> > The approach I was taking is defining a template with the regular
> expression
> > in it, checking for the pattern and then replacing the mac/ip address.
> >
> > However, I think this kind of pattern replacement is not supported with
> the
> > property replacer.
> > Is that correct?
> >
> > I am using rsyslog version 3.21 to try this out and here is an example of
> > what I am trying to do
> >
> > A sample log message
> >
> > May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE|All elements Connected
> >
> > I need to change it to
> >
> > May 25 08:41:13. [Hidden MAC][12703]: 4|172|CORE|All elements Connected
> >
> >
> > Before it gets forwarded to the external machine.
>
> in this case I believe that the MAC would show up in the programname (and
> syslog tag) fields.
>
> so instead of the default format which looks something like
> %timestamp% %hostname% %syslogtag%%msg%
>
> you would do something like
> %timestamp% %hostname% [Hidden MAC]%msg%
>
> and setup the conditions so that if it matches your condition it sends it
> out with the second format, but if it doesn't it uses the first.
>
> this isn't the search-and-replace functionality you were thinking of, but
> I think it will do the basic job (although the example I gave doesn't
> include the PID portion of the syslog tag, doing a substring on the syslog
> tag would let you add that back in if needed)
>
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From david at lang.hm  Wed Jun  3 04:04:16 2009
From: david at lang.hm (david at lang.hm)
Date: Tue, 2 Jun 2009 19:04:16 -0700 (PDT)
Subject: [rsyslog] Property replacer manipulation question
In-Reply-To: <87a1ae540906021656s34696amfc3f2e327a51b4be@mail.gmail.com>
References: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com>
	<alpine.DEB.1.10.0906021445440.7953@asgard>
	<87a1ae540906021656s34696amfc3f2e327a51b4be@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906021901220.7953@asgard>

On Tue, 2 Jun 2009, Sayan Chowdhury wrote:

> Hello David,
> Thanks for replying.
>
> I think I should have made my template clearer as well ...
> Actually these mac/ip addresses appear in the message portion. with
> different kind of logging message that the system generates
> so my rule should work for for a message like this as well
>
>
> May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE| IP 10.0.0.87 assigned to
> device DEVICE_0
> should become something like
>
>
> May 25 08:41:13. [HIDDEN MAC][12703]: 4|172|CORE| IP [HIDDEN IP] assigned to
> device DEVICE_0
>
>
> also, the other requirement I have is to change the facility Id in the
> outgoing syslog message.

there are few enough facility Ids that you can put them into the template

> That's when I thought about a search and replace functionality with the
> property replacer ...
> Not sure whether it is possible though, so far I have been able to write
> property replacers which can drop line feed, or replace / with an _ but not
> something like this.

I haven't seen any ability to do arbatrary search-and-replace in rsyslog.

David Lang


> Regards,
> Sayan
>
>
> On Tue, Jun 2, 2009 at 5:50 PM, <david at lang.hm> wrote:
>
>> On Tue, 2 Jun 2009, Sayan Chowdhury wrote:
>>
>>> Hello All,
>>>
>>> This is my first post to this group, apologize if the question seems
>> naive.
>>>
>>> I am trying to change strings within the syslog message before forwarding
>> it
>>> to an external log server. For example, if my regular expression matches
>> an
>>> mac or an IP address present within the message, I want to replace it
>> with a
>>> string like this "<hidden ip>"
>>> Another requirement is before forwarding the log to the external server,
>> I
>>> have to change the facility to a different number (say from local0 to
>>> local7)
>>>
>>> The approach I was taking is defining a template with the regular
>> expression
>>> in it, checking for the pattern and then replacing the mac/ip address.
>>>
>>> However, I think this kind of pattern replacement is not supported with
>> the
>>> property replacer.
>>> Is that correct?
>>>
>>> I am using rsyslog version 3.21 to try this out and here is an example of
>>> what I am trying to do
>>>
>>> A sample log message
>>>
>>> May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE|All elements Connected
>>>
>>> I need to change it to
>>>
>>> May 25 08:41:13. [Hidden MAC][12703]: 4|172|CORE|All elements Connected
>>>
>>>
>>> Before it gets forwarded to the external machine.
>>
>> in this case I believe that the MAC would show up in the programname (and
>> syslog tag) fields.
>>
>> so instead of the default format which looks something like
>> %timestamp% %hostname% %syslogtag%%msg%
>>
>> you would do something like
>> %timestamp% %hostname% [Hidden MAC]%msg%
>>
>> and setup the conditions so that if it matches your condition it sends it
>> out with the second format, but if it doesn't it uses the first.
>>
>> this isn't the search-and-replace functionality you were thinking of, but
>> I think it will do the basic job (although the example I gave doesn't
>> include the PID portion of the syslog tag, doing a substring on the syslog
>> tag would let you add that back in if needed)
>>
>> David Lang
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Wed Jun  3 07:36:49 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Jun 2009 07:36:49 +0200
Subject: [rsyslog] Property replacer manipulation question
References: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com><alpine.DEB.1.10.0906021445440.7953@asgard><87a1ae540906021656s34696amfc3f2e327a51b4be@mail.gmail.com>
	<alpine.DEB.1.10.0906021901220.7953@asgard>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F853@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Wednesday, June 03, 2009 4:04 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Property replacer manipulation question
> 
> On Tue, 2 Jun 2009, Sayan Chowdhury wrote:
> 
> > Hello David,
> > Thanks for replying.
> >
> > I think I should have made my template clearer as well ...
> > Actually these mac/ip addresses appear in the message portion. with
> > different kind of logging message that the system generates
> > so my rule should work for for a message like this as well
> >
> >
> > May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE| IP 10.0.0.87 assigned
to
> > device DEVICE_0
> > should become something like
> >
> >
> > May 25 08:41:13. [HIDDEN MAC][12703]: 4|172|CORE| IP [HIDDEN IP] assigned
to
> > device DEVICE_0
> >
> >
> > also, the other requirement I have is to change the facility Id in the
> > outgoing syslog message.
> 
> there are few enough facility Ids that you can put them into the template
> 
> > That's when I thought about a search and replace functionality with the
> > property replacer ...
> > Not sure whether it is possible though, so far I have been able to write
> > property replacers which can drop line feed, or replace / with an _ but
not
> > something like this.
> 
> I haven't seen any ability to do arbatrary search-and-replace in rsyslog.

This is not available, but sounds like a good idea. I suggest to add a
feature enhancement ticket to the bug tracker. Be warned, though, that I
currently simply have no time to implement it. But it may happen as a
by-activity when I change things in the property replacer, so it would be
good not to forget about it.

Rainer


From rgerhards at hq.adiscon.com  Wed Jun  3 18:33:19 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Jun 2009 18:33:19 +0200
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>

Hi all,

I have begun to redesign the pretty old omfile output plugin. Large parts of
this work are covered by a sponsorship.

I will probably post a number of questions during the redesign process.
Please speak up if you have any opinion on them. Once I have finished the new
design, it is not likely to change during the next three to five years. For
the same reason, please speak up now if you are desperately missing something
in omfile.

A concrete question I have is on the $OutChannel directive[1]. It is
scheduled for replacement for quite a while, and now may be the time that it
finally goes away. If you use $OutChannel, please let me know why.

The most prominent use case is to size-limit files. $OutChannel provides the
ability to execute a command when a certain file size is reached. The command
may then rotate the file. After command execution, omfile checks the file
size again. If it still is above the configured limit, this file will be
disabled (no longer be written to until restart).

If you use this for size-limiting, please let me know how you do it. Please
also let me know if you would like to see things handled differently.

Looking forward to your feedback.

Thanks,
Rainer


[1] http://www.rsyslog.com/doc-rsyslog_conf_output.html


From iamsayan at gmail.com  Wed Jun  3 19:28:24 2009
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Wed, 3 Jun 2009 13:28:24 -0400
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
Message-ID: <87a1ae540906031028s3584fbdat428ca35a17f638ef@mail.gmail.com>

Hello Rainer,

I do use this facility. This is how I do it

$outchannel log_rotation_16,/var/private/log/logsip,
10000000,/opt/platform/bin/log_rotate.pl

if $syslogfacility == '16' and $syslogseverity <= '6' then
$log_rotation_16;precise

The log_rotate.pl perl script that I have rotates the files
and maintains a set of 10 files. When the current file reaches the 10 mb
limit, the older file is zipped and copied over to a different directory.
I am on release 3.21 now

I would definitely like to have the feature be present in the future
releases as well. I am happy as long as this feature of file size based
rotation is there and I can migrate my rules/scripts to whatever new
mechanism that get implemented.

Regards,
Sayan






On Wed, Jun 3, 2009 at 12:33 PM, Rainer Gerhards
<rgerhards at hq.adiscon.com>wrote:

> Hi all,
>
> I have begun to redesign the pretty old omfile output plugin. Large parts
> of
> this work are covered by a sponsorship.
>
> I will probably post a number of questions during the redesign process.
> Please speak up if you have any opinion on them. Once I have finished the
> new
> design, it is not likely to change during the next three to five years. For
> the same reason, please speak up now if you are desperately missing
> something
> in omfile.
>
> A concrete question I have is on the $OutChannel directive[1]. It is
> scheduled for replacement for quite a while, and now may be the time that
> it
> finally goes away. If you use $OutChannel, please let me know why.
>
> The most prominent use case is to size-limit files. $OutChannel provides
> the
> ability to execute a command when a certain file size is reached. The
> command
> may then rotate the file. After command execution, omfile checks the file
> size again. If it still is above the configured limit, this file will be
> disabled (no longer be written to until restart).
>
> If you use this for size-limiting, please let me know how you do it. Please
> also let me know if you would like to see things handled differently.
>
> Looking forward to your feedback.
>
> Thanks,
> Rainer
>
>
> [1] http://www.rsyslog.com/doc-rsyslog_conf_output.html
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Thu Jun  4 11:01:46 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 4 Jun 2009 11:01:46 +0200
Subject: [rsyslog] request for feedback: dynafile cache size
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F879@GRFEXC.intern.adiscon.com>

Hi all,

as you probably know, dynafiles keep a cache of the n most recently used
files open. This is done in order to speed up processing. I wonder how many
files you usually expect to be open. The root of my question is that I think
about the algorithm that finds cache entries. So far, a simple linked list is
used, but I wonder if something more efficient (like a binary tree) would be
useful. On the other hand, if the cache size is really small, there is not
much difference between O(n) and O(log n), so it may not justify added code
complexity.

Feedback appreciated.

Rainer


From david at lang.hm  Thu Jun  4 13:50:42 2009
From: david at lang.hm (david at lang.hm)
Date: Thu, 4 Jun 2009 04:50:42 -0700 (PDT)
Subject: [rsyslog] request for feedback: dynafile cache size
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F879@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F879@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906040448240.11064@asgard>

On Thu, 4 Jun 2009, Rainer Gerhards wrote:

> Hi all,
>
> as you probably know, dynafiles keep a cache of the n most recently used
> files open. This is done in order to speed up processing. I wonder how many
> files you usually expect to be open. The root of my question is that I think
> about the algorithm that finds cache entries. So far, a simple linked list is
> used, but I wonder if something more efficient (like a binary tree) would be
> useful. On the other hand, if the cache size is really small, there is not
> much difference between O(n) and O(log n), so it may not justify added code
> complexity.
>
> Feedback appreciated.

I have a use case where it could be several thousand files

I would expect the normal worst case to be dozens to a few hundred files.

David Lang


From rgerhards at hq.adiscon.com  Thu Jun  4 13:52:18 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 4 Jun 2009 13:52:18 +0200
Subject: [rsyslog] request for feedback: dynafile cache size
References: <9B6E2A8877C38245BFB15CC491A11DA706F879@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906040448240.11064@asgard>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F888@GRFEXC.intern.adiscon.com>

Thanks for the feedback. That definitely asks for a binary tree search. So I
am convinced now (I'll not do it immediately, but probably towards the end of
my redesign work).

Raienr

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Thursday, June 04, 2009 1:51 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] request for feedback: dynafile cache size
> 
> On Thu, 4 Jun 2009, Rainer Gerhards wrote:
> 
> > Hi all,
> >
> > as you probably know, dynafiles keep a cache of the n most recently used
> > files open. This is done in order to speed up processing. I wonder how
many
> > files you usually expect to be open. The root of my question is that I
think
> > about the algorithm that finds cache entries. So far, a simple linked
list
> is
> > used, but I wonder if something more efficient (like a binary tree) would
be
> > useful. On the other hand, if the cache size is really small, there is
not
> > much difference between O(n) and O(log n), so it may not justify added
code
> > complexity.
> >
> > Feedback appreciated.
> 
> I have a use case where it could be several thousand files
> 
> I would expect the normal worst case to be dozens to a few hundred files.
> 
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From nathan at gossamer-threads.com  Sat Jun  6 00:09:16 2009
From: nathan at gossamer-threads.com (Nathan March)
Date: Fri, 05 Jun 2009 15:09:16 -0700
Subject: [rsyslog] Possible bug? 100% cpu on a server thread and
 the	client stops sending data
In-Reply-To: <4A1A57A0.1060903@net-m.de>
References: <4A15999D.3000909@gossamer-threads.com>
	<4A1A37DE.8050004@net-m.de>	<9B6E2A8877C38245BFB15CC491A11DA702B0FB@GRFEXC.intern.adiscon.com>
	<4A1A57A0.1060903@net-m.de>
Message-ID: <4A29978C.9080505@gossamer-threads.com>

Any hope of this bug getting some attention soon? =)

Thanks,
Nathan


Patrick Shen wrote:
> Hi Rainer,
>
> Thanks for the quick reply.
>
> My environment is like below:
>
> CLIENT ---
>          |
>          | (TCP)
>          --------> SERVER ----
>                              |
>                              | (RELP Relay)
>                              --------------> SERVER Standby
>
> I've tested, even if the standby server is under high load, one of our
> clients won't accept any new network connections.
>
> Just for your information.
>
> Best regards,
> Patrick
>
> Rainer Gerhards wrote:
>   
>> Hi all,
>>
>> thanks for the detailed info. Will look at it asap, but I am currently tied
>> up with some other work...
>>
>> Please let me know anything else that may be relevant.
>>
>> Thanks,
>> Rainer
>>
>>     
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of Patrick Shen
>>> Sent: Monday, May 25, 2009 8:17 AM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server thread and the
>>> client stops sending data
>>>
>>> Hi Nathan,
>>>
>>> In my company, we have the similar symptom. We have 50+ servers which
>>> send logs to the central logging servers. Both clients and server use
>>> rsyslog v3.20.5 right now. In the past when we use v3.20.0, the symptom
>>> is more common when server is under high load average.
>>>
>>> It's also weird for us, not all of clients were lost of connections. We
>>> have 2-3 clients which is running the same application will looks like
>>> hanging up when server is high-load. New ssh connection is very slow at
>>> that time. But the connected ssh session works normal (We have some
>>> prepared screen session on other clients which connect to them via ssh).
>>>
>>> When the clients are lost of connections. So far our solution is to
>>> restart rsyslog both on affected clients and server. Then the clients
>>> get back and look normal.
>>>
>>> We've suffered it for a while, but I didn't take chance to analyze it
>>> like you.
>>>
>>> Best regards,
>>> Patrick
>>>
>>> Nathan March wrote:
>>>       
>>>> Having a weird issue here..... Testing out a new rsyslog deployment and
>>>> I've got around 30 servers logging to one machine. On one of the clients
>>>> we had an issue where ssh was mysteriously very slow to login and
>>>> tracked it down to rsyslog causing issues. Simultaneously, one of the
>>>> threads on the logging server spiked up to 100%. New log data from the
>>>> client would stop showing up on the server at this poitn.
>>>>
>>>> On the client after attempting to gracefully stop rsyslog:
>>>>
>>>> vanguard etc # ps aux | grep -i rsyslog
>>>> root      3764  0.0  0.1  5092 2564 ?        S    15:06   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f /etc/rsyslog.conf
>>>> root      3766  0.0  0.1  5092 2564 ?        S    15:06   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f /etc/rsyslog.conf
>>>> root      6203  0.0  0.0  1512  524 pts/3    S    15:09   0:00 grep -i
>>>> rsyslog
>>>> vanguard etc # strace -p 3764
>>>> Process 3764 attached - interrupt to quit
>>>> send(11, "\25\3\2\1\0uE\254\304\n\311\377\204}p\237O\211\322\211"...,
>>>> 261, 0 <unfinished ...>
>>>> Process 3764 detached
>>>>
>>>> FD 11 being the socket to the logging server, it never does anything
>>>> asides from that send.
>>>>
>>>> On the server:
>>>>
>>>> ldap ~ # ps aux | grep rsyslog
>>>> root      7370  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7379  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7380  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7381  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7382  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7383  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7384  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7385 28.3  0.0  83936  3380 ?        R    14:54   5:07
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> Stracing the 7385 pid just shows this repeating over and over (FD 4
>>>> being the network socket again):
>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 EAGAIN (Resource
>>>> temporarily unavailable)
>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 EAGAIN (Resource
>>>> temporarily unavailable)
>>>>
>>>> It happened 3 times within a relatively short period of time, sometimes
>>>> within minutes of me restarting rsyslog on both machines. I enabled
>>>> debugging and it took around 8 hours to pop up again.
>>>>
>>>> With debugging, I pulled this from the rsyslog server:
>>>>
>>>> 6640.408108000:imtcp.c: New connect on NSD 0x59ddc0.
>>>> 6640.408129000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: pszRcvBuf (nil),
>>>> lenRcvBuf 0
>>>> 6640.408144000:imtcp.c: hasRcvInBuffer on nsd 0x59dcd0: pszRcvBuf
>>>> 0x5c11c0, lenRcvBuf -1
>>>> 6640.408158000:imtcp.c: hasRcvInBuffer on nsd 0x59dc10: pszRcvBuf
>>>> 0x5cb510, lenRcvBuf -1
>>>> 6640.408171000:imtcp.c: hasRcvInBuffer on nsd 0x5c3d70: pszRcvBuf
>>>> 0x5d45c0, lenRcvBuf -1
>>>> 6640.408185000:imtcp.c: hasRcvInBuffer on nsd 0x5cf460: pszRcvBuf
>>>> 0x5dd010, lenRcvBuf -1
>>>> 6640.408199000:imtcp.c: hasRcvInBuffer on nsd 0x5d8600: pszRcvBuf
>>>> 0x5e58b0, lenRcvBuf -1
>>>> 6640.408213000:imtcp.c: hasRcvInBuffer on nsd 0x59d690: pszRcvBuf
>>>> 0x5ee150, lenRcvBuf -1
>>>> 6640.408227000:imtcp.c: hasRcvInBuffer on nsd 0x59d2f0: pszRcvBuf
>>>> 0x5f7320, lenRcvBuf -1
>>>> 6640.408241000:imtcp.c: hasRcvInBuffer on nsd 0x5e8840: pszRcvBuf
>>>> 0x5fe0a0, lenRcvBuf -1
>>>> 6640.408255000:imtcp.c: hasRcvInBuffer on nsd 0x6027b0: pszRcvBuf
>>>> 0x608350, lenRcvBuf -1
>>>> 6640.408269000:imtcp.c: hasRcvInBuffer on nsd 0x5c3660: pszRcvBuf
>>>> 0x612110, lenRcvBuf -1
>>>> 6640.408283000:imtcp.c: hasRcvInBuffer on nsd 0x602690: pszRcvBuf
>>>> 0x615040, lenRcvBuf -1
>>>> 6640.408296000:imtcp.c: hasRcvInBuffer on nsd 0x5c3b60: pszRcvBuf
>>>> 0x6218f0, lenRcvBuf -1
>>>> 6640.408310000:imtcp.c: hasRcvInBuffer on nsd 0x60c150: pszRcvBuf
>>>> 0x62a190, lenRcvBuf -1
>>>> 6640.408324000:imtcp.c: hasRcvInBuffer on nsd 0x625420: pszRcvBuf
>>>> 0x634fe0, lenRcvBuf -1
>>>> 6640.408338000:imtcp.c: hasRcvInBuffer on nsd 0x6117b0: pszRcvBuf
>>>> 0x63b000, lenRcvBuf -1
>>>> 6640.408352000:imtcp.c: hasRcvInBuffer on nsd 0x603990: pszRcvBuf
>>>> 0x643c70, lenRcvBuf -1
>>>> 6640.408365000:imtcp.c: hasRcvInBuffer on nsd 0x633c00: pszRcvBuf
>>>> 0x64dbb0, lenRcvBuf -1
>>>> 6640.408379000:imtcp.c: hasRcvInBuffer on nsd 0x62cfa0: pszRcvBuf
>>>> 0x650960, lenRcvBuf -1
>>>> 6640.408393000:imtcp.c: hasRcvInBuffer on nsd 0x633d10: pszRcvBuf
>>>> 0x65dbc0, lenRcvBuf -1
>>>> 6640.408407000:imtcp.c: hasRcvInBuffer on nsd 0x5f2860: pszRcvBuf
>>>> 0x666020, lenRcvBuf -1
>>>> 6640.408421000:imtcp.c: hasRcvInBuffer on nsd 0x64d250: pszRcvBuf
>>>> 0x66e480, lenRcvBuf -1
>>>> 6640.408435000:imtcp.c: hasRcvInBuffer on nsd 0x659b30: pszRcvBuf
>>>> 0x676d20, lenRcvBuf 78
>>>> 6640.408448000:imtcp.c: hasRcvInBuffer on nsd 0x669fb0: pszRcvBuf
>>>> 0x67fdd0, lenRcvBuf -1
>>>> 6640.408462000:imtcp.c: hasRcvInBuffer on nsd 0x59a630: pszRcvBuf
>>>> 0x687f10, lenRcvBuf -1
>>>> 6640.408476000:imtcp.c: hasRcvInBuffer on nsd 0x6614c0: pszRcvBuf
>>>> 0x6907b0, lenRcvBuf -1
>>>> 6640.408490000:imtcp.c: hasRcvInBuffer on nsd 0x60c2d0: pszRcvBuf
>>>> 0x699050, lenRcvBuf -1
>>>> 6640.408504000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: pszRcvBuf (nil),
>>>> lenRcvBuf 0
>>>>
>>>> This repeats enough to generate around 29 million lines in the debug log
>>>> file.
>>>>
>>>> In the client log, there's plenty of these which seems somewhat normal:
>>>>
>>>> 6509.743164000:imuxsock.c: --------imuxsock calling select, active file
>>>> descriptors (max 11): 11
>>>> 6509.743231000:imuxsock.c: Message from UNIX socket: #11
>>>> 6509.743262000:imuxsock.c: logmsg: flags 4, from 'vanguard', msg May 20
>>>> 23:15:09 bin/qmail-local[23198]: using .qmail .qmail-cfsc-forum
>>>> 6509.743281000:imuxsock.c: Message has legacy syslog format.
>>>> 6509.743302000:imuxsock.c: main queue: entry added, size now 579 entries
>>>> 6509.743325000:imuxsock.c: wtpAdviseMaxWorkers signals busy
>>>> 6509.743343000:imuxsock.c: main queue: EnqueueMsg advised worker start
>>>>
>>>> However comparing it to an earlier point in the log  there's none of the
>>>> associated tcp queue entries like this:
>>>>
>>>> 3587.234178000:main queue:Reg/w0: TCP sent 107 bytes, requested 107
>>>> 3587.234204000:main queue:Reg/w0: main queue: entering rate limiter
>>>> 3587.234229000:main queue:Reg/w0: main queue: entry deleted, state 0,
>>>> size now 0 entries
>>>> 3587.234255000:main queue:Reg/w0: Called action, logging to builtin-file
>>>> 3587.234285000:main queue:Reg/w0:  (/var/log/maillog)
>>>> 3587.234314000:main queue:Reg/w0: Called action, logging to builtin-fwd
>>>> 3587.234340000:main queue:Reg/w0:  ldap.nmsrv.com
>>>> 3587.234359000:main queue:Reg/w0:  ldap.nmsrv.com:10514/tcp
>>>> 3587.234425000:main queue:Reg/w0: TCP sent 78 bytes, requested 78
>>>> 3587.234446000:main queue:Reg/w0: main queue: entering rate limiter
>>>> 3587.234471000:main queue:Reg/w0: main queue:Reg/w0: worker IDLE,
>>>> waiting for work.
>>>>
>>>> Anyone able to shed some light on this situation? I've got the full
>>>> debug logs if it's useful to anyone to take a closer look.
>>>>
>>>> Thanks!
>>>>
>>>> - Nathan
>>>>
>>>>
>>>>
>>>>         
>> ----------------------------------------------------------------------------
>>     
>>> --------
>>>       
>>>> Server config file
>>>>
>>>>         
>> ----------------------------------------------------------------------------
>>     
>>> --------
>>>       
>>>> $ModLoad immark.so # provides --MARK-- message capability
>>>> $ModLoad imuxsock.so # provides support for local system logging (e.g.
>>>> via logger command)
>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
>>>>
>>>> $umask 0137
>>>> $DirCreateMode 0640
>>>> $FileCreateMode 0640
>>>> $FileOwner root
>>>> $FileGroup admin
>>>>
>>>> $template DynFile,"/var/log/rsyslog/%HOSTNAME%/messages"
>>>> $template DynFileCron,"/var/log/rsyslog/%HOSTNAME%/cron"
>>>> $template DynFileSecure,"/var/log/rsyslog/%HOSTNAME%/secure"
>>>> $template DynFileMail,"/var/log/rsyslog/%HOSTNAME%/maillog"
>>>> $template DynFileSpool,"/var/log/rsyslog/%HOSTNAME%/spooler"
>>>> $template DynFileBoot,"/var/log/rsyslog/%HOSTNAME%/boot"
>>>> $template DynFileSyslog,"/var/log/rsyslog/%HOSTNAME%/syslog"
>>>>
>>>> :msg, contains, "no keys found for"            ~
>>>> :msg, contains, "session opened for user"        ~
>>>>
>>>> # Log all kernel messages to the console.
>>>> # Logging much else clutters up the screen.
>>>> kern.*                                                 /dev/console
>>>>
>>>> # Log anything (except mail) of level info or higher.
>>>> # Don't log private authentication messages!
>>>> *.info;mail.none;cron.none                     ?DynFile
>>>>
>>>> *.warn;\
>>>>         authpriv.none;cron.none;mail.none;news.none     ?DynFileSyslog
>>>>
>>>> # Log all the mail messages in one place.
>>>> mail.*                                                  ?DynFileMail
>>>>
>>>> # Log cron stuff
>>>> cron.*                                                  ?DynFileCron
>>>>
>>>> # Everybody gets emergency messages
>>>> *.emerg                                                 *
>>>>
>>>> # Save news errors of level crit and higher in a special file
>>>> uucp,news.crit                                          ?DynFileSpool
>>>>
>>>> # Save boot messages also to boot.log
>>>> local7.*                                                ?DynFileBoot
>>>>
>>>> # Relp config for PCI
>>>> $ModLoad imrelp
>>>> $InputRELPServerRun 2515
>>>>
>>>> # make gtls driver the default
>>>> $DefaultNetstreamDriver gtls
>>>>
>>>> # certificate files
>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
>>>> $DefaultNetstreamDriverCertFile /etc/ssl/servercrt.pem
>>>> $DefaultNetstreamDriverKeyFile /etc/ssl/serverkey.pem
>>>>
>>>> $ModLoad imtcp.so
>>>>
>>>> $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
>>>> $InputTCPServerStreamDriverAuthMode anon # client is NOT authenticated
>>>> $InputTCPServerRun 10514 # start up listener at port 10514
>>>> $InputTCPMaxSessions 500
>>>>
>>>>
>>>>         
>> ----------------------------------------------------------------------------
>>     
>>> --------
>>>       
>>>> Client config file
>>>>
>>>>         
>> ----------------------------------------------------------------------------
>>     
>>> --------
>>>       
>>>> $ModLoad imuxsock.so # provides support for local system logging (e.g.
>>>> via logger command)
>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
>>>>
>>>> $umask 0137
>>>> $DirCreateMode 0640
>>>> $FileCreateMode 0640
>>>> $FileOwner root
>>>> $FileGroup admin
>>>>
>>>>
>>>> # Log anything 'info' or higher, but lower than 'warn'.
>>>> # Exclude authpriv, cron, mail, and news.  These are logged elsewhere.
>>>> *.info;*.!warn;\
>>>>         authpriv.none;cron.none;mail.none;news.none     /var/log/messages
>>>>
>>>> # Log anything 'warn' or higher.
>>>> # Exclude authpriv, cron, mail, and news.  These are logged elsewhere.
>>>> *.warn;\
>>>>         authpriv.none;cron.none;mail.none;news.none     /var/log/syslog
>>>>
>>>> # Debugging information is logged here.
>>>> *.=debug                                                /var/log/debug
>>>>
>>>> # Private authentication message logging:
>>>> authpriv.*                                              /var/log/secure
>>>>
>>>> # Cron related logs:
>>>> cron.*                                                  /var/log/cron
>>>>
>>>> # Mail related logs:
>>>> mail.*                                                  /var/log/maillog
>>>>
>>>> # Emergency level messages go to all users:
>>>> *.emerg                                                 *
>>>>
>>>>
>>>> ## Uncomment these lines to use RELP instead for PCI compliance (stunnel
>>>> required)
>>>> #$ModLoad omrelp
>>>> #*.*;mail.none :omrelp:localhost:2515;RSYSLOG_ForwardFormat
>>>>
>>>> # certificate files - just CA for a client
>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
>>>>
>>>> # set up the action
>>>> $DefaultNetstreamDriver gtls # use gtls netstream driver
>>>> $ActionSendStreamDriverMode 1 # require TLS for the connection
>>>> $ActionSendStreamDriverAuthMode anon # server is NOT authenticated
>>>> *.* @@(o)ldap.nmsrv.com:10514 # send (all) messages
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.comoften
>>>>         
>>> --
>>> Patrick Shen
>>> Operations Engineer
>>>
>>> net mobile AG Shanghai office
>>>
>>> T: +86  21 61350900 - 222
>>> F: +86  21 61350906
>>> M: +86  13122245730
>>>
>>> FE55 6A7F 3192 F359 C1DF  12F6 9078 58B4 57AF 0BE3
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>       
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>     
>
>
>   


From rgerhards at hq.adiscon.com  Sun Jun  7 20:46:54 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 7 Jun 2009 20:46:54 +0200
Subject: [rsyslog] Possible bug? 100% cpu on a server thread and
	the	client stops sending data
References: <4A15999D.3000909@gossamer-threads.com><4A1A37DE.8050004@net-m.de>	<9B6E2A8877C38245BFB15CC491A11DA702B0FB@GRFEXC.intern.adiscon.com><4A1A57A0.1060903@net-m.de>
	<4A29978C.9080505@gossamer-threads.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8BA@GRFEXC.intern.adiscon.com>

Please give v4 a try, the bug is potentially fixed there. If it actually is,
this gives me a clue of where to look at.

Rainer 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Nathan March
> Sent: Saturday, June 06, 2009 12:09 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server 
> thread and the client stops sending data
> 
> Any hope of this bug getting some attention soon? =)
> 
> Thanks,
> Nathan
> 
> 
> Patrick Shen wrote:
> > Hi Rainer,
> >
> > Thanks for the quick reply.
> >
> > My environment is like below:
> >
> > CLIENT ---
> >          |
> >          | (TCP)
> >          --------> SERVER ----
> >                              |
> >                              | (RELP Relay)
> >                              --------------> SERVER Standby
> >
> > I've tested, even if the standby server is under high load, 
> one of our
> > clients won't accept any new network connections.
> >
> > Just for your information.
> >
> > Best regards,
> > Patrick
> >
> > Rainer Gerhards wrote:
> >   
> >> Hi all,
> >>
> >> thanks for the detailed info. Will look at it asap, but I 
> am currently tied
> >> up with some other work...
> >>
> >> Please let me know anything else that may be relevant.
> >>
> >> Thanks,
> >> Rainer
> >>
> >>     
> >>> -----Original Message-----
> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>> bounces at lists.adiscon.com] On Behalf Of Patrick Shen
> >>> Sent: Monday, May 25, 2009 8:17 AM
> >>> To: rsyslog-users
> >>> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server 
> thread and the
> >>> client stops sending data
> >>>
> >>> Hi Nathan,
> >>>
> >>> In my company, we have the similar symptom. We have 50+ 
> servers which
> >>> send logs to the central logging servers. Both clients 
> and server use
> >>> rsyslog v3.20.5 right now. In the past when we use 
> v3.20.0, the symptom
> >>> is more common when server is under high load average.
> >>>
> >>> It's also weird for us, not all of clients were lost of 
> connections. We
> >>> have 2-3 clients which is running the same application 
> will looks like
> >>> hanging up when server is high-load. New ssh connection 
> is very slow at
> >>> that time. But the connected ssh session works normal (We 
> have some
> >>> prepared screen session on other clients which connect to 
> them via ssh).
> >>>
> >>> When the clients are lost of connections. So far our 
> solution is to
> >>> restart rsyslog both on affected clients and server. Then 
> the clients
> >>> get back and look normal.
> >>>
> >>> We've suffered it for a while, but I didn't take chance 
> to analyze it
> >>> like you.
> >>>
> >>> Best regards,
> >>> Patrick
> >>>
> >>> Nathan March wrote:
> >>>       
> >>>> Having a weird issue here..... Testing out a new rsyslog 
> deployment and
> >>>> I've got around 30 servers logging to one machine. On 
> one of the clients
> >>>> we had an issue where ssh was mysteriously very slow to login and
> >>>> tracked it down to rsyslog causing issues. 
> Simultaneously, one of the
> >>>> threads on the logging server spiked up to 100%. New log 
> data from the
> >>>> client would stop showing up on the server at this poitn.
> >>>>
> >>>> On the client after attempting to gracefully stop rsyslog:
> >>>>
> >>>> vanguard etc # ps aux | grep -i rsyslog
> >>>> root      3764  0.0  0.1  5092 2564 ?        S    15:06   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f 
> /etc/rsyslog.conf
> >>>> root      3766  0.0  0.1  5092 2564 ?        S    15:06   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f 
> /etc/rsyslog.conf
> >>>> root      6203  0.0  0.0  1512  524 pts/3    S    15:09  
>  0:00 grep -i
> >>>> rsyslog
> >>>> vanguard etc # strace -p 3764
> >>>> Process 3764 attached - interrupt to quit
> >>>> send(11, 
> "\25\3\2\1\0uE\254\304\n\311\377\204}p\237O\211\322\211"...,
> >>>> 261, 0 <unfinished ...>
> >>>> Process 3764 detached
> >>>>
> >>>> FD 11 being the socket to the logging server, it never 
> does anything
> >>>> asides from that send.
> >>>>
> >>>> On the server:
> >>>>
> >>>> ldap ~ # ps aux | grep rsyslog
> >>>> root      7370  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7379  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7380  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7381  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7382  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7383  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7384  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7385 28.3  0.0  83936  3380 ?        R    14:54   5:07
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> Stracing the 7385 pid just shows this repeating over and 
> over (FD 4
> >>>> being the network socket again):
> >>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 
> EAGAIN (Resource
> >>>> temporarily unavailable)
> >>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 
> EAGAIN (Resource
> >>>> temporarily unavailable)
> >>>>
> >>>> It happened 3 times within a relatively short period of 
> time, sometimes
> >>>> within minutes of me restarting rsyslog on both 
> machines. I enabled
> >>>> debugging and it took around 8 hours to pop up again.
> >>>>
> >>>> With debugging, I pulled this from the rsyslog server:
> >>>>
> >>>> 6640.408108000:imtcp.c: New connect on NSD 0x59ddc0.
> >>>> 6640.408129000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: 
> pszRcvBuf (nil),
> >>>> lenRcvBuf 0
> >>>> 6640.408144000:imtcp.c: hasRcvInBuffer on nsd 0x59dcd0: pszRcvBuf
> >>>> 0x5c11c0, lenRcvBuf -1
> >>>> 6640.408158000:imtcp.c: hasRcvInBuffer on nsd 0x59dc10: pszRcvBuf
> >>>> 0x5cb510, lenRcvBuf -1
> >>>> 6640.408171000:imtcp.c: hasRcvInBuffer on nsd 0x5c3d70: pszRcvBuf
> >>>> 0x5d45c0, lenRcvBuf -1
> >>>> 6640.408185000:imtcp.c: hasRcvInBuffer on nsd 0x5cf460: pszRcvBuf
> >>>> 0x5dd010, lenRcvBuf -1
> >>>> 6640.408199000:imtcp.c: hasRcvInBuffer on nsd 0x5d8600: pszRcvBuf
> >>>> 0x5e58b0, lenRcvBuf -1
> >>>> 6640.408213000:imtcp.c: hasRcvInBuffer on nsd 0x59d690: pszRcvBuf
> >>>> 0x5ee150, lenRcvBuf -1
> >>>> 6640.408227000:imtcp.c: hasRcvInBuffer on nsd 0x59d2f0: pszRcvBuf
> >>>> 0x5f7320, lenRcvBuf -1
> >>>> 6640.408241000:imtcp.c: hasRcvInBuffer on nsd 0x5e8840: pszRcvBuf
> >>>> 0x5fe0a0, lenRcvBuf -1
> >>>> 6640.408255000:imtcp.c: hasRcvInBuffer on nsd 0x6027b0: pszRcvBuf
> >>>> 0x608350, lenRcvBuf -1
> >>>> 6640.408269000:imtcp.c: hasRcvInBuffer on nsd 0x5c3660: pszRcvBuf
> >>>> 0x612110, lenRcvBuf -1
> >>>> 6640.408283000:imtcp.c: hasRcvInBuffer on nsd 0x602690: pszRcvBuf
> >>>> 0x615040, lenRcvBuf -1
> >>>> 6640.408296000:imtcp.c: hasRcvInBuffer on nsd 0x5c3b60: pszRcvBuf
> >>>> 0x6218f0, lenRcvBuf -1
> >>>> 6640.408310000:imtcp.c: hasRcvInBuffer on nsd 0x60c150: pszRcvBuf
> >>>> 0x62a190, lenRcvBuf -1
> >>>> 6640.408324000:imtcp.c: hasRcvInBuffer on nsd 0x625420: pszRcvBuf
> >>>> 0x634fe0, lenRcvBuf -1
> >>>> 6640.408338000:imtcp.c: hasRcvInBuffer on nsd 0x6117b0: pszRcvBuf
> >>>> 0x63b000, lenRcvBuf -1
> >>>> 6640.408352000:imtcp.c: hasRcvInBuffer on nsd 0x603990: pszRcvBuf
> >>>> 0x643c70, lenRcvBuf -1
> >>>> 6640.408365000:imtcp.c: hasRcvInBuffer on nsd 0x633c00: pszRcvBuf
> >>>> 0x64dbb0, lenRcvBuf -1
> >>>> 6640.408379000:imtcp.c: hasRcvInBuffer on nsd 0x62cfa0: pszRcvBuf
> >>>> 0x650960, lenRcvBuf -1
> >>>> 6640.408393000:imtcp.c: hasRcvInBuffer on nsd 0x633d10: pszRcvBuf
> >>>> 0x65dbc0, lenRcvBuf -1
> >>>> 6640.408407000:imtcp.c: hasRcvInBuffer on nsd 0x5f2860: pszRcvBuf
> >>>> 0x666020, lenRcvBuf -1
> >>>> 6640.408421000:imtcp.c: hasRcvInBuffer on nsd 0x64d250: pszRcvBuf
> >>>> 0x66e480, lenRcvBuf -1
> >>>> 6640.408435000:imtcp.c: hasRcvInBuffer on nsd 0x659b30: pszRcvBuf
> >>>> 0x676d20, lenRcvBuf 78
> >>>> 6640.408448000:imtcp.c: hasRcvInBuffer on nsd 0x669fb0: pszRcvBuf
> >>>> 0x67fdd0, lenRcvBuf -1
> >>>> 6640.408462000:imtcp.c: hasRcvInBuffer on nsd 0x59a630: pszRcvBuf
> >>>> 0x687f10, lenRcvBuf -1
> >>>> 6640.408476000:imtcp.c: hasRcvInBuffer on nsd 0x6614c0: pszRcvBuf
> >>>> 0x6907b0, lenRcvBuf -1
> >>>> 6640.408490000:imtcp.c: hasRcvInBuffer on nsd 0x60c2d0: pszRcvBuf
> >>>> 0x699050, lenRcvBuf -1
> >>>> 6640.408504000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: 
> pszRcvBuf (nil),
> >>>> lenRcvBuf 0
> >>>>
> >>>> This repeats enough to generate around 29 million lines 
> in the debug log
> >>>> file.
> >>>>
> >>>> In the client log, there's plenty of these which seems 
> somewhat normal:
> >>>>
> >>>> 6509.743164000:imuxsock.c: --------imuxsock calling 
> select, active file
> >>>> descriptors (max 11): 11
> >>>> 6509.743231000:imuxsock.c: Message from UNIX socket: #11
> >>>> 6509.743262000:imuxsock.c: logmsg: flags 4, from 
> 'vanguard', msg May 20
> >>>> 23:15:09 bin/qmail-local[23198]: using .qmail .qmail-cfsc-forum
> >>>> 6509.743281000:imuxsock.c: Message has legacy syslog format.
> >>>> 6509.743302000:imuxsock.c: main queue: entry added, size 
> now 579 entries
> >>>> 6509.743325000:imuxsock.c: wtpAdviseMaxWorkers signals busy
> >>>> 6509.743343000:imuxsock.c: main queue: EnqueueMsg 
> advised worker start
> >>>>
> >>>> However comparing it to an earlier point in the log  
> there's none of the
> >>>> associated tcp queue entries like this:
> >>>>
> >>>> 3587.234178000:main queue:Reg/w0: TCP sent 107 bytes, 
> requested 107
> >>>> 3587.234204000:main queue:Reg/w0: main queue: entering 
> rate limiter
> >>>> 3587.234229000:main queue:Reg/w0: main queue: entry 
> deleted, state 0,
> >>>> size now 0 entries
> >>>> 3587.234255000:main queue:Reg/w0: Called action, logging 
> to builtin-file
> >>>> 3587.234285000:main queue:Reg/w0:  (/var/log/maillog)
> >>>> 3587.234314000:main queue:Reg/w0: Called action, logging 
> to builtin-fwd
> >>>> 3587.234340000:main queue:Reg/w0:  ldap.nmsrv.com
> >>>> 3587.234359000:main queue:Reg/w0:  ldap.nmsrv.com:10514/tcp
> >>>> 3587.234425000:main queue:Reg/w0: TCP sent 78 bytes, requested 78
> >>>> 3587.234446000:main queue:Reg/w0: main queue: entering 
> rate limiter
> >>>> 3587.234471000:main queue:Reg/w0: main queue:Reg/w0: worker IDLE,
> >>>> waiting for work.
> >>>>
> >>>> Anyone able to shed some light on this situation? I've 
> got the full
> >>>> debug logs if it's useful to anyone to take a closer look.
> >>>>
> >>>> Thanks!
> >>>>
> >>>> - Nathan
> >>>>
> >>>>
> >>>>
> >>>>         
> >> 
> --------------------------------------------------------------
> --------------
> >>     
> >>> --------
> >>>       
> >>>> Server config file
> >>>>
> >>>>         
> >> 
> --------------------------------------------------------------
> --------------
> >>     
> >>> --------
> >>>       
> >>>> $ModLoad immark.so # provides --MARK-- message capability
> >>>> $ModLoad imuxsock.so # provides support for local system 
> logging (e.g.
> >>>> via logger command)
> >>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> >>>>
> >>>> $umask 0137
> >>>> $DirCreateMode 0640
> >>>> $FileCreateMode 0640
> >>>> $FileOwner root
> >>>> $FileGroup admin
> >>>>
> >>>> $template DynFile,"/var/log/rsyslog/%HOSTNAME%/messages"
> >>>> $template DynFileCron,"/var/log/rsyslog/%HOSTNAME%/cron"
> >>>> $template DynFileSecure,"/var/log/rsyslog/%HOSTNAME%/secure"
> >>>> $template DynFileMail,"/var/log/rsyslog/%HOSTNAME%/maillog"
> >>>> $template DynFileSpool,"/var/log/rsyslog/%HOSTNAME%/spooler"
> >>>> $template DynFileBoot,"/var/log/rsyslog/%HOSTNAME%/boot"
> >>>> $template DynFileSyslog,"/var/log/rsyslog/%HOSTNAME%/syslog"
> >>>>
> >>>> :msg, contains, "no keys found for"            ~
> >>>> :msg, contains, "session opened for user"        ~
> >>>>
> >>>> # Log all kernel messages to the console.
> >>>> # Logging much else clutters up the screen.
> >>>> kern.*                                                 
> /dev/console
> >>>>
> >>>> # Log anything (except mail) of level info or higher.
> >>>> # Don't log private authentication messages!
> >>>> *.info;mail.none;cron.none                     ?DynFile
> >>>>
> >>>> *.warn;\
> >>>>         authpriv.none;cron.none;mail.none;news.none     
> ?DynFileSyslog
> >>>>
> >>>> # Log all the mail messages in one place.
> >>>> mail.*                                                  
> ?DynFileMail
> >>>>
> >>>> # Log cron stuff
> >>>> cron.*                                                  
> ?DynFileCron
> >>>>
> >>>> # Everybody gets emergency messages
> >>>> *.emerg                                                 *
> >>>>
> >>>> # Save news errors of level crit and higher in a special file
> >>>> uucp,news.crit                                          
> ?DynFileSpool
> >>>>
> >>>> # Save boot messages also to boot.log
> >>>> local7.*                                                
> ?DynFileBoot
> >>>>
> >>>> # Relp config for PCI
> >>>> $ModLoad imrelp
> >>>> $InputRELPServerRun 2515
> >>>>
> >>>> # make gtls driver the default
> >>>> $DefaultNetstreamDriver gtls
> >>>>
> >>>> # certificate files
> >>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
> >>>> $DefaultNetstreamDriverCertFile /etc/ssl/servercrt.pem
> >>>> $DefaultNetstreamDriverKeyFile /etc/ssl/serverkey.pem
> >>>>
> >>>> $ModLoad imtcp.so
> >>>>
> >>>> $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
> >>>> $InputTCPServerStreamDriverAuthMode anon # client is NOT 
> authenticated
> >>>> $InputTCPServerRun 10514 # start up listener at port 10514
> >>>> $InputTCPMaxSessions 500
> >>>>
> >>>>
> >>>>         
> >> 
> --------------------------------------------------------------
> --------------
> >>     
> >>> --------
> >>>       
> >>>> Client config file
> >>>>
> >>>>         
> >> 
> --------------------------------------------------------------
> --------------
> >>     
> >>> --------
> >>>       
> >>>> $ModLoad imuxsock.so # provides support for local system 
> logging (e.g.
> >>>> via logger command)
> >>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> >>>>
> >>>> $umask 0137
> >>>> $DirCreateMode 0640
> >>>> $FileCreateMode 0640
> >>>> $FileOwner root
> >>>> $FileGroup admin
> >>>>
> >>>>
> >>>> # Log anything 'info' or higher, but lower than 'warn'.
> >>>> # Exclude authpriv, cron, mail, and news.  These are 
> logged elsewhere.
> >>>> *.info;*.!warn;\
> >>>>         authpriv.none;cron.none;mail.none;news.none     
> /var/log/messages
> >>>>
> >>>> # Log anything 'warn' or higher.
> >>>> # Exclude authpriv, cron, mail, and news.  These are 
> logged elsewhere.
> >>>> *.warn;\
> >>>>         authpriv.none;cron.none;mail.none;news.none     
> /var/log/syslog
> >>>>
> >>>> # Debugging information is logged here.
> >>>> *.=debug                                                
> /var/log/debug
> >>>>
> >>>> # Private authentication message logging:
> >>>> authpriv.*                                              
> /var/log/secure
> >>>>
> >>>> # Cron related logs:
> >>>> cron.*                                                  
> /var/log/cron
> >>>>
> >>>> # Mail related logs:
> >>>> mail.*                                                  
> /var/log/maillog
> >>>>
> >>>> # Emergency level messages go to all users:
> >>>> *.emerg                                                 *
> >>>>
> >>>>
> >>>> ## Uncomment these lines to use RELP instead for PCI 
> compliance (stunnel
> >>>> required)
> >>>> #$ModLoad omrelp
> >>>> #*.*;mail.none :omrelp:localhost:2515;RSYSLOG_ForwardFormat
> >>>>
> >>>> # certificate files - just CA for a client
> >>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
> >>>>
> >>>> # set up the action
> >>>> $DefaultNetstreamDriver gtls # use gtls netstream driver
> >>>> $ActionSendStreamDriverMode 1 # require TLS for the connection
> >>>> $ActionSendStreamDriverAuthMode anon # server is NOT 
> authenticated
> >>>> *.* @@(o)ldap.nmsrv.com:10514 # send (all) messages
> >>>>
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.comoften
> >>>>         
> >>> --
> >>> Patrick Shen
> >>> Operations Engineer
> >>>
> >>> net mobile AG Shanghai office
> >>>
> >>> T: +86  21 61350900 - 222
> >>> F: +86  21 61350906
> >>> M: +86  13122245730
> >>>
> >>> FE55 6A7F 3192 F359 C1DF  12F6 9078 58B4 57AF 0BE3
> >>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>       
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>     
> >
> >
> >   
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 


From rgerhards at hq.adiscon.com  Mon Jun  8 17:33:00 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 8 Jun 2009 17:33:00 +0200
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>

Hi all,

I am working on the ultra-reliable file writing capability. I have revisited
the doc and am considering fsync() vs. fdatasync(). From the man pages, it
looks like fdatasync() is sufficient if I just need to ensure that any data
written can also be read. The advantage of fdatasync() is that it is somewhat
faster as not all meta data is necessarily written.

I would appreciate if someone could comment on this issue.

Thanks,
Rainer


From david at lang.hm  Mon Jun  8 18:09:56 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 8 Jun 2009 09:09:56 -0700 (PDT)
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906080908030.32468@asgard>

On Mon, 8 Jun 2009, Rainer Gerhards wrote:

> Hi all,
>
> I am working on the ultra-reliable file writing capability. I have revisited
> the doc and am considering fsync() vs. fdatasync(). From the man pages, it
> looks like fdatasync() is sufficient if I just need to ensure that any data
> written can also be read. The advantage of fdatasync() is that it is somewhat
> faster as not all meta data is necessarily written.
>
> I would appreciate if someone could comment on this issue.

my understanding is that if you are growing the file you need to do a 
fsync of the directory, but if you are re-writing part of an already 
allocated file fdatasync is the better thing to use.

David Lang


From rgerhards at hq.adiscon.com  Mon Jun  8 18:18:12 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 8 Jun 2009 18:18:12 +0200
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906080908030.32468@asgard>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8C6@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Monday, June 08, 2009 6:10 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
> 
> On Mon, 8 Jun 2009, Rainer Gerhards wrote:
> 
> > Hi all,
> >
> > I am working on the ultra-reliable file writing capability. I have
revisited
> > the doc and am considering fsync() vs. fdatasync(). From the man pages,
it
> > looks like fdatasync() is sufficient if I just need to ensure that any
data
> > written can also be read. The advantage of fdatasync() is that it is
> somewhat
> > faster as not all meta data is necessarily written.
> >
> > I would appreciate if someone could comment on this issue.
> 
> my understanding is that if you are growing the file you need to do a
> fsync of the directory, but if you are re-writing part of an already
> allocated file fdatasync is the better thing to use.

The man page[1] is somewhat vague. It tells:

"Calling fsync() does not necessarily ensure that the entry in the directory
containing the file has also reached disk. For that an explicit fsync() on a
file descriptor for the directory is also needed."

But also

"fdatasync() is similar to fsync(), but does not flush modified metadata
unless that metadata is needed in order to allow a subsequent data retrieval
to be correctly handled. For example, changes to st_atime or st_mtime
(respectively, time of last access and time of last modification; see
stat(2)) do not not require flushing because they are not necessary for a
subsequent data read to be handled correctly. On the other hand, a change to
the file size (st_size, as made by say ftruncate(2)), would require a
metadata flush.
"

So my current understanding is that fdatasync() is always safe, but there
still exist situations where a sync on the directory is needed. What I can
think of is a newly created file.

I have to admit that I am a bit puzzled of what it takes to do the "explicit
fsync on the directory". Most importantly, I need to have the file handle
open for writing, else the fsync will potentially fail. I don't think this is
such an excellent idea on directories.. I am right now working on this, and
my search has not brought up many good hints. So what I currently plan to do
is an open(dirname, O_RDWR) on the directory and then do an fsync on it
later. 

And I guess even after the fsync() on the directeroy, there potentially is an
issue with newly created files inside newly create directories, but I don't
think I will address this immediately. But I may be useful/necessary to add a
tree walker who actually walks the full path name and calls the fsyncs on all
of them...

Comments are highly appreciated.

Rainer

[1] http://linux.die.net/man/2/fsync


From rgerhards at hq.adiscon.com  Mon Jun  8 18:31:28 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 8 Jun 2009 18:31:28 +0200
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906080908030.32468@asgard>
	<9B6E2A8877C38245BFB15CC491A11DA706F8C6@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8C7@GRFEXC.intern.adiscon.com>

> my search has not brought up many good hints. So what I currently plan to
do
> is an open(dirname, O_RDWR) on the directory and then do an fsync on it
> later.

... which does not work, because I receive EISDIR ;)

No I do an O_RDONLY, which works and then even fsync() on it works as well.
So this is what I will now be doing, but with a somewhat bad feeling...

Rainer


From epiphani at gmail.com  Mon Jun  8 18:12:13 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 8 Jun 2009 12:12:13 -0400
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
In-Reply-To: <alpine.DEB.1.10.0906080908030.32468@asgard>
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906080908030.32468@asgard>
Message-ID: <e7ca40f70906080912i189153a6m1fe4633008048c12@mail.gmail.com>

On Mon, Jun 8, 2009 at 12:09 PM, <david at lang.hm> wrote:
>> I am working on the ultra-reliable file writing capability. I have revisited
>> the doc and am considering fsync() vs. fdatasync(). From the man pages, it
>> looks like fdatasync() is sufficient if I just need to ensure that any data
>> written can also be read. The advantage of fdatasync() is that it is somewhat
>> faster as not all meta data is necessarily written.
>>
>> I would appreciate if someone could comment on this issue.
>
> my understanding is that if you are growing the file you need to do a
> fsync of the directory, but if you are re-writing part of an already
> allocated file fdatasync is the better thing to use.

While following this path, it would also be a good idea to check NFS
compatibility, as I would bet that is a common configuration.  NFS can
have its own unique quirks....

-Aaron


From rgerhards at hq.adiscon.com  Mon Jun  8 18:53:18 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 8 Jun 2009 18:53:18 +0200
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906080908030.32468@asgard>
	<e7ca40f70906080912i189153a6m1fe4633008048c12@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8C8@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> Sent: Monday, June 08, 2009 6:12 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
> 
> On Mon, Jun 8, 2009 at 12:09 PM, <david at lang.hm> wrote:
> >> I am working on the ultra-reliable file writing capability. I have
> revisited
> >> the doc and am considering fsync() vs. fdatasync(). From the man pages,
it
> >> looks like fdatasync() is sufficient if I just need to ensure that any
data
> >> written can also be read. The advantage of fdatasync() is that it is
> somewhat
> >> faster as not all meta data is necessarily written.
> >>
> >> I would appreciate if someone could comment on this issue.
> >
> > my understanding is that if you are growing the file you need to do a
> > fsync of the directory, but if you are re-writing part of an already
> > allocated file fdatasync is the better thing to use.
> 
> While following this path, it would also be a good idea to check NFS
> compatibility, as I would bet that is a common configuration.  NFS can
> have its own unique quirks....

>From what I have seen during my research, I would expect that NFS does not
always correctly fsync(). It seems to depend a lot on the actual
implementation, maybe version of it. What I currently do is ignore an error
during fsync() (fdatasync to be more precise). The reason is that there is
not much I could do if I see the error. Aborting the run would probably be
much more fatal than ignoring the error. The same goes if the fsync() for the
directory fails or if I can not open the directory.

I would assume that if a truly reliable system is requested, local disks must
be used.

Comments welcome,
Rainer


From david at lang.hm  Mon Jun  8 19:07:35 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 8 Jun 2009 10:07:35 -0700 (PDT)
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F8C6@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906080908030.32468@asgard>
	<9B6E2A8877C38245BFB15CC491A11DA706F8C6@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906080956030.9159@asgard>

On Mon, 8 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Mon, 8 Jun 2009, Rainer Gerhards wrote:
>>
>>> Hi all,
>>>
>>> I am working on the ultra-reliable file writing capability. I have
> revisited
>>> the doc and am considering fsync() vs. fdatasync(). From the man pages,
> it
>>> looks like fdatasync() is sufficient if I just need to ensure that any
> data
>>> written can also be read. The advantage of fdatasync() is that it is
>> somewhat
>>> faster as not all meta data is necessarily written.
>>>
>>> I would appreciate if someone could comment on this issue.
>>
>> my understanding is that if you are growing the file you need to do a
>> fsync of the directory, but if you are re-writing part of an already
>> allocated file fdatasync is the better thing to use.
>
> The man page[1] is somewhat vague. It tells:
>
> "Calling fsync() does not necessarily ensure that the entry in the directory
> containing the file has also reached disk. For that an explicit fsync() on a
> file descriptor for the directory is also needed."
>
> But also
>
> "fdatasync() is similar to fsync(), but does not flush modified metadata
> unless that metadata is needed in order to allow a subsequent data retrieval
> to be correctly handled. For example, changes to st_atime or st_mtime
> (respectively, time of last access and time of last modification; see
> stat(2)) do not not require flushing because they are not necessary for a
> subsequent data read to be handled correctly. On the other hand, a change to
> the file size (st_size, as made by say ftruncate(2)), would require a
> metadata flush.
> "
>
> So my current understanding is that fdatasync() is always safe, but there
> still exist situations where a sync on the directory is needed. What I can
> think of is a newly created file.

any time the file size changes (so that the metadata about the disk 
allocation is safe)

> I have to admit that I am a bit puzzled of what it takes to do the "explicit
> fsync on the directory". Most importantly, I need to have the file handle
> open for writing, else the fsync will potentially fail. I don't think this is
> such an excellent idea on directories.. I am right now working on this, and
> my search has not brought up many good hints. So what I currently plan to do
> is an open(dirname, O_RDWR) on the directory and then do an fsync on it
> later.

I believe that that's correct.

> And I guess even after the fsync() on the directeroy, there potentially is an
> issue with newly created files inside newly create directories, but I don't
> think I will address this immediately. But I may be useful/necessary to add a
> tree walker who actually walks the full path name and calls the fsyncs on all
> of them...

in an ideal world yes, but in practice you would only need to do this when 
you create a new file and that new directory entry caused the filesystem 
to allocate more space on disk to hole the directory entry, so it's almost 
never an issue.

another advantage of fdatasync is that it tells the OS exactly what 
portion of the file it needs to work on, with a fsync it needs to search 
all buffers related to that file to figure out what needs to be written 
(not a big hit, but a hit)

David Lang


From nathan at gossamer-threads.com  Tue Jun  9 18:57:49 2009
From: nathan at gossamer-threads.com (Nathan March)
Date: Tue, 09 Jun 2009 09:57:49 -0700
Subject: [rsyslog] Possible bug? 100% cpu on a server thread
 and	the	client stops sending data
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F8BA@GRFEXC.intern.adiscon.com>
References: <4A15999D.3000909@gossamer-threads.com><4A1A37DE.8050004@net-m.de>	<9B6E2A8877C38245BFB15CC491A11DA702B0FB@GRFEXC.intern.adiscon.com><4A1A57A0.1060903@net-m.de>	<4A29978C.9080505@gossamer-threads.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F8BA@GRFEXC.intern.adiscon.com>
Message-ID: <4A2E948D.4060809@gossamer-threads.com>

Confirmed as still happening on 4.1.7

- Nathan

Rainer Gerhards wrote:
> Please give v4 a try, the bug is potentially fixed there. If it actually is,
> this gives me a clue of where to look at.
>
> Rainer 
>
>   
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com 
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Nathan March
>> Sent: Saturday, June 06, 2009 12:09 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server 
>> thread and the client stops sending data
>>
>> Any hope of this bug getting some attention soon? =)
>>
>> Thanks,
>> Nathan
>>
>>
>> Patrick Shen wrote:
>>     
>>> Hi Rainer,
>>>
>>> Thanks for the quick reply.
>>>
>>> My environment is like below:
>>>
>>> CLIENT ---
>>>          |
>>>          | (TCP)
>>>          --------> SERVER ----
>>>                              |
>>>                              | (RELP Relay)
>>>                              --------------> SERVER Standby
>>>
>>> I've tested, even if the standby server is under high load, 
>>>       
>> one of our
>>     
>>> clients won't accept any new network connections.
>>>
>>> Just for your information.
>>>
>>> Best regards,
>>> Patrick
>>>
>>> Rainer Gerhards wrote:
>>>   
>>>       
>>>> Hi all,
>>>>
>>>> thanks for the detailed info. Will look at it asap, but I 
>>>>         
>> am currently tied
>>     
>>>> up with some other work...
>>>>
>>>> Please let me know anything else that may be relevant.
>>>>
>>>> Thanks,
>>>> Rainer
>>>>
>>>>     
>>>>         
>>>>> -----Original Message-----
>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>> bounces at lists.adiscon.com] On Behalf Of Patrick Shen
>>>>> Sent: Monday, May 25, 2009 8:17 AM
>>>>> To: rsyslog-users
>>>>> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server 
>>>>>           
>> thread and the
>>     
>>>>> client stops sending data
>>>>>
>>>>> Hi Nathan,
>>>>>
>>>>> In my company, we have the similar symptom. We have 50+ 
>>>>>           
>> servers which
>>     
>>>>> send logs to the central logging servers. Both clients 
>>>>>           
>> and server use
>>     
>>>>> rsyslog v3.20.5 right now. In the past when we use 
>>>>>           
>> v3.20.0, the symptom
>>     
>>>>> is more common when server is under high load average.
>>>>>
>>>>> It's also weird for us, not all of clients were lost of 
>>>>>           
>> connections. We
>>     
>>>>> have 2-3 clients which is running the same application 
>>>>>           
>> will looks like
>>     
>>>>> hanging up when server is high-load. New ssh connection 
>>>>>           
>> is very slow at
>>     
>>>>> that time. But the connected ssh session works normal (We 
>>>>>           
>> have some
>>     
>>>>> prepared screen session on other clients which connect to 
>>>>>           
>> them via ssh).
>>     
>>>>> When the clients are lost of connections. So far our 
>>>>>           
>> solution is to
>>     
>>>>> restart rsyslog both on affected clients and server. Then 
>>>>>           
>> the clients
>>     
>>>>> get back and look normal.
>>>>>
>>>>> We've suffered it for a while, but I didn't take chance 
>>>>>           
>> to analyze it
>>     
>>>>> like you.
>>>>>
>>>>> Best regards,
>>>>> Patrick
>>>>>
>>>>> Nathan March wrote:
>>>>>       
>>>>>           
>>>>>> Having a weird issue here..... Testing out a new rsyslog 
>>>>>>             
>> deployment and
>>     
>>>>>> I've got around 30 servers logging to one machine. On 
>>>>>>             
>> one of the clients
>>     
>>>>>> we had an issue where ssh was mysteriously very slow to login and
>>>>>> tracked it down to rsyslog causing issues. 
>>>>>>             
>> Simultaneously, one of the
>>     
>>>>>> threads on the logging server spiked up to 100%. New log 
>>>>>>             
>> data from the
>>     
>>>>>> client would stop showing up on the server at this poitn.
>>>>>>
>>>>>> On the client after attempting to gracefully stop rsyslog:
>>>>>>
>>>>>> vanguard etc # ps aux | grep -i rsyslog
>>>>>> root      3764  0.0  0.1  5092 2564 ?        S    15:06   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f 
>>>>>>             
>> /etc/rsyslog.conf
>>     
>>>>>> root      3766  0.0  0.1  5092 2564 ?        S    15:06   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f 
>>>>>>             
>> /etc/rsyslog.conf
>>     
>>>>>> root      6203  0.0  0.0  1512  524 pts/3    S    15:09  
>>>>>>             
>>  0:00 grep -i
>>     
>>>>>> rsyslog
>>>>>> vanguard etc # strace -p 3764
>>>>>> Process 3764 attached - interrupt to quit
>>>>>> send(11, 
>>>>>>             
>> "\25\3\2\1\0uE\254\304\n\311\377\204}p\237O\211\322\211"...,
>>     
>>>>>> 261, 0 <unfinished ...>
>>>>>> Process 3764 detached
>>>>>>
>>>>>> FD 11 being the socket to the logging server, it never 
>>>>>>             
>> does anything
>>     
>>>>>> asides from that send.
>>>>>>
>>>>>> On the server:
>>>>>>
>>>>>> ldap ~ # ps aux | grep rsyslog
>>>>>> root      7370  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7379  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7380  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7381  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7382  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7383  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7384  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7385 28.3  0.0  83936  3380 ?        R    14:54   5:07
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> Stracing the 7385 pid just shows this repeating over and 
>>>>>>             
>> over (FD 4
>>     
>>>>>> being the network socket again):
>>>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 
>>>>>>             
>> EAGAIN (Resource
>>     
>>>>>> temporarily unavailable)
>>>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 
>>>>>>             
>> EAGAIN (Resource
>>     
>>>>>> temporarily unavailable)
>>>>>>
>>>>>> It happened 3 times within a relatively short period of 
>>>>>>             
>> time, sometimes
>>     
>>>>>> within minutes of me restarting rsyslog on both 
>>>>>>             
>> machines. I enabled
>>     
>>>>>> debugging and it took around 8 hours to pop up again.
>>>>>>
>>>>>> With debugging, I pulled this from the rsyslog server:
>>>>>>
>>>>>> 6640.408108000:imtcp.c: New connect on NSD 0x59ddc0.
>>>>>> 6640.408129000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: 
>>>>>>             
>> pszRcvBuf (nil),
>>     
>>>>>> lenRcvBuf 0
>>>>>> 6640.408144000:imtcp.c: hasRcvInBuffer on nsd 0x59dcd0: pszRcvBuf
>>>>>> 0x5c11c0, lenRcvBuf -1
>>>>>> 6640.408158000:imtcp.c: hasRcvInBuffer on nsd 0x59dc10: pszRcvBuf
>>>>>> 0x5cb510, lenRcvBuf -1
>>>>>> 6640.408171000:imtcp.c: hasRcvInBuffer on nsd 0x5c3d70: pszRcvBuf
>>>>>> 0x5d45c0, lenRcvBuf -1
>>>>>> 6640.408185000:imtcp.c: hasRcvInBuffer on nsd 0x5cf460: pszRcvBuf
>>>>>> 0x5dd010, lenRcvBuf -1
>>>>>> 6640.408199000:imtcp.c: hasRcvInBuffer on nsd 0x5d8600: pszRcvBuf
>>>>>> 0x5e58b0, lenRcvBuf -1
>>>>>> 6640.408213000:imtcp.c: hasRcvInBuffer on nsd 0x59d690: pszRcvBuf
>>>>>> 0x5ee150, lenRcvBuf -1
>>>>>> 6640.408227000:imtcp.c: hasRcvInBuffer on nsd 0x59d2f0: pszRcvBuf
>>>>>> 0x5f7320, lenRcvBuf -1
>>>>>> 6640.408241000:imtcp.c: hasRcvInBuffer on nsd 0x5e8840: pszRcvBuf
>>>>>> 0x5fe0a0, lenRcvBuf -1
>>>>>> 6640.408255000:imtcp.c: hasRcvInBuffer on nsd 0x6027b0: pszRcvBuf
>>>>>> 0x608350, lenRcvBuf -1
>>>>>> 6640.408269000:imtcp.c: hasRcvInBuffer on nsd 0x5c3660: pszRcvBuf
>>>>>> 0x612110, lenRcvBuf -1
>>>>>> 6640.408283000:imtcp.c: hasRcvInBuffer on nsd 0x602690: pszRcvBuf
>>>>>> 0x615040, lenRcvBuf -1
>>>>>> 6640.408296000:imtcp.c: hasRcvInBuffer on nsd 0x5c3b60: pszRcvBuf
>>>>>> 0x6218f0, lenRcvBuf -1
>>>>>> 6640.408310000:imtcp.c: hasRcvInBuffer on nsd 0x60c150: pszRcvBuf
>>>>>> 0x62a190, lenRcvBuf -1
>>>>>> 6640.408324000:imtcp.c: hasRcvInBuffer on nsd 0x625420: pszRcvBuf
>>>>>> 0x634fe0, lenRcvBuf -1
>>>>>> 6640.408338000:imtcp.c: hasRcvInBuffer on nsd 0x6117b0: pszRcvBuf
>>>>>> 0x63b000, lenRcvBuf -1
>>>>>> 6640.408352000:imtcp.c: hasRcvInBuffer on nsd 0x603990: pszRcvBuf
>>>>>> 0x643c70, lenRcvBuf -1
>>>>>> 6640.408365000:imtcp.c: hasRcvInBuffer on nsd 0x633c00: pszRcvBuf
>>>>>> 0x64dbb0, lenRcvBuf -1
>>>>>> 6640.408379000:imtcp.c: hasRcvInBuffer on nsd 0x62cfa0: pszRcvBuf
>>>>>> 0x650960, lenRcvBuf -1
>>>>>> 6640.408393000:imtcp.c: hasRcvInBuffer on nsd 0x633d10: pszRcvBuf
>>>>>> 0x65dbc0, lenRcvBuf -1
>>>>>> 6640.408407000:imtcp.c: hasRcvInBuffer on nsd 0x5f2860: pszRcvBuf
>>>>>> 0x666020, lenRcvBuf -1
>>>>>> 6640.408421000:imtcp.c: hasRcvInBuffer on nsd 0x64d250: pszRcvBuf
>>>>>> 0x66e480, lenRcvBuf -1
>>>>>> 6640.408435000:imtcp.c: hasRcvInBuffer on nsd 0x659b30: pszRcvBuf
>>>>>> 0x676d20, lenRcvBuf 78
>>>>>> 6640.408448000:imtcp.c: hasRcvInBuffer on nsd 0x669fb0: pszRcvBuf
>>>>>> 0x67fdd0, lenRcvBuf -1
>>>>>> 6640.408462000:imtcp.c: hasRcvInBuffer on nsd 0x59a630: pszRcvBuf
>>>>>> 0x687f10, lenRcvBuf -1
>>>>>> 6640.408476000:imtcp.c: hasRcvInBuffer on nsd 0x6614c0: pszRcvBuf
>>>>>> 0x6907b0, lenRcvBuf -1
>>>>>> 6640.408490000:imtcp.c: hasRcvInBuffer on nsd 0x60c2d0: pszRcvBuf
>>>>>> 0x699050, lenRcvBuf -1
>>>>>> 6640.408504000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: 
>>>>>>             
>> pszRcvBuf (nil),
>>     
>>>>>> lenRcvBuf 0
>>>>>>
>>>>>> This repeats enough to generate around 29 million lines 
>>>>>>             
>> in the debug log
>>     
>>>>>> file.
>>>>>>
>>>>>> In the client log, there's plenty of these which seems 
>>>>>>             
>> somewhat normal:
>>     
>>>>>> 6509.743164000:imuxsock.c: --------imuxsock calling 
>>>>>>             
>> select, active file
>>     
>>>>>> descriptors (max 11): 11
>>>>>> 6509.743231000:imuxsock.c: Message from UNIX socket: #11
>>>>>> 6509.743262000:imuxsock.c: logmsg: flags 4, from 
>>>>>>             
>> 'vanguard', msg May 20
>>     
>>>>>> 23:15:09 bin/qmail-local[23198]: using .qmail .qmail-cfsc-forum
>>>>>> 6509.743281000:imuxsock.c: Message has legacy syslog format.
>>>>>> 6509.743302000:imuxsock.c: main queue: entry added, size 
>>>>>>             
>> now 579 entries
>>     
>>>>>> 6509.743325000:imuxsock.c: wtpAdviseMaxWorkers signals busy
>>>>>> 6509.743343000:imuxsock.c: main queue: EnqueueMsg 
>>>>>>             
>> advised worker start
>>     
>>>>>> However comparing it to an earlier point in the log  
>>>>>>             
>> there's none of the
>>     
>>>>>> associated tcp queue entries like this:
>>>>>>
>>>>>> 3587.234178000:main queue:Reg/w0: TCP sent 107 bytes, 
>>>>>>             
>> requested 107
>>     
>>>>>> 3587.234204000:main queue:Reg/w0: main queue: entering 
>>>>>>             
>> rate limiter
>>     
>>>>>> 3587.234229000:main queue:Reg/w0: main queue: entry 
>>>>>>             
>> deleted, state 0,
>>     
>>>>>> size now 0 entries
>>>>>> 3587.234255000:main queue:Reg/w0: Called action, logging 
>>>>>>             
>> to builtin-file
>>     
>>>>>> 3587.234285000:main queue:Reg/w0:  (/var/log/maillog)
>>>>>> 3587.234314000:main queue:Reg/w0: Called action, logging 
>>>>>>             
>> to builtin-fwd
>>     
>>>>>> 3587.234340000:main queue:Reg/w0:  ldap.nmsrv.com
>>>>>> 3587.234359000:main queue:Reg/w0:  ldap.nmsrv.com:10514/tcp
>>>>>> 3587.234425000:main queue:Reg/w0: TCP sent 78 bytes, requested 78
>>>>>> 3587.234446000:main queue:Reg/w0: main queue: entering 
>>>>>>             
>> rate limiter
>>     
>>>>>> 3587.234471000:main queue:Reg/w0: main queue:Reg/w0: worker IDLE,
>>>>>> waiting for work.
>>>>>>
>>>>>> Anyone able to shed some light on this situation? I've 
>>>>>>             
>> got the full
>>     
>>>>>> debug logs if it's useful to anyone to take a closer look.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> - Nathan
>>>>>>
>>>>>>
>>>>>>
>>>>>>         
>>>>>>             
>> --------------------------------------------------------------
>> --------------
>>     
>>>>     
>>>>         
>>>>> --------
>>>>>       
>>>>>           
>>>>>> Server config file
>>>>>>
>>>>>>         
>>>>>>             
>> --------------------------------------------------------------
>> --------------
>>     
>>>>     
>>>>         
>>>>> --------
>>>>>       
>>>>>           
>>>>>> $ModLoad immark.so # provides --MARK-- message capability
>>>>>> $ModLoad imuxsock.so # provides support for local system 
>>>>>>             
>> logging (e.g.
>>     
>>>>>> via logger command)
>>>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
>>>>>>
>>>>>> $umask 0137
>>>>>> $DirCreateMode 0640
>>>>>> $FileCreateMode 0640
>>>>>> $FileOwner root
>>>>>> $FileGroup admin
>>>>>>
>>>>>> $template DynFile,"/var/log/rsyslog/%HOSTNAME%/messages"
>>>>>> $template DynFileCron,"/var/log/rsyslog/%HOSTNAME%/cron"
>>>>>> $template DynFileSecure,"/var/log/rsyslog/%HOSTNAME%/secure"
>>>>>> $template DynFileMail,"/var/log/rsyslog/%HOSTNAME%/maillog"
>>>>>> $template DynFileSpool,"/var/log/rsyslog/%HOSTNAME%/spooler"
>>>>>> $template DynFileBoot,"/var/log/rsyslog/%HOSTNAME%/boot"
>>>>>> $template DynFileSyslog,"/var/log/rsyslog/%HOSTNAME%/syslog"
>>>>>>
>>>>>> :msg, contains, "no keys found for"            ~
>>>>>> :msg, contains, "session opened for user"        ~
>>>>>>
>>>>>> # Log all kernel messages to the console.
>>>>>> # Logging much else clutters up the screen.
>>>>>> kern.*                                                 
>>>>>>             
>> /dev/console
>>     
>>>>>> # Log anything (except mail) of level info or higher.
>>>>>> # Don't log private authentication messages!
>>>>>> *.info;mail.none;cron.none                     ?DynFile
>>>>>>
>>>>>> *.warn;\
>>>>>>         authpriv.none;cron.none;mail.none;news.none     
>>>>>>             
>> ?DynFileSyslog
>>     
>>>>>> # Log all the mail messages in one place.
>>>>>> mail.*                                                  
>>>>>>             
>> ?DynFileMail
>>     
>>>>>> # Log cron stuff
>>>>>> cron.*                                                  
>>>>>>             
>> ?DynFileCron
>>     
>>>>>> # Everybody gets emergency messages
>>>>>> *.emerg                                                 *
>>>>>>
>>>>>> # Save news errors of level crit and higher in a special file
>>>>>> uucp,news.crit                                          
>>>>>>             
>> ?DynFileSpool
>>     
>>>>>> # Save boot messages also to boot.log
>>>>>> local7.*                                                
>>>>>>             
>> ?DynFileBoot
>>     
>>>>>> # Relp config for PCI
>>>>>> $ModLoad imrelp
>>>>>> $InputRELPServerRun 2515
>>>>>>
>>>>>> # make gtls driver the default
>>>>>> $DefaultNetstreamDriver gtls
>>>>>>
>>>>>> # certificate files
>>>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
>>>>>> $DefaultNetstreamDriverCertFile /etc/ssl/servercrt.pem
>>>>>> $DefaultNetstreamDriverKeyFile /etc/ssl/serverkey.pem
>>>>>>
>>>>>> $ModLoad imtcp.so
>>>>>>
>>>>>> $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
>>>>>> $InputTCPServerStreamDriverAuthMode anon # client is NOT 
>>>>>>             
>> authenticated
>>     
>>>>>> $InputTCPServerRun 10514 # start up listener at port 10514
>>>>>> $InputTCPMaxSessions 500
>>>>>>
>>>>>>
>>>>>>         
>>>>>>             
>> --------------------------------------------------------------
>> --------------
>>     
>>>>     
>>>>         
>>>>> --------
>>>>>       
>>>>>           
>>>>>> Client config file
>>>>>>
>>>>>>         
>>>>>>             
>> --------------------------------------------------------------
>> --------------
>>     
>>>>     
>>>>         
>>>>> --------
>>>>>       
>>>>>           
>>>>>> $ModLoad imuxsock.so # provides support for local system 
>>>>>>             
>> logging (e.g.
>>     
>>>>>> via logger command)
>>>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
>>>>>>
>>>>>> $umask 0137
>>>>>> $DirCreateMode 0640
>>>>>> $FileCreateMode 0640
>>>>>> $FileOwner root
>>>>>> $FileGroup admin
>>>>>>
>>>>>>
>>>>>> # Log anything 'info' or higher, but lower than 'warn'.
>>>>>> # Exclude authpriv, cron, mail, and news.  These are 
>>>>>>             
>> logged elsewhere.
>>     
>>>>>> *.info;*.!warn;\
>>>>>>         authpriv.none;cron.none;mail.none;news.none     
>>>>>>             
>> /var/log/messages
>>     
>>>>>> # Log anything 'warn' or higher.
>>>>>> # Exclude authpriv, cron, mail, and news.  These are 
>>>>>>             
>> logged elsewhere.
>>     
>>>>>> *.warn;\
>>>>>>         authpriv.none;cron.none;mail.none;news.none     
>>>>>>             
>> /var/log/syslog
>>     
>>>>>> # Debugging information is logged here.
>>>>>> *.=debug                                                
>>>>>>             
>> /var/log/debug
>>     
>>>>>> # Private authentication message logging:
>>>>>> authpriv.*                                              
>>>>>>             
>> /var/log/secure
>>     
>>>>>> # Cron related logs:
>>>>>> cron.*                                                  
>>>>>>             
>> /var/log/cron
>>     
>>>>>> # Mail related logs:
>>>>>> mail.*                                                  
>>>>>>             
>> /var/log/maillog
>>     
>>>>>> # Emergency level messages go to all users:
>>>>>> *.emerg                                                 *
>>>>>>
>>>>>>
>>>>>> ## Uncomment these lines to use RELP instead for PCI 
>>>>>>             
>> compliance (stunnel
>>     
>>>>>> required)
>>>>>> #$ModLoad omrelp
>>>>>> #*.*;mail.none :omrelp:localhost:2515;RSYSLOG_ForwardFormat
>>>>>>
>>>>>> # certificate files - just CA for a client
>>>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
>>>>>>
>>>>>> # set up the action
>>>>>> $DefaultNetstreamDriver gtls # use gtls netstream driver
>>>>>> $ActionSendStreamDriverMode 1 # require TLS for the connection
>>>>>> $ActionSendStreamDriverAuthMode anon # server is NOT 
>>>>>>             
>> authenticated
>>     
>>>>>> *.* @@(o)ldap.nmsrv.com:10514 # send (all) messages
>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.comoften
>>>>>>         
>>>>>>             
>>>>> --
>>>>> Patrick Shen
>>>>> Operations Engineer
>>>>>
>>>>> net mobile AG Shanghai office
>>>>>
>>>>> T: +86  21 61350900 - 222
>>>>> F: +86  21 61350906
>>>>> M: +86  13122245730
>>>>>
>>>>> FE55 6A7F 3192 F359 C1DF  12F6 9078 58B4 57AF 0BE3
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>>       
>>>>>           
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>     
>>>>         
>>>   
>>>       
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
>>     
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>   


From rgerhards at hq.adiscon.com  Wed Jun 10 18:12:38 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 10 Jun 2009 18:12:38 +0200
Subject: [rsyslog] Possible bug? 100% cpu on a server thread
	and	the	client stops sending data
References: <4A15999D.3000909@gossamer-threads.com><4A1A37DE.8050004@net-m.de>	<9B6E2A8877C38245BFB15CC491A11DA702B0FB@GRFEXC.intern.adiscon.com><4A1A57A0.1060903@net-m.de>	<4A29978C.9080505@gossamer-threads.com><9B6E2A8877C38245BFB15CC491A11DA706F8BA@GRFEXC.intern.adiscon.com>
	<4A2E948D.4060809@gossamer-threads.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8FD@GRFEXC.intern.adiscon.com>

Args, sorry.. I hadn't thought about the beta. Could you retry with the
current devel version? This is the one that potentially has a fix.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Nathan March
> Sent: Tuesday, June 09, 2009 6:58 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server thread and the
> client stops sending data
> 
> Confirmed as still happening on 4.1.7
> 
> - Nathan
> 
> Rainer Gerhards wrote:
> > Please give v4 a try, the bug is potentially fixed there. If it actually
is,
> > this gives me a clue of where to look at.
> >
> > Rainer
> >
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Nathan March
> >> Sent: Saturday, June 06, 2009 12:09 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server
> >> thread and the client stops sending data
> >>
> >> Any hope of this bug getting some attention soon? =)
> >>
> >> Thanks,
> >> Nathan
> >>
> >>
> >> Patrick Shen wrote:
> >>
> >>> Hi Rainer,
> >>>
> >>> Thanks for the quick reply.
> >>>
> >>> My environment is like below:
> >>>
> >>> CLIENT ---
> >>>          |
> >>>          | (TCP)
> >>>          --------> SERVER ----
> >>>                              |
> >>>                              | (RELP Relay)
> >>>                              --------------> SERVER Standby
> >>>
> >>> I've tested, even if the standby server is under high load,
> >>>
> >> one of our
> >>
> >>> clients won't accept any new network connections.
> >>>
> >>> Just for your information.
> >>>
> >>> Best regards,
> >>> Patrick
> >>>
> >>> Rainer Gerhards wrote:
> >>>
> >>>
> >>>> Hi all,
> >>>>
> >>>> thanks for the detailed info. Will look at it asap, but I
> >>>>
> >> am currently tied
> >>
> >>>> up with some other work...
> >>>>
> >>>> Please let me know anything else that may be relevant.
> >>>>
> >>>> Thanks,
> >>>> Rainer
> >>>>
> >>>>
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>>> bounces at lists.adiscon.com] On Behalf Of Patrick Shen
> >>>>> Sent: Monday, May 25, 2009 8:17 AM
> >>>>> To: rsyslog-users
> >>>>> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server
> >>>>>
> >> thread and the
> >>
> >>>>> client stops sending data
> >>>>>
> >>>>> Hi Nathan,
> >>>>>
> >>>>> In my company, we have the similar symptom. We have 50+
> >>>>>
> >> servers which
> >>
> >>>>> send logs to the central logging servers. Both clients
> >>>>>
> >> and server use
> >>
> >>>>> rsyslog v3.20.5 right now. In the past when we use
> >>>>>
> >> v3.20.0, the symptom
> >>
> >>>>> is more common when server is under high load average.
> >>>>>
> >>>>> It's also weird for us, not all of clients were lost of
> >>>>>
> >> connections. We
> >>
> >>>>> have 2-3 clients which is running the same application
> >>>>>
> >> will looks like
> >>
> >>>>> hanging up when server is high-load. New ssh connection
> >>>>>
> >> is very slow at
> >>
> >>>>> that time. But the connected ssh session works normal (We
> >>>>>
> >> have some
> >>
> >>>>> prepared screen session on other clients which connect to
> >>>>>
> >> them via ssh).
> >>
> >>>>> When the clients are lost of connections. So far our
> >>>>>
> >> solution is to
> >>
> >>>>> restart rsyslog both on affected clients and server. Then
> >>>>>
> >> the clients
> >>
> >>>>> get back and look normal.
> >>>>>
> >>>>> We've suffered it for a while, but I didn't take chance
> >>>>>
> >> to analyze it
> >>
> >>>>> like you.
> >>>>>
> >>>>> Best regards,
> >>>>> Patrick
> >>>>>
> >>>>> Nathan March wrote:
> >>>>>
> >>>>>
> >>>>>> Having a weird issue here..... Testing out a new rsyslog
> >>>>>>
> >> deployment and
> >>
> >>>>>> I've got around 30 servers logging to one machine. On
> >>>>>>
> >> one of the clients
> >>
> >>>>>> we had an issue where ssh was mysteriously very slow to login and
> >>>>>> tracked it down to rsyslog causing issues.
> >>>>>>
> >> Simultaneously, one of the
> >>
> >>>>>> threads on the logging server spiked up to 100%. New log
> >>>>>>
> >> data from the
> >>
> >>>>>> client would stop showing up on the server at this poitn.
> >>>>>>
> >>>>>> On the client after attempting to gracefully stop rsyslog:
> >>>>>>
> >>>>>> vanguard etc # ps aux | grep -i rsyslog
> >>>>>> root      3764  0.0  0.1  5092 2564 ?        S    15:06   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >> /etc/rsyslog.conf
> >>
> >>>>>> root      3766  0.0  0.1  5092 2564 ?        S    15:06   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >> /etc/rsyslog.conf
> >>
> >>>>>> root      6203  0.0  0.0  1512  524 pts/3    S    15:09
> >>>>>>
> >>  0:00 grep -i
> >>
> >>>>>> rsyslog
> >>>>>> vanguard etc # strace -p 3764
> >>>>>> Process 3764 attached - interrupt to quit
> >>>>>> send(11,
> >>>>>>
> >> "\25\3\2\1\0uE\254\304\n\311\377\204}p\237O\211\322\211"...,
> >>
> >>>>>> 261, 0 <unfinished ...>
> >>>>>> Process 3764 detached
> >>>>>>
> >>>>>> FD 11 being the socket to the logging server, it never
> >>>>>>
> >> does anything
> >>
> >>>>>> asides from that send.
> >>>>>>
> >>>>>> On the server:
> >>>>>>
> >>>>>> ldap ~ # ps aux | grep rsyslog
> >>>>>> root      7370  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7379  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7380  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7381  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7382  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7383  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7384  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7385 28.3  0.0  83936  3380 ?        R    14:54   5:07
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> Stracing the 7385 pid just shows this repeating over and
> >>>>>>
> >> over (FD 4
> >>
> >>>>>> being the network socket again):
> >>>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1
> >>>>>>
> >> EAGAIN (Resource
> >>
> >>>>>> temporarily unavailable)
> >>>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1
> >>>>>>
> >> EAGAIN (Resource
> >>
> >>>>>> temporarily unavailable)
> >>>>>>
> >>>>>> It happened 3 times within a relatively short period of
> >>>>>>
> >> time, sometimes
> >>
> >>>>>> within minutes of me restarting rsyslog on both
> >>>>>>
> >> machines. I enabled
> >>
> >>>>>> debugging and it took around 8 hours to pop up again.
> >>>>>>
> >>>>>> With debugging, I pulled this from the rsyslog server:
> >>>>>>
> >>>>>> 6640.408108000:imtcp.c: New connect on NSD 0x59ddc0.
> >>>>>> 6640.408129000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550:
> >>>>>>
> >> pszRcvBuf (nil),
> >>
> >>>>>> lenRcvBuf 0
> >>>>>> 6640.408144000:imtcp.c: hasRcvInBuffer on nsd 0x59dcd0: pszRcvBuf
> >>>>>> 0x5c11c0, lenRcvBuf -1
> >>>>>> 6640.408158000:imtcp.c: hasRcvInBuffer on nsd 0x59dc10: pszRcvBuf
> >>>>>> 0x5cb510, lenRcvBuf -1
> >>>>>> 6640.408171000:imtcp.c: hasRcvInBuffer on nsd 0x5c3d70: pszRcvBuf
> >>>>>> 0x5d45c0, lenRcvBuf -1
> >>>>>> 6640.408185000:imtcp.c: hasRcvInBuffer on nsd 0x5cf460: pszRcvBuf
> >>>>>> 0x5dd010, lenRcvBuf -1
> >>>>>> 6640.408199000:imtcp.c: hasRcvInBuffer on nsd 0x5d8600: pszRcvBuf
> >>>>>> 0x5e58b0, lenRcvBuf -1
> >>>>>> 6640.408213000:imtcp.c: hasRcvInBuffer on nsd 0x59d690: pszRcvBuf
> >>>>>> 0x5ee150, lenRcvBuf -1
> >>>>>> 6640.408227000:imtcp.c: hasRcvInBuffer on nsd 0x59d2f0: pszRcvBuf
> >>>>>> 0x5f7320, lenRcvBuf -1
> >>>>>> 6640.408241000:imtcp.c: hasRcvInBuffer on nsd 0x5e8840: pszRcvBuf
> >>>>>> 0x5fe0a0, lenRcvBuf -1
> >>>>>> 6640.408255000:imtcp.c: hasRcvInBuffer on nsd 0x6027b0: pszRcvBuf
> >>>>>> 0x608350, lenRcvBuf -1
> >>>>>> 6640.408269000:imtcp.c: hasRcvInBuffer on nsd 0x5c3660: pszRcvBuf
> >>>>>> 0x612110, lenRcvBuf -1
> >>>>>> 6640.408283000:imtcp.c: hasRcvInBuffer on nsd 0x602690: pszRcvBuf
> >>>>>> 0x615040, lenRcvBuf -1
> >>>>>> 6640.408296000:imtcp.c: hasRcvInBuffer on nsd 0x5c3b60: pszRcvBuf
> >>>>>> 0x6218f0, lenRcvBuf -1
> >>>>>> 6640.408310000:imtcp.c: hasRcvInBuffer on nsd 0x60c150: pszRcvBuf
> >>>>>> 0x62a190, lenRcvBuf -1
> >>>>>> 6640.408324000:imtcp.c: hasRcvInBuffer on nsd 0x625420: pszRcvBuf
> >>>>>> 0x634fe0, lenRcvBuf -1
> >>>>>> 6640.408338000:imtcp.c: hasRcvInBuffer on nsd 0x6117b0: pszRcvBuf
> >>>>>> 0x63b000, lenRcvBuf -1
> >>>>>> 6640.408352000:imtcp.c: hasRcvInBuffer on nsd 0x603990: pszRcvBuf
> >>>>>> 0x643c70, lenRcvBuf -1
> >>>>>> 6640.408365000:imtcp.c: hasRcvInBuffer on nsd 0x633c00: pszRcvBuf
> >>>>>> 0x64dbb0, lenRcvBuf -1
> >>>>>> 6640.408379000:imtcp.c: hasRcvInBuffer on nsd 0x62cfa0: pszRcvBuf
> >>>>>> 0x650960, lenRcvBuf -1
> >>>>>> 6640.408393000:imtcp.c: hasRcvInBuffer on nsd 0x633d10: pszRcvBuf
> >>>>>> 0x65dbc0, lenRcvBuf -1
> >>>>>> 6640.408407000:imtcp.c: hasRcvInBuffer on nsd 0x5f2860: pszRcvBuf
> >>>>>> 0x666020, lenRcvBuf -1
> >>>>>> 6640.408421000:imtcp.c: hasRcvInBuffer on nsd 0x64d250: pszRcvBuf
> >>>>>> 0x66e480, lenRcvBuf -1
> >>>>>> 6640.408435000:imtcp.c: hasRcvInBuffer on nsd 0x659b30: pszRcvBuf
> >>>>>> 0x676d20, lenRcvBuf 78
> >>>>>> 6640.408448000:imtcp.c: hasRcvInBuffer on nsd 0x669fb0: pszRcvBuf
> >>>>>> 0x67fdd0, lenRcvBuf -1
> >>>>>> 6640.408462000:imtcp.c: hasRcvInBuffer on nsd 0x59a630: pszRcvBuf
> >>>>>> 0x687f10, lenRcvBuf -1
> >>>>>> 6640.408476000:imtcp.c: hasRcvInBuffer on nsd 0x6614c0: pszRcvBuf
> >>>>>> 0x6907b0, lenRcvBuf -1
> >>>>>> 6640.408490000:imtcp.c: hasRcvInBuffer on nsd 0x60c2d0: pszRcvBuf
> >>>>>> 0x699050, lenRcvBuf -1
> >>>>>> 6640.408504000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550:
> >>>>>>
> >> pszRcvBuf (nil),
> >>
> >>>>>> lenRcvBuf 0
> >>>>>>
> >>>>>> This repeats enough to generate around 29 million lines
> >>>>>>
> >> in the debug log
> >>
> >>>>>> file.
> >>>>>>
> >>>>>> In the client log, there's plenty of these which seems
> >>>>>>
> >> somewhat normal:
> >>
> >>>>>> 6509.743164000:imuxsock.c: --------imuxsock calling
> >>>>>>
> >> select, active file
> >>
> >>>>>> descriptors (max 11): 11
> >>>>>> 6509.743231000:imuxsock.c: Message from UNIX socket: #11
> >>>>>> 6509.743262000:imuxsock.c: logmsg: flags 4, from
> >>>>>>
> >> 'vanguard', msg May 20
> >>
> >>>>>> 23:15:09 bin/qmail-local[23198]: using .qmail .qmail-cfsc-forum
> >>>>>> 6509.743281000:imuxsock.c: Message has legacy syslog format.
> >>>>>> 6509.743302000:imuxsock.c: main queue: entry added, size
> >>>>>>
> >> now 579 entries
> >>
> >>>>>> 6509.743325000:imuxsock.c: wtpAdviseMaxWorkers signals busy
> >>>>>> 6509.743343000:imuxsock.c: main queue: EnqueueMsg
> >>>>>>
> >> advised worker start
> >>
> >>>>>> However comparing it to an earlier point in the log
> >>>>>>
> >> there's none of the
> >>
> >>>>>> associated tcp queue entries like this:
> >>>>>>
> >>>>>> 3587.234178000:main queue:Reg/w0: TCP sent 107 bytes,
> >>>>>>
> >> requested 107
> >>
> >>>>>> 3587.234204000:main queue:Reg/w0: main queue: entering
> >>>>>>
> >> rate limiter
> >>
> >>>>>> 3587.234229000:main queue:Reg/w0: main queue: entry
> >>>>>>
> >> deleted, state 0,
> >>
> >>>>>> size now 0 entries
> >>>>>> 3587.234255000:main queue:Reg/w0: Called action, logging
> >>>>>>
> >> to builtin-file
> >>
> >>>>>> 3587.234285000:main queue:Reg/w0:  (/var/log/maillog)
> >>>>>> 3587.234314000:main queue:Reg/w0: Called action, logging
> >>>>>>
> >> to builtin-fwd
> >>
> >>>>>> 3587.234340000:main queue:Reg/w0:  ldap.nmsrv.com
> >>>>>> 3587.234359000:main queue:Reg/w0:  ldap.nmsrv.com:10514/tcp
> >>>>>> 3587.234425000:main queue:Reg/w0: TCP sent 78 bytes, requested 78
> >>>>>> 3587.234446000:main queue:Reg/w0: main queue: entering
> >>>>>>
> >> rate limiter
> >>
> >>>>>> 3587.234471000:main queue:Reg/w0: main queue:Reg/w0: worker IDLE,
> >>>>>> waiting for work.
> >>>>>>
> >>>>>> Anyone able to shed some light on this situation? I've
> >>>>>>
> >> got the full
> >>
> >>>>>> debug logs if it's useful to anyone to take a closer look.
> >>>>>>
> >>>>>> Thanks!
> >>>>>>
> >>>>>> - Nathan
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >> --------------------------------------------------------------
> >> --------------
> >>
> >>>>
> >>>>
> >>>>> --------
> >>>>>
> >>>>>
> >>>>>> Server config file
> >>>>>>
> >>>>>>
> >>>>>>
> >> --------------------------------------------------------------
> >> --------------
> >>
> >>>>
> >>>>
> >>>>> --------
> >>>>>
> >>>>>
> >>>>>> $ModLoad immark.so # provides --MARK-- message capability
> >>>>>> $ModLoad imuxsock.so # provides support for local system
> >>>>>>
> >> logging (e.g.
> >>
> >>>>>> via logger command)
> >>>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> >>>>>>
> >>>>>> $umask 0137
> >>>>>> $DirCreateMode 0640
> >>>>>> $FileCreateMode 0640
> >>>>>> $FileOwner root
> >>>>>> $FileGroup admin
> >>>>>>
> >>>>>> $template DynFile,"/var/log/rsyslog/%HOSTNAME%/messages"
> >>>>>> $template DynFileCron,"/var/log/rsyslog/%HOSTNAME%/cron"
> >>>>>> $template DynFileSecure,"/var/log/rsyslog/%HOSTNAME%/secure"
> >>>>>> $template DynFileMail,"/var/log/rsyslog/%HOSTNAME%/maillog"
> >>>>>> $template DynFileSpool,"/var/log/rsyslog/%HOSTNAME%/spooler"
> >>>>>> $template DynFileBoot,"/var/log/rsyslog/%HOSTNAME%/boot"
> >>>>>> $template DynFileSyslog,"/var/log/rsyslog/%HOSTNAME%/syslog"
> >>>>>>
> >>>>>> :msg, contains, "no keys found for"            ~
> >>>>>> :msg, contains, "session opened for user"        ~
> >>>>>>
> >>>>>> # Log all kernel messages to the console.
> >>>>>> # Logging much else clutters up the screen.
> >>>>>> kern.*
> >>>>>>
> >> /dev/console
> >>
> >>>>>> # Log anything (except mail) of level info or higher.
> >>>>>> # Don't log private authentication messages!
> >>>>>> *.info;mail.none;cron.none                     ?DynFile
> >>>>>>
> >>>>>> *.warn;\
> >>>>>>         authpriv.none;cron.none;mail.none;news.none
> >>>>>>
> >> ?DynFileSyslog
> >>
> >>>>>> # Log all the mail messages in one place.
> >>>>>> mail.*
> >>>>>>
> >> ?DynFileMail
> >>
> >>>>>> # Log cron stuff
> >>>>>> cron.*
> >>>>>>
> >> ?DynFileCron
> >>
> >>>>>> # Everybody gets emergency messages
> >>>>>> *.emerg                                                 *
> >>>>>>
> >>>>>> # Save news errors of level crit and higher in a special file
> >>>>>> uucp,news.crit
> >>>>>>
> >> ?DynFileSpool
> >>
> >>>>>> # Save boot messages also to boot.log
> >>>>>> local7.*
> >>>>>>
> >> ?DynFileBoot
> >>
> >>>>>> # Relp config for PCI
> >>>>>> $ModLoad imrelp
> >>>>>> $InputRELPServerRun 2515
> >>>>>>
> >>>>>> # make gtls driver the default
> >>>>>> $DefaultNetstreamDriver gtls
> >>>>>>
> >>>>>> # certificate files
> >>>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
> >>>>>> $DefaultNetstreamDriverCertFile /etc/ssl/servercrt.pem
> >>>>>> $DefaultNetstreamDriverKeyFile /etc/ssl/serverkey.pem
> >>>>>>
> >>>>>> $ModLoad imtcp.so
> >>>>>>
> >>>>>> $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
> >>>>>> $InputTCPServerStreamDriverAuthMode anon # client is NOT
> >>>>>>
> >> authenticated
> >>
> >>>>>> $InputTCPServerRun 10514 # start up listener at port 10514
> >>>>>> $InputTCPMaxSessions 500
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >> --------------------------------------------------------------
> >> --------------
> >>
> >>>>
> >>>>
> >>>>> --------
> >>>>>
> >>>>>
> >>>>>> Client config file
> >>>>>>
> >>>>>>
> >>>>>>
> >> --------------------------------------------------------------
> >> --------------
> >>
> >>>>
> >>>>
> >>>>> --------
> >>>>>
> >>>>>
> >>>>>> $ModLoad imuxsock.so # provides support for local system
> >>>>>>
> >> logging (e.g.
> >>
> >>>>>> via logger command)
> >>>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> >>>>>>
> >>>>>> $umask 0137
> >>>>>> $DirCreateMode 0640
> >>>>>> $FileCreateMode 0640
> >>>>>> $FileOwner root
> >>>>>> $FileGroup admin
> >>>>>>
> >>>>>>
> >>>>>> # Log anything 'info' or higher, but lower than 'warn'.
> >>>>>> # Exclude authpriv, cron, mail, and news.  These are
> >>>>>>
> >> logged elsewhere.
> >>
> >>>>>> *.info;*.!warn;\
> >>>>>>         authpriv.none;cron.none;mail.none;news.none
> >>>>>>
> >> /var/log/messages
> >>
> >>>>>> # Log anything 'warn' or higher.
> >>>>>> # Exclude authpriv, cron, mail, and news.  These are
> >>>>>>
> >> logged elsewhere.
> >>
> >>>>>> *.warn;\
> >>>>>>         authpriv.none;cron.none;mail.none;news.none
> >>>>>>
> >> /var/log/syslog
> >>
> >>>>>> # Debugging information is logged here.
> >>>>>> *.=debug
> >>>>>>
> >> /var/log/debug
> >>
> >>>>>> # Private authentication message logging:
> >>>>>> authpriv.*
> >>>>>>
> >> /var/log/secure
> >>
> >>>>>> # Cron related logs:
> >>>>>> cron.*
> >>>>>>
> >> /var/log/cron
> >>
> >>>>>> # Mail related logs:
> >>>>>> mail.*
> >>>>>>
> >> /var/log/maillog
> >>
> >>>>>> # Emergency level messages go to all users:
> >>>>>> *.emerg                                                 *
> >>>>>>
> >>>>>>
> >>>>>> ## Uncomment these lines to use RELP instead for PCI
> >>>>>>
> >> compliance (stunnel
> >>
> >>>>>> required)
> >>>>>> #$ModLoad omrelp
> >>>>>> #*.*;mail.none :omrelp:localhost:2515;RSYSLOG_ForwardFormat
> >>>>>>
> >>>>>> # certificate files - just CA for a client
> >>>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
> >>>>>>
> >>>>>> # set up the action
> >>>>>> $DefaultNetstreamDriver gtls # use gtls netstream driver
> >>>>>> $ActionSendStreamDriverMode 1 # require TLS for the connection
> >>>>>> $ActionSendStreamDriverAuthMode anon # server is NOT
> >>>>>>
> >> authenticated
> >>
> >>>>>> *.* @@(o)ldap.nmsrv.com:10514 # send (all) messages
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> rsyslog mailing list
> >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>> http://www.rsyslog.comoften
> >>>>>>
> >>>>>>
> >>>>> --
> >>>>> Patrick Shen
> >>>>> Operations Engineer
> >>>>>
> >>>>> net mobile AG Shanghai office
> >>>>>
> >>>>> T: +86  21 61350900 - 222
> >>>>> F: +86  21 61350906
> >>>>> M: +86  13122245730
> >>>>>
> >>>>> FE55 6A7F 3192 F359 C1DF  12F6 9078 58B4 57AF 0BE3
> >>>>>
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>>>
> >>>>>
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>>>
> >>>>
> >>>
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Mon Jun 15 16:48:23 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 16:48:23 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>

Hi all,

I am having a hard time finding a suitable-fast but tickles solution to
obtaining low-precision time API under Linux. Please read here:

http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html

If I don't find a better solution, I'll probably be forced to run rsyslogd on
a tick, which would not be a good thing from a power consumption point of
view.

Comments and suggestions are highly appreciated.

Thanks,
Rainer


From epiphani at gmail.com  Mon Jun 15 16:55:57 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 15 Jun 2009 10:55:57 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
Message-ID: <e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>

I'm slightly confused - whats wrong with gettimeofday()?  It may be
higher precision, but it still
gives you the ability to find the nearest second with a few extra operations.

-Aaron

On Mon, Jun 15, 2009 at 10:48 AM, Rainer
Gerhards<rgerhards at hq.adiscon.com> wrote:
> Hi all,
>
> I am having a hard time finding a suitable-fast but tickles solution to
> obtaining low-precision time API under Linux. Please read here:
>
> http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html
>
> If I don't find a better solution, I'll probably be forced to run rsyslogd on
> a tick, which would not be a good thing from a power consumption point of
> view.
>
> Comments and suggestions are highly appreciated.
>
> Thanks,
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Mon Jun 15 16:59:21 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 16:59:21 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F937@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> Sent: Monday, June 15, 2009 4:56 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> I'm slightly confused - whats wrong with gettimeofday()?  It may be
> higher precision, but it still
> gives you the ability to find the nearest second with a few extra
operations.
> 
> -Aaron

Much too slow. For example, you can often write a sector quicker to disk than
gettimeofday() returns...

If I would base buffer-writing optimizations on gettimeofday(), the
gettimeofday() would eat up much more time than I can save with the
optimization.

Rainer

> 
> On Mon, Jun 15, 2009 at 10:48 AM, Rainer
> Gerhards<rgerhards at hq.adiscon.com> wrote:
> > Hi all,
> >
> > I am having a hard time finding a suitable-fast but tickles solution to
> > obtaining low-precision time API under Linux. Please read here:
> >
> >
http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html
> >
> > If I don't find a better solution, I'll probably be forced to run
rsyslogd
> on
> > a tick, which would not be a good thing from a power consumption point of
> > view.
> >
> > Comments and suggestions are highly appreciated.
> >
> > Thanks,
> > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Mon Jun 15 17:10:20 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 17:10:20 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>

I have updated the blogpost to hopefully provide a better description of the
situation:

http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> Sent: Monday, June 15, 2009 4:56 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> I'm slightly confused - whats wrong with gettimeofday()?  It may be
> higher precision, but it still
> gives you the ability to find the nearest second with a few extra
operations.
> 
> -Aaron
> 
> On Mon, Jun 15, 2009 at 10:48 AM, Rainer
> Gerhards<rgerhards at hq.adiscon.com> wrote:
> > Hi all,
> >
> > I am having a hard time finding a suitable-fast but tickles solution to
> > obtaining low-precision time API under Linux. Please read here:
> >
> >
http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html
> >
> > If I don't find a better solution, I'll probably be forced to run
rsyslogd
> on
> > a tick, which would not be a good thing from a power consumption point of
> > view.
> >
> > Comments and suggestions are highly appreciated.
> >
> > Thanks,
> > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From jmoyer at redhat.com  Mon Jun 15 17:13:16 2009
From: jmoyer at redhat.com (Jeff Moyer)
Date: Mon, 15 Jun 2009 11:13:16 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F937@GRFEXC.intern.adiscon.com>
	(Rainer Gerhards's message of "Mon, 15 Jun 2009 16:59:21 +0200")
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F937@GRFEXC.intern.adiscon.com>
Message-ID: <x49y6rtlecz.fsf@segfault.boston.devel.redhat.com>

"Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
>> Sent: Monday, June 15, 2009 4:56 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] high-performance, low-precision time API under
> linux?
>> 
>> I'm slightly confused - whats wrong with gettimeofday()?  It may be
>> higher precision, but it still
>> gives you the ability to find the nearest second with a few extra
> operations.
>> 
>> -Aaron
>
> Much too slow. For example, you can often write a sector quicker to disk than
> gettimeofday() returns...

Really?  That doesn't sound right to me.  Writing a sector to disk can
take 8ms or more.  One would hope gettimeofday would be faster than
that, given that it reports microseconds.  ;)  The exact resolution, of
course, depends on how your kernel is configured.  Here are the
measurements (in CPU cycles) I get running each call (time(NULL) and
gettimeofday(&tv, NULL)) 1000 times:

$ ./some-cycle-costs 
                                    fewest cycles  avg cycles
syscall(NR_time)                    2134           2721.99
syscall(NR_gettimeofday)            2156           2622.87

If I understand your original question properly, you don't need to
always have a timer running.  You can only schedule a timer when there
is I/O buffered.  I'm not sure whether your code lends itself to that or
not.

I would be interested in hearing more about what conditions cause
gettimeofday to be so slow.

Cheers,
Jeff

>> On Mon, Jun 15, 2009 at 10:48 AM, Rainer
>> Gerhards<rgerhards at hq.adiscon.com> wrote:
>> > Hi all,
>> >
>> > I am having a hard time finding a suitable-fast but tickles solution to
>> > obtaining low-precision time API under Linux. Please read here:
>> >
>> >
> http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html
>> >
>> > If I don't find a better solution, I'll probably be forced to run
> rsyslogd
>> on
>> > a tick, which would not be a good thing from a power consumption point of
>> > view.
>> >
>> > Comments and suggestions are highly appreciated.
>> >
>> > Thanks,
>> > Rainer


From epiphani at gmail.com  Mon Jun 15 17:32:13 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 15 Jun 2009 11:32:13 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
Message-ID: <e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>

I went over your post...

You shouldn't be seeing a 4ms response time on a gettimeofday() call -
this is my test:

gettimeofday({1245079355, 445434}, NULL) = 0 <0.000004>
gettimeofday({1245079355, 445503}, NULL) = 0 <0.000004>
gettimeofday({1245079355, 445560}, NULL) = 0 <0.000003>
gettimeofday({1245079355, 445629}, NULL) = 0 <0.000004>

That's 3-4 microseconds.

For the sake of argument, I checked time() as well, which is obsoleted
by gettimeofday()...

ime(NULL)                              = 1245079501 <0.000008>
time(NULL)                              = 1245079501 <0.000007>
time(NULL)                              = 1245079501 <0.000004>
time(NULL)                              = 1245079501 <0.000008>
time(NULL)                              = 1245079501 <0.000004>

About half the speed, but still in the microsecond range.

gettimeofday() is one of the fastest syscalls you can issue.
Obviously you don't want to beat on it, but I wouldn't worry about
calling it a few dozen times per second.

-Aaron


From rgerhards at hq.adiscon.com  Mon Jun 15 17:36:22 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 17:36:22 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F937@GRFEXC.intern.adiscon.com>
	<x49y6rtlecz.fsf@segfault.boston.devel.redhat.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F93B@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jeff Moyer
> Sent: Monday, June 15, 2009 5:13 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> "Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:
> 
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> >> Sent: Monday, June 15, 2009 4:56 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] high-performance, low-precision time API under
> > linux?
> >>
> >> I'm slightly confused - whats wrong with gettimeofday()?  It may be
> >> higher precision, but it still
> >> gives you the ability to find the nearest second with a few extra
> > operations.
> >>
> >> -Aaron
> >
> > Much too slow. For example, you can often write a sector quicker to disk
> than
> > gettimeofday() returns...
> 
> Really?  That doesn't sound right to me.  Writing a sector to disk can
> take 8ms or more.  One would hope gettimeofday would be faster than
> that, given that it reports microseconds.  ;)  The exact resolution, of
> course, depends on how your kernel is configured.  Here are the
> measurements (in CPU cycles) I get running each call (time(NULL) and
> gettimeofday(&tv, NULL)) 1000 times:
> 
> $ ./some-cycle-costs
>                                     fewest cycles  avg cycles
> syscall(NR_time)                    2134           2721.99
> syscall(NR_gettimeofday)            2156           2622.87

I've updated the blog post with my test program and some numbers:

http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html

(sorry for keeping this on the blog, but that makes it far easier to also ask
at other places...)

> If I understand your original question properly, you don't need to
> always have a timer running.  You can only schedule a timer when there
> is I/O buffered.  I'm not sure whether your code lends itself to that or
> not.

The code in question would not just be used for buffered writes. The problem
is also that I need to frequently re-set the timer, because it must not write
data if any new has arrived. In order to do this re-set handling properly, I
also need to know the system time.

Of course, I can schedule a specific timer while data is unwritten. But that
is just another tick processing if I don't reset the timer when there is no
need. But... all suggestions are welcome. The overall idea is that I would
like to implement something like the original callouot table. However,
callout worked on a tick. I intended to replace that with timers (more
precisely their pthreads-equivalent, which would give me some aids in awaking
the system in unexpected cases). That was the plan, but I overlooked that it
required many of the costly time() calls.

Rainer

> I would be interested in hearing more about what conditions cause
> gettimeofday to be so slow.
> 
> Cheers,
> Jeff
> 
> >> On Mon, Jun 15, 2009 at 10:48 AM, Rainer
> >> Gerhards<rgerhards at hq.adiscon.com> wrote:
> >> > Hi all,
> >> >
> >> > I am having a hard time finding a suitable-fast but tickles solution
to
> >> > obtaining low-precision time API under Linux. Please read here:
> >> >
> >> >
> >
http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html
> >> >
> >> > If I don't find a better solution, I'll probably be forced to run
> > rsyslogd
> >> on
> >> > a tick, which would not be a good thing from a power consumption point
of
> >> > view.
> >> >
> >> > Comments and suggestions are highly appreciated.
> >> >
> >> > Thanks,
> >> > Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From epiphani at gmail.com  Mon Jun 15 17:36:29 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 15 Jun 2009 11:36:29 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
Message-ID: <e7ca40f70906150836h38d3580fnb42aeef002145199@mail.gmail.com>

As more followup...

I would suggest running a timer per thread.  I assume every thread
would contain some type of
event-trigger loop, and I would make your timer calls in that loop,
caching for the remainder of
that iteration.  I'm not quite sure how well this would work, but you
can most certainly get away
with a fair bit of drift, especially if high-precision timers are not
required for log output.

-Aaron

On Mon, Jun 15, 2009 at 11:32 AM, Aaron Wiebe<epiphani at gmail.com> wrote:
> I went over your post...
>
> You shouldn't be seeing a 4ms response time on a gettimeofday() call -
> this is my test:
>
> gettimeofday({1245079355, 445434}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445503}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445560}, NULL) = 0 <0.000003>
> gettimeofday({1245079355, 445629}, NULL) = 0 <0.000004>
>
> That's 3-4 microseconds.
>
> For the sake of argument, I checked time() as well, which is obsoleted
> by gettimeofday()...
>
> ime(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000008>
> time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000007>
> time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000004>
> time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000008>
> time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000004>
>
> About half the speed, but still in the microsecond range.
>
> gettimeofday() is one of the fastest syscalls you can issue.
> Obviously you don't want to beat on it, but I wouldn't worry about
> calling it a few dozen times per second.
>
> -Aaron
>


From rgerhards at hq.adiscon.com  Mon Jun 15 17:45:55 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 17:45:55 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
	<e7ca40f70906150836h38d3580fnb42aeef002145199@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F93C@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> Sent: Monday, June 15, 2009 5:36 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> As more followup...
> 
> I would suggest running a timer per thread.  I assume every thread
> would contain some type of
> event-trigger loop, and I would make your timer calls in that loop,
> caching for the remainder of
> that iteration.  I'm not quite sure how well this would work, but you
> can most certainly get away
> with a fair bit of drift, especially if high-precision timers are not
> required for log output.

But that makes me running on that tick...

(more below)

> 
> -Aaron
> 
> On Mon, Jun 15, 2009 at 11:32 AM, Aaron Wiebe<epiphani at gmail.com> wrote:
> > I went over your post...
> >
> > You shouldn't be seeing a 4ms response time on a gettimeofday() call -
> > this is my test:
> >
> > gettimeofday({1245079355, 445434}, NULL) = 0 <0.000004>
> > gettimeofday({1245079355, 445503}, NULL) = 0 <0.000004>
> > gettimeofday({1245079355, 445560}, NULL) = 0 <0.000003>
> > gettimeofday({1245079355, 445629}, NULL) = 0 <0.000004>
> >
> > That's 3-4 microseconds.

Oops, sorry, yes, that is what I meant...

> > For the sake of argument, I checked time() as well, which is obsoleted
> > by gettimeofday()...
> >
> > ime(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000008>
> > time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000007>
> > time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000004>
> > time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000008>
> > time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000004>
> >
> > About half the speed, but still in the microsecond range.
> >
> > gettimeofday() is one of the fastest syscalls you can issue.
> > Obviously you don't want to beat on it, but I wouldn't worry about
> > calling it a few dozen times per second.

It's not about a few dozen times. I need to re-set the timer when I logically
write something to the file. The problem is that I don't know how long it was
since I did this the last time. So I would end up with calling time once per
logical write, that is a in the range of tenthousand to hunderedthousand
times per second.

A solution for this case may be to not reset any started timer and simply let
it fire whenever that be. So if there is a time started, don't care about the
other writes but do this only once the timer is done. That, of course, comes
at the cost of thread synchronization. Again, atomics may help here. And
while it is tempting to assume atomic writes for simple 32 bit values, this
is not always the case. So in the extreme case, and without (reliable)
atomics, one may end up with timers not being started where they should be.
Using sync primitives, however, is also very costly...

Rainer
> >
> > -Aaron
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From epiphani at gmail.com  Mon Jun 15 17:57:38 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 15 Jun 2009 11:57:38 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F93C@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
	<e7ca40f70906150836h38d3580fnb42aeef002145199@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F93C@GRFEXC.intern.adiscon.com>
Message-ID: <e7ca40f70906150857r53cd1b00yc0a748ba226c3a7c@mail.gmail.com>

On Mon, Jun 15, 2009 at 11:45 AM, Rainer
Gerhards<rgerhards at hq.adiscon.com> wrote:
>
> It's not about a few dozen times. I need to re-set the timer when I logically
> write something to the file. The problem is that I don't know how long it was
> since I did this the last time. So I would end up with calling time once per
> logical write, that is a in the range of tenthousand to hunderedthousand
> times per second.

I'll suggest this first, since it has the possibility of being the
best solution:  posix aio.  Have a look at `man 3 aio_write'.  That
should reduce your write() latency enough to not worry about IO
blocking and messing with the clock reasonably well.  I've used this
method in the past, and it generally works wonders - plus its not too
hard to implement.  The downsize is that it isn't very portable.

Otherwise, you're looking at batched writes and timer resets after the
batch completes.  There is nothing required in terms of delivering
messages to a file within a few seconds, so you could batch up writes
up to a high watermark / specified time prior to starting a write
cycle.

-Aaron


From rgerhards at hq.adiscon.com  Mon Jun 15 18:10:40 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 18:10:40 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com><e7ca40f70906150836h38d3580fnb42aeef002145199@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F93C@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150857r53cd1b00yc0a748ba226c3a7c@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F93D@GRFEXC.intern.adiscon.com>

OK, I've also had some other discussions. Looks like there is no quicker way
to do it. So I think I'll ditch the approach to have a generic capability to
do timeout processing and rather do specific code for each use case. As you
write, this is no problem for the file writer. I'll probably go to write
every 10 seconds or so (if there is a buffer flush request pending). I'll
evaluate the other use cases when I come along them.

Thanks everyone for commenting :)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> Sent: Monday, June 15, 2009 5:58 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> On Mon, Jun 15, 2009 at 11:45 AM, Rainer
> Gerhards<rgerhards at hq.adiscon.com> wrote:
> >
> > It's not about a few dozen times. I need to re-set the timer when I
> logically
> > write something to the file. The problem is that I don't know how long it
> was
> > since I did this the last time. So I would end up with calling time once
per
> > logical write, that is a in the range of tenthousand to hunderedthousand
> > times per second.
> 
> I'll suggest this first, since it has the possibility of being the
> best solution:  posix aio.  Have a look at `man 3 aio_write'.  That
> should reduce your write() latency enough to not worry about IO
> blocking and messing with the clock reasonably well.  I've used this
> method in the past, and it generally works wonders - plus its not too
> hard to implement.  The downsize is that it isn't very portable.
> 
> Otherwise, you're looking at batched writes and timer resets after the
> batch completes.  There is nothing required in terms of delivering
> messages to a file within a few seconds, so you could batch up writes
> up to a high watermark / specified time prior to starting a write
> cycle.
> 
> -Aaron
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From david at lang.hm  Mon Jun 15 19:01:01 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 15 Jun 2009 10:01:01 -0700 (PDT)
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906150956580.9159@asgard>

On Mon, 15 Jun 2009, Aaron Wiebe wrote:

> I went over your post...
>
> You shouldn't be seeing a 4ms response time on a gettimeofday() call -
> this is my test:
>
> gettimeofday({1245079355, 445434}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445503}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445560}, NULL) = 0 <0.000003>
> gettimeofday({1245079355, 445629}, NULL) = 0 <0.000004>
>
> That's 3-4 microseconds.
>
> For the sake of argument, I checked time() as well, which is obsoleted
> by gettimeofday()...
>
> ime(NULL)                              = 1245079501 <0.000008>
> time(NULL)                              = 1245079501 <0.000007>
> time(NULL)                              = 1245079501 <0.000004>
> time(NULL)                              = 1245079501 <0.000008>
> time(NULL)                              = 1245079501 <0.000004>
>
> About half the speed, but still in the microsecond range.
>
> gettimeofday() is one of the fastest syscalls you can issue.
> Obviously you don't want to beat on it, but I wouldn't worry about
> calling it a few dozen times per second.

this depends greatly on your specific system. linux has some wonderful 
speedups available (basicly it internally does updates only when time 
changes and the system call gets optimized to a read from memory), but 
this is not available on all systems and requires support from glibc. I 
don't remember if the limit is the CPU, 32 bit address space, kernel 
config options, or something else.

I remember seeing that without this kernel optimization a very fast 
pentium (3GHz+) took more than 1 microsecond to do a gettimeofday() call 
(tested by doing 1m calls and it took more than a second)

David Lang


From david at lang.hm  Mon Jun 15 19:12:01 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 15 Jun 2009 10:12:01 -0700 (PDT)
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906151007200.9159@asgard>

On Mon, 15 Jun 2009, Aaron Wiebe wrote:

> I went over your post...
>
> You shouldn't be seeing a 4ms response time on a gettimeofday() call -
> this is my test:
>
> gettimeofday({1245079355, 445434}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445503}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445560}, NULL) = 0 <0.000003>
> gettimeofday({1245079355, 445629}, NULL) = 0 <0.000004>
>
> That's 3-4 microseconds.
>
> For the sake of argument, I checked time() as well, which is obsoleted
> by gettimeofday()...
>
> ime(NULL)                              = 1245079501 <0.000008>
> time(NULL)                              = 1245079501 <0.000007>
> time(NULL)                              = 1245079501 <0.000004>
> time(NULL)                              = 1245079501 <0.000008>
> time(NULL)                              = 1245079501 <0.000004>
>
> About half the speed, but still in the microsecond range.
>
> gettimeofday() is one of the fastest syscalls you can issue.
> Obviously you don't want to beat on it, but I wouldn't worry about
> calling it a few dozen times per second.

Aaron,

rsyslog is able to recieve (not fully process, but recieve into the queue) 
over 300,000 messages/second from UDP without dropping any packets (256 
byte messages over gig ethernet), without maxing out the CPU on the thread 
recieving the messages. In the next week or so I hope to be able to do 
some tests with 10G ethernet ;-)

with 4 microseconds per gettimeofday() call, you can only do 250,000 
gettimeofday() calls per second. just doing a single call per message will 
take more time than you have.

David Lang


From epiphani at gmail.com  Mon Jun 15 19:19:49 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 15 Jun 2009 13:19:49 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <alpine.DEB.1.10.0906151007200.9159@asgard>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
	<alpine.DEB.1.10.0906151007200.9159@asgard>
Message-ID: <e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com>

Greetings David,

On Mon, Jun 15, 2009 at 1:12 PM, <david at lang.hm> wrote:

>> gettimeofday() is one of the fastest syscalls you can issue.
>> Obviously you don't want to beat on it, but I wouldn't worry about
>> calling it a few dozen times per second.
>
> Aaron,
>
> rsyslog is able to recieve (not fully process, but recieve into the queue)
> over 300,000 messages/second from UDP without dropping any packets (256
> byte messages over gig ethernet), without maxing out the CPU on the thread
> recieving the messages. In the next week or so I hope to be able to do
> some tests with 10G ethernet ;-)
>
> with 4 microseconds per gettimeofday() call, you can only do 250,000
> gettimeofday() calls per second. just doing a single call per message will
> take more time than you have.

I never meant to suggest that doing a gettimeofday() on every received
message made sense.  My point was that a gettimeofday() call is a very
lightweight system call - one of the lightest.  Using a light system
call is of course no replacement for proper application design, and
Rainers' question was a valid one.  Unfortunately, there isn't much in
the way of changes to the system calls being used that would improve
the execution.  (Though dropping in gettimeofday() instead of time()
would probably be a good idea, since time() is documented as being
obsolete to gettimeofday().)

-Aaron


From Thomas.Mieslinger at 1und1.de  Mon Jun 15 19:57:43 2009
From: Thomas.Mieslinger at 1und1.de (Thomas Mieslinger)
Date: Mon, 15 Jun 2009 19:57:43 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com><alpine.DEB.1.10.0906151007200.9159@asgard>
	<e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com>
Message-ID: <3DBDD4313FFFEA4F86E79AEF498136C3079B636D@WEBEXCHANGE.webde.local>

BTW: In the Informix DBMS the Software Architects decided to implement an OS independent timer thread to work arround (very costly in the late 90s) gettimeofday() calls.

I've run the code on some Solaris 10 boxes, because they were known for their slow gettimeofday() implementation

S11 snv_105 Pentium 4 3GHz 
time ./a.out 100000
real    0m0.087s
user    0m0.082s
sys     0m0.004s

S10 Generic_138889-03 Athlon 2,2GHz
time ./a.out 100000
real    0m0.028s
user    0m0.026s
sys     0m0.002s

ok, lets try this on an older boxes...
suse70 2.2.21 libc-2.1.3-141 2xP3 800
time ./a.out 100000

real    0m0.100s
user    0m0.080s
sys     0m0.020s

So, what are you using as a development box?


From rgerhards at hq.adiscon.com  Mon Jun 15 20:23:17 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 20:23:17 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com><alpine.DEB.1.10.0906151007200.9159@asgard><e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com>
	<3DBDD4313FFFEA4F86E79AEF498136C3079B636D@WEBEXCHANGE.webde.local>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F93E@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Thomas Mieslinger
> Sent: Monday, June 15, 2009 7:58 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> BTW: In the Informix DBMS the Software Architects decided to implement an
OS
> independent timer thread to work arround (very costly in the late 90s)
> gettimeofday() calls.
> 
> I've run the code on some Solaris 10 boxes, because they were known for
their
> slow gettimeofday() implementation
> 
> S11 snv_105 Pentium 4 3GHz
> time ./a.out 100000
> real    0m0.087s
> user    0m0.082s
> sys     0m0.004s
> 
> S10 Generic_138889-03 Athlon 2,2GHz
> time ./a.out 100000
> real    0m0.028s
> user    0m0.026s
> sys     0m0.002s
> 
> ok, lets try this on an older boxes...
> suse70 2.2.21 libc-2.1.3-141 2xP3 800
> time ./a.out 100000
> 
> real    0m0.100s
> user    0m0.080s
> sys     0m0.020s
> 
> So, what are you using as a development box?

I should have mentioned that I am typically running inside a virtualized
environment as I routinely need to run three to four machines in parallel for
testing.

If anyone is up for hardware contributions, just let me know ;)

Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From david at lang.hm  Mon Jun 15 20:27:15 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 15 Jun 2009 11:27:15 -0700 (PDT)
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F93E@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com><alpine.DEB.1.10.0906151007200.9159@asgard><e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com>
	<3DBDD4313FFFEA4F86E79AEF498136C3079B636D@WEBEXCHANGE.webde.local>
	<9B6E2A8877C38245BFB15CC491A11DA706F93E@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906151126180.9159@asgard>

On Mon, 15 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Thomas Mieslinger
>> Sent: Monday, June 15, 2009 7:58 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] high-performance, low-precision time API under
> linux?
>>
>> BTW: In the Informix DBMS the Software Architects decided to implement an
> OS
>> independent timer thread to work arround (very costly in the late 90s)
>> gettimeofday() calls.
>>
>> I've run the code on some Solaris 10 boxes, because they were known for
> their
>> slow gettimeofday() implementation
>>
>> S11 snv_105 Pentium 4 3GHz
>> time ./a.out 100000
>> real    0m0.087s
>> user    0m0.082s
>> sys     0m0.004s
>>
>> S10 Generic_138889-03 Athlon 2,2GHz
>> time ./a.out 100000
>> real    0m0.028s
>> user    0m0.026s
>> sys     0m0.002s
>>
>> ok, lets try this on an older boxes...
>> suse70 2.2.21 libc-2.1.3-141 2xP3 800
>> time ./a.out 100000
>>
>> real    0m0.100s
>> user    0m0.080s
>> sys     0m0.020s
>>
>> So, what are you using as a development box?
>
> I should have mentioned that I am typically running inside a virtualized
> environment as I routinely need to run three to four machines in parallel for
> testing.

ahh, virtualized boxes are especially bad for gettimeofday().
which virtualization are you using?

David Lang

> If anyone is up for hardware contributions, just let me know ;)
>
> Rainer
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Mon Jun 15 20:31:42 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 20:31:42 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com><alpine.DEB.1.10.0906151007200.9159@asgard><e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com><3DBDD4313FFFEA4F86E79AEF498136C3079B636D@WEBEXCHANGE.webde.local><9B6E2A8877C38245BFB15CC491A11DA706F93E@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906151126180.9159@asgard>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F93F@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Monday, June 15, 2009 8:27 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> On Mon, 15 Jun 2009, Rainer Gerhards wrote:
> 
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Thomas Mieslinger
> >> Sent: Monday, June 15, 2009 7:58 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] high-performance, low-precision time API under
> > linux?
> >>
> >> BTW: In the Informix DBMS the Software Architects decided to implement
an
> > OS
> >> independent timer thread to work arround (very costly in the late 90s)
> >> gettimeofday() calls.
> >>
> >> I've run the code on some Solaris 10 boxes, because they were known for
> > their
> >> slow gettimeofday() implementation
> >>
> >> S11 snv_105 Pentium 4 3GHz
> >> time ./a.out 100000
> >> real    0m0.087s
> >> user    0m0.082s
> >> sys     0m0.004s
> >>
> >> S10 Generic_138889-03 Athlon 2,2GHz
> >> time ./a.out 100000
> >> real    0m0.028s
> >> user    0m0.026s
> >> sys     0m0.002s
> >>
> >> ok, lets try this on an older boxes...
> >> suse70 2.2.21 libc-2.1.3-141 2xP3 800
> >> time ./a.out 100000
> >>
> >> real    0m0.100s
> >> user    0m0.080s
> >> sys     0m0.020s
> >>
> >> So, what are you using as a development box?
> >
> > I should have mentioned that I am typically running inside a virtualized
> > environment as I routinely need to run three to four machines in parallel
> for
> > testing.
> 
> ahh, virtualized boxes are especially bad for gettimeofday().
> which virtualization are you using?

This one is using vmware, but that doesn't really matter. Even on a real box,
as you said, the toll to pay for gettimeofday() is high compared to the rest
of the workload. So I actually do not care how much the absolute time is but
rather how it goes into the overall mix. The context switch itself is
expensive enough, so this makes everything that goes to kernel space rather
expensive.

As I had written earlier, I will probably give up the approach of a
generalized timeout handler in favor of many targeted implementations. It
would have been nice to have this callout-tablish facility, but it looks like
it is too expensive to implement.

Rainer

> 
> David Lang
> 
> > If anyone is up for hardware contributions, just let me know ;)
> >
> > Rainer
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Wed Jun 17 09:25:20 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 17 Jun 2009 09:25:20 +0200
Subject: [rsyslog] General availability of atomic operations?
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F96E@GRFEXC.intern.adiscon.com>

Hi all,

I am currently facing a situation where I can re-structure message object
represenation in a very efficient way if I have atomic operations available.
However, if they are not available, the very same implementation would have a
heavy hit on concurrency and overhead.

What I need is atomic increments and decrements (for reference counting). So
I wonder if support for these operations is available on a sufficiently large
number of platforms. I would appreciate feedback on this issue, including
pointers to place where else I could put this question.

Thanks,
Rainer	


From thomas.mieslinger at 1und1.de  Wed Jun 17 10:15:18 2009
From: thomas.mieslinger at 1und1.de (Thomas Mieslinger)
Date: Wed, 17 Jun 2009 10:15:18 +0200
Subject: [rsyslog] General availability of atomic operations?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F96E@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F96E@GRFEXC.intern.adiscon.com>
Message-ID: <4A38A616.1010408@1und1.de>

Rainer Gerhards wrote:
> What I need is atomic increments and decrements (for reference counting). So
> I wonder if support for these operations is available on a sufficiently large
> number of platforms. I would appreciate feedback on this issue, including
> pointers to place where else I could put this question.

The Apache Portable Runtime includes atomic read/set/add/inc/dec 
implementations for many platforms. But for example the sparc 
implementation has been riped out because of unclear copyrights.

-- 
Thomas Mieslinger


From martinmie at PartyGaming.com  Thu Jun 18 14:18:04 2009
From: martinmie at PartyGaming.com (Martin Mielke)
Date: Thu, 18 Jun 2009 14:18:04 +0200
Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
Message-ID: <0B1B9163138571439EAF804646F3F6AA0767973E@GIBSVWIN004X.partygaming.local>

Hi all,

I need to find a way to either convert or import syslog-ng.conf files
into rsyslog or learn the syntax to classify/store the logfiles
depending on the source.

For example, taken from a syslog-ng file:
--
source s_network { udp(); tcp(); };

filter f_systems {
host("1\.2\.3\.\d+");
};

filter f_switches {
host("5\.6\.7\.8|5\.6\.7\.9");
};

destination d_systems {file("/systems.log"); };
destination d_switches {file("/switches.log"); };

log { source(s_network); filter(f_systems); destination(d_systems); };
log { source(s_network); filter(f_switches); destination(d_switches); };
--

I have read the online documentation but I'm unable to find the right
answer...

Thanks!

Martin


This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. 

Any views or opinions are solely those of the sender.  This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk.




From rgerhards at hq.adiscon.com  Thu Jun 18 15:11:27 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 18 Jun 2009 15:11:27 +0200
Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
References: <0B1B9163138571439EAF804646F3F6AA0767973E@GIBSVWIN004X.partygaming.local>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9A4@GRFEXC.intern.adiscon.com>

FYI, there was also a forum post:

http://kb.monitorware.com/syslog-filters-and-destinations-under-rsyslog-t9858
.html

I'd appreciate if any other responses also go to the forum (so that others,
when searching, find the complete info).

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Martin Mielke
> Sent: Thursday, June 18, 2009 2:18 PM
> To: rsyslog-users
> Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
> 
> Hi all,
> 
> I need to find a way to either convert or import syslog-ng.conf files
> into rsyslog or learn the syntax to classify/store the logfiles
> depending on the source.
> 
> For example, taken from a syslog-ng file:
> --
> source s_network { udp(); tcp(); };
> 
> filter f_systems {
> host("1\.2\.3\.\d+");
> };
> 
> filter f_switches {
> host("5\.6\.7\.8|5\.6\.7\.9");
> };
> 
> destination d_systems {file("/systems.log"); };
> destination d_switches {file("/switches.log"); };
> 
> log { source(s_network); filter(f_systems); destination(d_systems); };
> log { source(s_network); filter(f_switches); destination(d_switches); };
> --
> 
> I have read the online documentation but I'm unable to find the right
> answer...
> 
> Thanks!
> 
> Martin
> 
> 
> This email and any attachments are confidential, and may be legally
privileged
> and protected by copyright. If you are not the intended recipient
> dissemination or copying of this email is prohibited. If you have received
> this in error, please notify the sender by replying by email and then
delete
> the email completely from your system.
> 
> Any views or opinions are solely those of the sender.  This communication
is
> not intended to form a binding contract unless expressly indicated to the
> contrary and properly authorised. Any actions taken on the basis of this
email
> are at the recipient's own risk.
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From alex at samad.com.au  Fri Jun 19 10:05:01 2009
From: alex at samad.com.au (Alex Samad)
Date: Fri, 19 Jun 2009 18:05:01 +1000
Subject: [rsyslog] Help with configurin rsyslog
Message-ID: <20090619080501.GB11392@samad.com.au>

Hi

I have a syslog line that looks like this

Jun 18 10:35:53 Thu Jun 18 10:42:37 2009 dlink-dir-855 System Log:
Stored configuration to non-volatile memory

notice the double dates stamps and it seems like I should be able to
rectify this with rsyslog, but I am not sure how.

I have had a quick look at the doco and the wiki, but there aren't any
example's of replacing/reformatting from specific hosts

note the above line is from a dlink dir-855 router/access point

I also have these
Jun 18 10:55:00 127.0.0.1 time="2009-06-18 10:55:00" msg="LOGIN User
logged in on TELNET (192.168.1.100)"

from a speedtouch 546v6 then I need to munge.

Any pointers or working examples would be helpful

Thanks


From rgerhards at hq.adiscon.com  Fri Jun 19 18:14:20 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Jun 2009 18:14:20 +0200
Subject: [rsyslog] memory alloc problems - any advise?
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>

Hi all,

during testing, I found out that some strange things happen when a queue is
set to large size. If I set e.g. the main message queue to 500,000 entries
and then shuffle messages very quickly at rsyslogd, it consumes considerable
memory. This is not surprising, as it is buffering not yet-processed
messages.

What is surprising is that the memory is not given back to the OS when the
data has been processed. Well... I know that some data blocks may still be
held and thus prevent giving memory back. But that should be solved when a
few new messages come in. Still, nothing changes in that scenario. What
puzzles me even more is that when I start a new burst of messages (I send
around one million each), it looks like almost only new memory is obtained
from the OS for them. At least here I had expected that the allocator would
re-use what was already claimed.

The interesting things is that when the queue size is small (50,000 message
for example), this behavior does not occur. Then, memory allocation and
deallocation works as expected.

I have also run the valgrind memory debugger on the app, and it does not
report any leaked memory.

If someone has an explanation of what causes this behavior, I would really
love to hear about it...

Thanks,
Rainer


From aoz.syn at gmail.com  Fri Jun 19 18:26:46 2009
From: aoz.syn at gmail.com (RB)
Date: Fri, 19 Jun 2009 10:26:46 -0600
Subject: [rsyslog] memory alloc problems - any advise?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
Message-ID: <4255c2570906190926v2399fcc7y7feddb010009faf5@mail.gmail.com>

On Fri, Jun 19, 2009 at 10:14, Rainer Gerhards<rgerhards at hq.adiscon.com> wrote:
> What is surprising is that the memory is not given back to the OS when the
> data has been processed. Well... I know that some data blocks may still be
> held and thus prevent giving memory back. But that should be solved when a
> few new messages come in. Still, nothing changes in that scenario. What
> puzzles me even more is that when I start a new burst of messages (I send
> around one million each), it looks like almost only new memory is obtained
> from the OS for them. At least here I had expected that the allocator would
> re-use what was already claimed.

Can you clarify what you mean by "not given back to the OS"?  Do you
mean that the RSS of rsyslog remains the same, or that the system
'free' memory does not go back up, or something else entirely?  I'm
wondering if you're seeing caching effects.


From rgerhards at hq.adiscon.com  Fri Jun 19 18:36:05 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Jun 2009 18:36:05 +0200
Subject: [rsyslog] memory alloc problems - any advise?
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
	<4255c2570906190926v2399fcc7y7feddb010009faf5@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9E0@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of RB
> Sent: Friday, June 19, 2009 6:27 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> 
> On Fri, Jun 19, 2009 at 10:14, Rainer Gerhards<rgerhards at hq.adiscon.com>
> wrote:
> > What is surprising is that the memory is not given back to the OS when
the
> > data has been processed. Well... I know that some data blocks may still
be
> > held and thus prevent giving memory back. But that should be solved when
a
> > few new messages come in. Still, nothing changes in that scenario. What
> > puzzles me even more is that when I start a new burst of messages (I send
> > around one million each), it looks like almost only new memory is
obtained
> > from the OS for them. At least here I had expected that the allocator
would
> > re-use what was already claimed.
> 
> Can you clarify what you mean by "not given back to the OS"?  Do you
> mean that the RSS of rsyslog remains the same, or that the system
> 'free' memory does not go back up, or something else entirely?  I'm
> wondering if you're seeing caching effects.

Rss remains the same, looks like sbrk() is not called. I poked somemore
around, and it looks like I need to do a malloc_trim() call in glibc to solve
this issue. Will post some stats soon...

Rainer 


From rgerhards at hq.adiscon.com  Fri Jun 19 18:57:06 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Jun 2009 18:57:06 +0200
Subject: [rsyslog] memory alloc problems - any advise?
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com><4255c2570906190926v2399fcc7y7feddb010009faf5@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E0@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9E1@GRFEXC.intern.adiscon.com>

The promised stats. I used GNU's malloc_stats() and malloc_trim().

I queried stats every 5 seconds (lines with # are my comments):

#Right after startup:
Arena 0:
system bytes     =     245760
in use bytes     =     244624
Total (incl. mmap):
system bytes     =    4247552
in use bytes     =    4246416
max mmap regions =          1
max mmap bytes   =    4001792

#First burst:
Arena 0:
system bytes     =  108318720
in use bytes     =  107797520
Arena 1:
system bytes     =   67661824
in use bytes     =   18954784
Total (incl. mmap):
system bytes     =  179982336
in use bytes     =  130754096
max mmap regions =          1
max mmap bytes   =    4001792

#processing...
Arena 0:
system bytes     =  143970304
in use bytes     =   93414224
Arena 1:
system bytes     =   91770880
in use bytes     =       2336
Arena 2:
system bytes     =  106659840
in use bytes     =       2336
Total (incl. mmap):
system bytes     =  346402816
in use bytes     =   97420688
max mmap regions =          1
max mmap bytes   =    4001792

#idle
Arena 0:
system bytes     =  143970304
in use bytes     =     255248
Arena 1:
system bytes     =   91770880
in use bytes     =       2288
Arena 2:
system bytes     =  106659840
in use bytes     =       2288
Total (incl. mmap):
system bytes     =  346402816
in use bytes     =    4261616
max mmap regions =          1
max mmap bytes   =    4001792

#second burst - so it looks like the memory is reused (good!)
Arena 0:
system bytes     =  143970304
in use bytes     =  102348464
Arena 1:
system bytes     =   91770880
in use bytes     =   14172352
Arena 2:
system bytes     =  106659840
in use bytes     =       3840
Total (incl. mmap):
system bytes     =  346402816
in use bytes     =  120526448
max mmap regions =          1
max mmap bytes   =    4001792

# first and second burst were equal, same messages, same amount of them.
# Still not all allocs are re-used (they must have been identical, too). 
# As it looks > 50% were not reused...
Arena 0:
system bytes     =  206884864
in use bytes     =  206110592
Arena 1:
system bytes     =   91770880
in use bytes     =       2336
Arena 2:
system bytes     =  162009088
in use bytes     =       2400
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =  210117120
max mmap regions =          1
max mmap bytes   =    4001792

#idle again
Arena 0:
system bytes     =  206884864
in use bytes     =     256000
Arena 1:
system bytes     =   91770880
in use bytes     =       3456
Arena 2:
system bytes     =  162009088
in use bytes     =       2352
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =    4263600
max mmap regions =          1
max mmap bytes   =    4001792

# still idle 
Arena 0:
system bytes     =  206884864
in use bytes     =     256000
Arena 1:
system bytes     =   91770880
in use bytes     =       3456
Arena 2:
system bytes     =  162009088
in use bytes     =       2352
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =    4263600
max mmap regions =          1
max mmap bytes   =    4001792

# still idle
Arena 0:
system bytes     =  206884864
in use bytes     =     256000
Arena 1:
system bytes     =   91770880
in use bytes     =       3456
Arena 2:
system bytes     =  162009088
in use bytes     =       2352
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =    4263600
max mmap regions =          1
max mmap bytes   =    4001792

# I did a malloc_trim() every 10th stat counter and here it happens.
# The number still look the same, but in the OS system monitor, I see freed
# memory (and it actually is freed if I look at how the system then
performs).
# So it looks like only the virtual address space was kept?
malloc trim (1)!
Arena 0:
system bytes     =  206884864
in use bytes     =     256000
Arena 1:
system bytes     =   91770880
in use bytes     =       3456
Arena 2:
system bytes     =  162009088
in use bytes     =       2352
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =    4263600
max mmap regions =          1
max mmap bytes   =    4001792

# no activity, so it remains at this level...
Arena 0:
system bytes     =  206884864
in use bytes     =     256000
Arena 1:
system bytes     =   91770880
in use bytes     =       3456
Arena 2:
system bytes     =  162009088
in use bytes     =       2352
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =    4263600
max mmap regions =          1
max mmap bytes   =    4001792

With that information, I'll probably better turn to a GNU mailing list. Still
I wonder if someone here has an opinion. If so, please let me know. For the
time being, it looks like I need to do some period malloc_trim() calls...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Friday, June 19, 2009 6:36 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of RB
> > Sent: Friday, June 19, 2009 6:27 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] memory alloc problems - any advise?
> >
> > On Fri, Jun 19, 2009 at 10:14, Rainer Gerhards<rgerhards at hq.adiscon.com>
> > wrote:
> > > What is surprising is that the memory is not given back to the OS when
> the
> > > data has been processed. Well... I know that some data blocks may still
> be
> > > held and thus prevent giving memory back. But that should be solved
when
> a
> > > few new messages come in. Still, nothing changes in that scenario. What
> > > puzzles me even more is that when I start a new burst of messages (I
send
> > > around one million each), it looks like almost only new memory is
> obtained
> > > from the OS for them. At least here I had expected that the allocator
> would
> > > re-use what was already claimed.
> >
> > Can you clarify what you mean by "not given back to the OS"?  Do you
> > mean that the RSS of rsyslog remains the same, or that the system
> > 'free' memory does not go back up, or something else entirely?  I'm
> > wondering if you're seeing caching effects.
> 
> Rss remains the same, looks like sbrk() is not called. I poked somemore
> around, and it looks like I need to do a malloc_trim() call in glibc to
solve
> this issue. Will post some stats soon...
> 
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From david at lang.hm  Fri Jun 19 22:14:16 2009
From: david at lang.hm (david at lang.hm)
Date: Fri, 19 Jun 2009 13:14:16 -0700 (PDT)
Subject: [rsyslog] Help with configurin rsyslog
In-Reply-To: <20090619080501.GB11392@samad.com.au>
References: <20090619080501.GB11392@samad.com.au>
Message-ID: <alpine.DEB.1.10.0906191305150.20154@asgard.lang.hm>

On Fri, 19 Jun 2009, Alex Samad wrote:

> I have a syslog line that looks like this
>
> Jun 18 10:35:53 Thu Jun 18 10:42:37 2009 dlink-dir-855 System Log:
> Stored configuration to non-volatile memory
>
> notice the double dates stamps and it seems like I should be able to
> rectify this with rsyslog, but I am not sure how.
>
> I have had a quick look at the doco and the wiki, but there aren't any
> example's of replacing/reformatting from specific hosts
>
> note the above line is from a dlink dir-855 router/access point
>
> I also have these
> Jun 18 10:55:00 127.0.0.1 time="2009-06-18 10:55:00" msg="LOGIN User
> logged in on TELNET (192.168.1.100)"
>
> from a speedtouch 546v6 then I need to munge.
>
> Any pointers or working examples would be helpful

I think that the key approach to do this is to create additional formats 
and then do conditional logging

something like the following pseudocode

if hostip = '1.1.1.1' then /var/log/messages;CustomFormat1
if hostip != '1.1.1.1' then /var/log/messages;StandardFormat

the first issue is to figure out how to filter the different messages 
seperate from each other

depending on how badly the message is misformed on the wire this may be 
hard to do. if you do not go through any relay servers you can filter 
based on the source IP of the connection to rsyslog

after you get the different logs filtered from each other you can figure 
out how to reformat each type of message.

hopefully the 'odd' messages only comes from a small number of sources so 
that you can do conditionals for the specific sources reasonably. (rsyslog 
doesn't currently have a mechanism to effectivly do table lookups)

I hope this helps.

David Lang


From david at lang.hm  Fri Jun 19 22:29:30 2009
From: david at lang.hm (david at lang.hm)
Date: Fri, 19 Jun 2009 13:29:30 -0700 (PDT)
Subject: [rsyslog] memory alloc problems - any advise?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906191324100.20154@asgard.lang.hm>

On Fri, 19 Jun 2009, Rainer Gerhards wrote:

> Hi all,
>
> during testing, I found out that some strange things happen when a queue is
> set to large size. If I set e.g. the main message queue to 500,000 entries
> and then shuffle messages very quickly at rsyslogd, it consumes considerable
> memory. This is not surprising, as it is buffering not yet-processed
> messages.
>
> What is surprising is that the memory is not given back to the OS when the
> data has been processed. Well... I know that some data blocks may still be
> held and thus prevent giving memory back. But that should be solved when a
> few new messages come in. Still, nothing changes in that scenario. What
> puzzles me even more is that when I start a new burst of messages (I send
> around one million each), it looks like almost only new memory is obtained
> from the OS for them. At least here I had expected that the allocator would
> re-use what was already claimed.
>
> The interesting things is that when the queue size is small (50,000 message
> for example), this behavior does not occur. Then, memory allocation and
> deallocation works as expected.

hmm, this doesn't match what I was seeing with the 3.x series. I had a 
machine with 32G of ram and was setting the max queue size to 1M entries.

I don't remember if it was freeing the memory as the queue emptied, but I 
know that I could hit it repeatedly with large peaks and it would not use 
additional memory.

> I have also run the valgrind memory debugger on the app, and it does not
> report any leaked memory.

remember that it will only consider it a leak if you no longer have _any_ 
refrence to the memory. if you have some pointer to it stashed away 
somewhere that you won't use it, it won't consider it a leak.

early next week I should have some new boxes in that I can do high-speed 
tests on again (64G ram with 10G network)

David Lang

> If someone has an explanation of what causes this behavior, I would really
> love to hear about it...
>
> Thanks,
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Fri Jun 19 22:34:48 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Jun 2009 22:34:48 +0200
Subject: [rsyslog] memory alloc problems - any advise?
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906191324100.20154@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9E3@GRFEXC.intern.adiscon.com>


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, June 19, 2009 10:30 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> 
> On Fri, 19 Jun 2009, Rainer Gerhards wrote:
> 
> > Hi all,
> >
> > during testing, I found out that some strange things happen 
> when a queue is
> > set to large size. If I set e.g. the main message queue to 
> 500,000 entries
> > and then shuffle messages very quickly at rsyslogd, it 
> consumes considerable
> > memory. This is not surprising, as it is buffering not yet-processed
> > messages.
> >
> > What is surprising is that the memory is not given back to 
> the OS when the
> > data has been processed. Well... I know that some data 
> blocks may still be
> > held and thus prevent giving memory back. But that should 
> be solved when a
> > few new messages come in. Still, nothing changes in that 
> scenario. What
> > puzzles me even more is that when I start a new burst of 
> messages (I send
> > around one million each), it looks like almost only new 
> memory is obtained
> > from the OS for them. At least here I had expected that the 
> allocator would
> > re-use what was already claimed.
> >
> > The interesting things is that when the queue size is small 
> (50,000 message
> > for example), this behavior does not occur. Then, memory 
> allocation and
> > deallocation works as expected.
> 
> hmm, this doesn't match what I was seeing with the 3.x 
> series. I had a 
> machine with 32G of ram and was setting the max queue size to 
> 1M entries.

I checked with v3, and it had the same behavior (a little less strong, but
still).

> 
> I don't remember if it was freeing the memory as the queue 
> emptied, but I 
> know that I could hit it repeatedly with large peaks and it 
> would not use 
> additional memory.
> 
> > I have also run the valgrind memory debugger on the app, 
> and it does not
> > report any leaked memory.
> 
> remember that it will only consider it a leak if you no 
> longer have _any_ 
> refrence to the memory. if you have some pointer to it stashed away 
> somewhere that you won't use it, it won't consider it a leak.

Valgrind can print out those allocations. In short, there were nothing
unexpected (the usual 81 when rsyslogd terminates, and they pose no problem).
As you can see in my later posts, the issue seems actually to be rooted in
glibc.

> 
> early next week I should have some new boxes in that I can do 
> high-speed 
> tests on again (64G ram with 10G network)

That's good news. I did a lot of optimizations this week, and think I can
release a new version with them mid next week. I hope you will see even
further improvement. In the v5 engine, I think, I have discovered a memory
leak, but I didn't investigate today. Maybe just the same issue I've later
found wiht the malloc calls...

Rainer

> 
> David Lang
> 
> > If someone has an explanation of what causes this behavior, 
> I would really
> > love to hear about it...
> >
> > Thanks,
> > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 


From david at lang.hm  Fri Jun 19 23:36:31 2009
From: david at lang.hm (david at lang.hm)
Date: Fri, 19 Jun 2009 14:36:31 -0700 (PDT)
Subject: [rsyslog] memory alloc problems - any advise?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9E3@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906191324100.20154@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E3@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906191352480.20154@asgard.lang.hm>

On Fri, 19 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Fri, 19 Jun 2009, Rainer Gerhards wrote:
>>
>>> Hi all,
>>>
>>> during testing, I found out that some strange things happen
>> when a queue is
>>> set to large size. If I set e.g. the main message queue to
>> 500,000 entries
>>> and then shuffle messages very quickly at rsyslogd, it
>> consumes considerable
>>> memory. This is not surprising, as it is buffering not yet-processed
>>> messages.
>>>
>>> What is surprising is that the memory is not given back to
>> the OS when the
>>> data has been processed. Well... I know that some data
>> blocks may still be
>>> held and thus prevent giving memory back. But that should
>> be solved when a
>>> few new messages come in. Still, nothing changes in that
>> scenario. What
>>> puzzles me even more is that when I start a new burst of
>> messages (I send
>>> around one million each), it looks like almost only new
>> memory is obtained
>>> from the OS for them. At least here I had expected that the
>> allocator would
>>> re-use what was already claimed.
>>>
>>> The interesting things is that when the queue size is small
>> (50,000 message
>>> for example), this behavior does not occur. Then, memory
>> allocation and
>>> deallocation works as expected.
>>
>> hmm, this doesn't match what I was seeing with the 3.x
>> series. I had a
>> machine with 32G of ram and was setting the max queue size to
>> 1M entries.
>
> I checked with v3, and it had the same behavior (a little less strong, but
> still).

interesting.

when I was doing the tests that showed rsyslog able to recieve at gig-e 
line speed I would run tests repeatedly without any problems (with each 
test sending millions of logs, things sitting idle for hours, then another 
burst of millions of logs, repeat)

if I set the queue size too large for my ram the system would start paging 
and die, so I'm sure that it wasn't failing to release memory.

now, one thing could be that my tests were of messages that were all the 
same size, it could be that you are seeing some effect from fragmentation 
with different message size.

David Lang

>>
>> I don't remember if it was freeing the memory as the queue
>> emptied, but I
>> know that I could hit it repeatedly with large peaks and it
>> would not use
>> additional memory.
>>
>>> I have also run the valgrind memory debugger on the app,
>> and it does not
>>> report any leaked memory.
>>
>> remember that it will only consider it a leak if you no
>> longer have _any_
>> refrence to the memory. if you have some pointer to it stashed away
>> somewhere that you won't use it, it won't consider it a leak.
>
> Valgrind can print out those allocations. In short, there were nothing
> unexpected (the usual 81 when rsyslogd terminates, and they pose no problem).
> As you can see in my later posts, the issue seems actually to be rooted in
> glibc.
>
>>
>> early next week I should have some new boxes in that I can do
>> high-speed
>> tests on again (64G ram with 10G network)
>
> That's good news. I did a lot of optimizations this week, and think I can
> release a new version with them mid next week. I hope you will see even
> further improvement. In the v5 engine, I think, I have discovered a memory
> leak, but I didn't investigate today. Maybe just the same issue I've later
> found wiht the malloc calls...
>
> Rainer
>
>>
>> David Lang
>>
>>> If someone has an explanation of what causes this behavior,
>> I would really
>>> love to hear about it...
>>>
>>> Thanks,
>>> Rainer
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From jiri.hlinka at gmail.com  Sat Jun 20 21:47:45 2009
From: jiri.hlinka at gmail.com (=?ISO-8859-2?Q?Ji=F8=ED_Hlinka?=)
Date: Sat, 20 Jun 2009 21:47:45 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
Message-ID: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com>

Hello,
is there any info about what rsyslog version will be available in RHEL
6? I suppose it will be 3.x, but im not sure, maybe 4.x version in the
time "RHEL6 package freeze" will be in stable status?

Thank you,
Jiri H.


From rgerhards at hq.adiscon.com  Sun Jun 21 11:58:57 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 21 Jun 2009 11:58:57 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>

That a good question and I, too, would like to know the answer. In any case,
I hope it will not be v2. Giving that RHEL is a conservative distribution
(and it has to be for the target base), I would assume that if they go for a
newer version, it is v3.

Rainer 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
> Sent: Saturday, June 20, 2009 9:48 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] RSyslog version in RHEL 6
> 
> Hello,
> is there any info about what rsyslog version will be available in RHEL
> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in the
> time "RHEL6 package freeze" will be in stable status?
> 
> Thank you,
> Jiri H.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 


From david at lang.hm  Sun Jun 21 12:15:12 2009
From: david at lang.hm (david at lang.hm)
Date: Sun, 21 Jun 2009 03:15:12 -0700 (PDT)
Subject: [rsyslog] RSyslog version in RHEL 6
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906210309480.23743@asgard.lang.hm>

On Sun, 21 Jun 2009, Rainer Gerhards wrote:

> That a good question and I, too, would like to know the answer. In any case,
> I hope it will not be v2. Giving that RHEL is a conservative distribution
> (and it has to be for the target base), I would assume that if they go for a
> newer version, it is v3.

not knowing when the freeze date is it's hard to say.

remember that the huge performance benifits of v4 (and v5 when it comes 
out) would also be very useful for the target base.

not to mention all the subtle fixes you have done whiel pursuing 
performance.

David Lang

> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
>> Sent: Saturday, June 20, 2009 9:48 PM
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] RSyslog version in RHEL 6
>>
>> Hello,
>> is there any info about what rsyslog version will be available in RHEL
>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in the
>> time "RHEL6 package freeze" will be in stable status?
>>
>> Thank you,
>> Jiri H.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Sun Jun 21 14:54:49 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 21 Jun 2009 14:54:49 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906210309480.23743@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9E6@GRFEXC.intern.adiscon.com>


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Sunday, June 21, 2009 12:15 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] RSyslog version in RHEL 6
> 
> On Sun, 21 Jun 2009, Rainer Gerhards wrote:
> 
> > That a good question and I, too, would like to know the 
> answer. In any case,
> > I hope it will not be v2. Giving that RHEL is a 
> conservative distribution
> > (and it has to be for the target base), I would assume that 
> if they go for a
> > newer version, it is v3.
> 
> not knowing when the freeze date is it's hard to say.
> 
> remember that the huge performance benifits of v4 (and v5 
> when it comes 
> out) would also be very useful for the target base.
> 
> not to mention all the subtle fixes you have done whiel pursuing 
> performance.

Yeah, that's right. But anyway ... I am not even asked, so I am a bystander
as most of the rest of this list ;)

> 
> David Lang
> 
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
> >> Sent: Saturday, June 20, 2009 9:48 PM
> >> To: rsyslog at lists.adiscon.com
> >> Subject: [rsyslog] RSyslog version in RHEL 6
> >>
> >> Hello,
> >> is there any info about what rsyslog version will be 
> available in RHEL
> >> 6? I suppose it will be 3.x, but im not sure, maybe 4.x 
> version in the
> >> time "RHEL6 package freeze" will be in stable status?
> >>
> >> Thank you,
> >> Jiri H.
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 


From david at lang.hm  Sun Jun 21 23:29:11 2009
From: david at lang.hm (david at lang.hm)
Date: Sun, 21 Jun 2009 14:29:11 -0700 (PDT)
Subject: [rsyslog] RSyslog version in RHEL 6
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9E6@GRFEXC.intern.adiscon.com>
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906210309480.23743@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E6@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906211427280.23743@asgard.lang.hm>

On Sun, 21 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Sun, 21 Jun 2009, Rainer Gerhards wrote:
>>
>>> That a good question and I, too, would like to know the
>> answer. In any case,
>>> I hope it will not be v2. Giving that RHEL is a
>> conservative distribution
>>> (and it has to be for the target base), I would assume that
>> if they go for a
>>> newer version, it is v3.
>>
>> not knowing when the freeze date is it's hard to say.
>>
>> remember that the huge performance benifits of v4 (and v5
>> when it comes
>> out) would also be very useful for the target base.
>>
>> not to mention all the subtle fixes you have done whiel pursuing
>> performance.
>
> Yeah, that's right. But anyway ... I am not even asked, so I am a bystander
> as most of the rest of this list ;)

Iagree that you are not asked, but since you hae a lot of influance on the 
schedule of when 4.x moves to stable you are a little more than a 
bystander ;-)

does anyone know the RHEL schedule? I would hate to see them lock in 3.x a 
few days before 4.x becomes stable

David Lang

>>
>> David Lang
>>
>>> Rainer
>>>
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com
>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
>>>> Sent: Saturday, June 20, 2009 9:48 PM
>>>> To: rsyslog at lists.adiscon.com
>>>> Subject: [rsyslog] RSyslog version in RHEL 6
>>>>
>>>> Hello,
>>>> is there any info about what rsyslog version will be
>> available in RHEL
>>>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x
>> version in the
>>>> time "RHEL6 package freeze" will be in stable status?
>>>>
>>>> Thank you,
>>>> Jiri H.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rio at rio.st  Mon Jun 22 03:44:56 2009
From: rio at rio.st (=?ISO-2022-JP?B?GyRCTkcbKEIgGyRCRiNFRBsoQg==?=)
Date: Mon, 22 Jun 2009 10:44:56 +0900
Subject: [rsyslog] RSyslog version in RHEL 6
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
Message-ID: <44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st>

Hi all,

I guess RHEL6 will be based on Fedora 11 & 12.
Fedora 11 was shipped with rsyslog-3.21.11-1.fc11.
It's sure that RHEL6 includes v3 at least.

Feature Freeze of Fedora 12 is planned on 2009-07-28.
http://fedoraproject.org/wiki/Schedule
If v4 is stable and requested to be included in F12, v4 will be  
shipped in F12.

Best Rio.

On 2009/06/21, at 18:58, Rainer Gerhards wrote:

> That a good question and I, too, would like to know the answer. In  
> any case,
> I hope it will not be v2. Giving that RHEL is a conservative  
> distribution
> (and it has to be for the target base), I would assume that if they  
> go for a
> newer version, it is v3.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
>> Sent: Saturday, June 20, 2009 9:48 PM
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] RSyslog version in RHEL 6
>>
>> Hello,
>> is there any info about what rsyslog version will be available in  
>> RHEL
>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in  
>> the
>> time "RHEL6 package freeze" will be in stable status?
>>
>> Thank you,
>> Jiri H.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com



From rgerhards at hq.adiscon.com  Mon Jun 22 11:16:02 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 22 Jun 2009 11:16:02 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906210309480.23743@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA706F9E6@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906211427280.23743@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9EB@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Sunday, June 21, 2009 11:29 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] RSyslog version in RHEL 6
> 
> On Sun, 21 Jun 2009, Rainer Gerhards wrote:

> >
> > Yeah, that's right. But anyway ... I am not even asked, so I am a
bystander
> > as most of the rest of this list ;)
> 
> Iagree that you are not asked, but since you hae a lot of influance on the
> schedule of when 4.x moves to stable you are a little more than a
> bystander ;-)


Well, the good thing is that the current v4-beta was actually scheduled to be
declared stable "these days". Will see that I make this "tomorrow". However,
the version that has most of the optimizations needs to go through the beta
stage (and needs so for good reasons), so this is another ~ three month from
now. But it probably would be sufficient to have v4 included, hopefully this
means newer versions of that same major version will also be included...

Rainer


From rgerhards at hq.adiscon.com  Mon Jun 22 13:10:32 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 22 Jun 2009 13:10:32 +0200
Subject: [rsyslog] memory alloc problems - any advise?
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906191324100.20154@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA706F9E3@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906191352480.20154@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9EE@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, June 19, 2009 11:37 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> 
> On Fri, 19 Jun 2009, Rainer Gerhards wrote:
> 
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >>
> >> On Fri, 19 Jun 2009, Rainer Gerhards wrote:
> >>
> > I checked with v3, and it had the same behavior (a little less strong,
but
> > still).
> 
> interesting.
> 
> when I was doing the tests that showed rsyslog able to recieve at gig-e
> line speed I would run tests repeatedly without any problems (with each
> test sending millions of logs, things sitting idle for hours, then another
> burst of millions of logs, repeat)
> 
> if I set the queue size too large for my ram the system would start paging
> and die, so I'm sure that it wasn't failing to release memory.
> 
> now, one thing could be that my tests were of messages that were all the
> same size, it could be that you are seeing some effect from fragmentation
> with different message size.

My test driver produces almost identical messages, with mostly the same size.
I have not yet implemented any randomization into it. So in theory, it should
be fairly easy for the malloc() subsystem to reuse freed memory.

Anyhow, based on the malloc stats I got, this really looks like an issue with
the allocator itself. I'll for now implement an alloc_trim() every 100,000
messages or so, that should keep care of this anomaly. Interestingly, v4 now
does fewer allocs than v3. It almost looks like the alloc subsystem also has
a periodic "garbage collection" counter, which now seems to be triggered less
often.

For v5, I have different memory alloc on my mind. My long-term goal is to
have almost no alloc/free pre message (in the magnitude of 1/1024 per
message). But this is rather intrusive and will not have any immediate hard
benefit. Thus I am not targeting this is something that needs to be done now.
For v4, however, I intend to do some more optimizations, some of which
further reduce the number of alloc/free calls needed per message.

I would definitely be a perfect match if you could test out these things on a
high-performance machine.

Rainer



From pvrabec at redhat.com  Mon Jun 22 15:53:10 2009
From: pvrabec at redhat.com (Peter Vrabec)
Date: Mon, 22 Jun 2009 15:53:10 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
In-Reply-To: <44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st>
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
	<44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st>
Message-ID: <200906221553.10791.pvrabec@redhat.com>

Hi folks,

rsyslog maintainer is on vacation at the moment so I'm answering this Q. :)

At first we need v4 stable. If we have stable v4 we will push it into rawhide 
and F11. This must be done ASAP, I see deadline in mid-July.

Peter.


On Monday 22 June 2009 03:44:56 am ? ?? wrote:
> Hi all,
>
> I guess RHEL6 will be based on Fedora 11 & 12.
> Fedora 11 was shipped with rsyslog-3.21.11-1.fc11.
> It's sure that RHEL6 includes v3 at least.
>
> Feature Freeze of Fedora 12 is planned on 2009-07-28.
> http://fedoraproject.org/wiki/Schedule
> If v4 is stable and requested to be included in F12, v4 will be
> shipped in F12.
>
> Best Rio.
>
> On 2009/06/21, at 18:58, Rainer Gerhards wrote:
> > That a good question and I, too, would like to know the answer. In
> > any case,
> > I hope it will not be v2. Giving that RHEL is a conservative
> > distribution
> > (and it has to be for the target base), I would assume that if they
> > go for a
> > newer version, it is v3.
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
> >> Sent: Saturday, June 20, 2009 9:48 PM
> >> To: rsyslog at lists.adiscon.com
> >> Subject: [rsyslog] RSyslog version in RHEL 6
> >>
> >> Hello,
> >> is there any info about what rsyslog version will be available in
> >> RHEL
> >> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in
> >> the
> >> time "RHEL6 package freeze" will be in stable status?
> >>
> >> Thank you,
> >> Jiri H.
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com



From rgerhards at hq.adiscon.com  Mon Jun 22 14:24:13 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 22 Jun 2009 14:24:13 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com><44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st>
	<200906221553.10791.pvrabec@redhat.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com>

Hi Peter,

that sounds quite interesting. I'll see that I get the v4-stable out
tomorrow.

Thanks for following up...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Peter Vrabec
> Sent: Monday, June 22, 2009 3:53 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] RSyslog version in RHEL 6
> 
> Hi folks,
> 
> rsyslog maintainer is on vacation at the moment so I'm answering this Q. :)
> 
> At first we need v4 stable. If we have stable v4 we will push it into
rawhide
> and F11. This must be done ASAP, I see deadline in mid-July.
> 
> Peter.
> 
> 
> On Monday 22 June 2009 03:44:56 am ? ?? wrote:
> > Hi all,
> >
> > I guess RHEL6 will be based on Fedora 11 & 12.
> > Fedora 11 was shipped with rsyslog-3.21.11-1.fc11.
> > It's sure that RHEL6 includes v3 at least.
> >
> > Feature Freeze of Fedora 12 is planned on 2009-07-28.
> > http://fedoraproject.org/wiki/Schedule
> > If v4 is stable and requested to be included in F12, v4 will be
> > shipped in F12.
> >
> > Best Rio.
> >
> > On 2009/06/21, at 18:58, Rainer Gerhards wrote:
> > > That a good question and I, too, would like to know the answer. In
> > > any case,
> > > I hope it will not be v2. Giving that RHEL is a conservative
> > > distribution
> > > (and it has to be for the target base), I would assume that if they
> > > go for a
> > > newer version, it is v3.
> > >
> > > Rainer
> > >
> > >> -----Original Message-----
> > >> From: rsyslog-bounces at lists.adiscon.com
> > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
> > >> Sent: Saturday, June 20, 2009 9:48 PM
> > >> To: rsyslog at lists.adiscon.com
> > >> Subject: [rsyslog] RSyslog version in RHEL 6
> > >>
> > >> Hello,
> > >> is there any info about what rsyslog version will be available in
> > >> RHEL
> > >> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in
> > >> the
> > >> time "RHEL6 package freeze" will be in stable status?
> > >>
> > >> Thank you,
> > >> Jiri H.
> > >> _______________________________________________
> > >> rsyslog mailing list
> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> http://www.rsyslog.com
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Mon Jun 22 19:04:35 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 22 Jun 2009 19:04:35 +0200
Subject: [rsyslog] memory alloc problems - any advise?
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906191324100.20154@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA706F9E3@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906191352480.20154@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706F9EE@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9F6@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Monday, June 22, 2009 1:11 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> My test driver produces almost identical messages, with mostly the same
size.
> I have not yet implemented any randomization into it. So in theory, it
should
> be fairly easy for the malloc() subsystem to reuse freed memory.
> 
> Anyhow, based on the malloc stats I got, this really looks like an issue
with
> the allocator itself. I'll for now implement an alloc_trim() every 100,000
> messages or so, that should keep care of this anomaly. Interestingly, v4
now
> does fewer allocs than v3. It almost looks like the alloc subsystem also
has
> a periodic "garbage collection" counter, which now seems to be triggered
less
> often.

Just FYI, I have implemented this algorithm now and it works pretty well.
Also, if there is an impact on performance, it is barely noticeable. 

I have also fixed a memory leak in v5 today, and run the same tests with v5
that I ran against v4. It is really impressive how much better it performs
(factor two, at least). On my low-end lab virtual machine, I am currently
processing around 150,00 tcp msgs/s (around 70,000 with v4) and this is also
close to sustained rate (that probably is around 10% below for v5).

Just thought I let you know.

Rainer


From tbergfeld at hq.adiscon.com  Tue Jun 23 11:25:52 2009
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Tue, 23 Jun 2009 11:25:52 +0200
Subject: [rsyslog] rsyslog 4.2.0 (v4-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA01@GRFEXC.intern.adiscon.com>

Hi all,
We have released the first version of rsyslog v4-stable today. It is the next
major step after v3 most importantly offering large performance improvements.
This version went through a long beta-phase and we did not have any specifc
bug reports within the last couple of weeks, so we honestly think this
release is ready for production use.

Download:

http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-161.phtml

Changelog:

http://www.rsyslog.com/Article376.phtml

As always, feedback is appreciated.

Best regards,
Tom Bergfeld

--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From sachulinux at gmail.com  Tue Jun 23 13:59:20 2009
From: sachulinux at gmail.com (Sachin Sawant)
Date: Tue, 23 Jun 2009 17:29:20 +0530
Subject: [rsyslog] template for file input module in Rsyslog
Message-ID: <edc9be030906230459q4090383sc8427501d8a413a7@mail.gmail.com>

Hi All,



I would like to monitor PostgreSQL log files using file input module in
Rsyslog. PostgreSQL create log files based on date, so is it possible to
monitor file without using absolute path?



Regards,

* *

*Sachin Sawant

*


From rgerhards at hq.adiscon.com  Tue Jun 23 14:47:34 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 23 Jun 2009 14:47:34 +0200
Subject: [rsyslog] template for file input module in Rsyslog
References: <edc9be030906230459q4090383sc8427501d8a413a7@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA0D@GRFEXC.intern.adiscon.com>

I think this is not currently supported..

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Sachin Sawant
> Sent: Tuesday, June 23, 2009 1:59 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] template for file input module in Rsyslog
> 
> Hi All,
> 
> 
> 
> I would like to monitor PostgreSQL log files using file input module in
> Rsyslog. PostgreSQL create log files based on date, so is it possible to
> monitor file without using absolute path?
> 
> 
> 
> Regards,
> 
> * *
> 
> *Sachin Sawant
> 
> *
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From martinmie at PartyGaming.com  Tue Jun 23 17:07:35 2009
From: martinmie at PartyGaming.com (Martin Mielke)
Date: Tue, 23 Jun 2009 17:07:35 +0200
Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9A4@GRFEXC.intern.adiscon.com>
References: <0B1B9163138571439EAF804646F3F6AA0767973E@GIBSVWIN004X.partygaming.local>
	<9B6E2A8877C38245BFB15CC491A11DA706F9A4@GRFEXC.intern.adiscon.com>
Message-ID: <0B1B9163138571439EAF804646F3F6AA07878F4E@GIBSVWIN004X.partygaming.local>

Sorry for the noise.

It seems that the post is pending on some reply ;-)


Cheers,


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: 18 June 2009 15:11
> To: rsyslog-users
> Subject: Re: [rsyslog] syslog-ng filters and destinations under
rsyslog
> 
> FYI, there was also a forum post:
> 
>
http://kb.monitorware.com/syslog-filters-and-destinations-under-rsyslog-
> t9858
> .html
> 
> I'd appreciate if any other responses also go to the forum (so that
> others,
> when searching, find the complete info).
> 
> Rainer
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Martin Mielke
> > Sent: Thursday, June 18, 2009 2:18 PM
> > To: rsyslog-users
> > Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
> >
> > Hi all,
> >
> > I need to find a way to either convert or import syslog-ng.conf
files
> > into rsyslog or learn the syntax to classify/store the logfiles
> > depending on the source.
> >
> > For example, taken from a syslog-ng file:
> > --
> > source s_network { udp(); tcp(); };
> >
> > filter f_systems {
> > host("1\.2\.3\.\d+");
> > };
> >
> > filter f_switches {
> > host("5\.6\.7\.8|5\.6\.7\.9");
> > };
> >
> > destination d_systems {file("/systems.log"); };
> > destination d_switches {file("/switches.log"); };
> >
> > log { source(s_network); filter(f_systems); destination(d_systems);
};
> > log { source(s_network); filter(f_switches);
destination(d_switches); };
> > --
> >
> > I have read the online documentation but I'm unable to find the
right
> > answer...
> >
> > Thanks!
> >
> > Martin
> >
> >
> > This email and any attachments are confidential, and may be legally
> privileged
> > and protected by copyright. If you are not the intended recipient
> > dissemination or copying of this email is prohibited. If you have
> received
> > this in error, please notify the sender by replying by email and
then
> delete
> > the email completely from your system.
> >
> > Any views or opinions are solely those of the sender.  This
> communication
> is
> > not intended to form a binding contract unless expressly indicated
to
> the
> > contrary and properly authorised. Any actions taken on the basis of
this
> email
> > are at the recipient's own risk.
> >
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> yslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Tue Jun 23 17:23:05 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 23 Jun 2009 17:23:05 +0200
Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
References: <0B1B9163138571439EAF804646F3F6AA0767973E@GIBSVWIN004X.partygaming.local><9B6E2A8877C38245BFB15CC491A11DA706F9A4@GRFEXC.intern.adiscon.com>
	<0B1B9163138571439EAF804646F3F6AA07878F4E@GIBSVWIN004X.partygaming.local>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA11@GRFEXC.intern.adiscon.com>

Thanks. Overlooked, you now have a reply ;)

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Martin Mielke
> Sent: Tuesday, June 23, 2009 5:08 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] syslog-ng filters and destinations under rsyslog
> 
> Sorry for the noise.
> 
> It seems that the post is pending on some reply ;-)
> 
> 
> Cheers,
> 
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> > Sent: 18 June 2009 15:11
> > To: rsyslog-users
> > Subject: Re: [rsyslog] syslog-ng filters and destinations under
> rsyslog
> >
> > FYI, there was also a forum post:
> >
> >
> http://kb.monitorware.com/syslog-filters-and-destinations-under-rsyslog-
> > t9858
> > .html
> >
> > I'd appreciate if any other responses also go to the forum (so that
> > others,
> > when searching, find the complete info).
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > bounces at lists.adiscon.com] On Behalf Of Martin Mielke
> > > Sent: Thursday, June 18, 2009 2:18 PM
> > > To: rsyslog-users
> > > Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
> > >
> > > Hi all,
> > >
> > > I need to find a way to either convert or import syslog-ng.conf
> files
> > > into rsyslog or learn the syntax to classify/store the logfiles
> > > depending on the source.
> > >
> > > For example, taken from a syslog-ng file:
> > > --
> > > source s_network { udp(); tcp(); };
> > >
> > > filter f_systems {
> > > host("1\.2\.3\.\d+");
> > > };
> > >
> > > filter f_switches {
> > > host("5\.6\.7\.8|5\.6\.7\.9");
> > > };
> > >
> > > destination d_systems {file("/systems.log"); };
> > > destination d_switches {file("/switches.log"); };
> > >
> > > log { source(s_network); filter(f_systems); destination(d_systems);
> };
> > > log { source(s_network); filter(f_switches);
> destination(d_switches); };
> > > --
> > >
> > > I have read the online documentation but I'm unable to find the
> right
> > > answer...
> > >
> > > Thanks!
> > >
> > > Martin
> > >
> > >
> > > This email and any attachments are confidential, and may be legally
> > privileged
> > > and protected by copyright. If you are not the intended recipient
> > > dissemination or copying of this email is prohibited. If you have
> > received
> > > this in error, please notify the sender by replying by email and
> then
> > delete
> > > the email completely from your system.
> > >
> > > Any views or opinions are solely those of the sender.  This
> > communication
> > is
> > > not intended to form a binding contract unless expressly indicated
> to
> > the
> > > contrary and properly authorised. Any actions taken on the basis of
> this
> > email
> > > are at the recipient's own risk.
> > >
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> > yslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Tue Jun 23 17:57:11 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 23 Jun 2009 17:57:11 +0200
Subject: [rsyslog] template for file input module in Rsyslog
References: <edc9be030906230459q4090383sc8427501d8a413a7@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706FA0D@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA17@GRFEXC.intern.adiscon.com>

I just checked, it is NOT supported.

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Tuesday, June 23, 2009 2:48 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] template for file input module in Rsyslog
> 
> I think this is not currently supported..
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Sachin Sawant
> > Sent: Tuesday, June 23, 2009 1:59 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: [rsyslog] template for file input module in Rsyslog
> >
> > Hi All,
> >
> >
> >
> > I would like to monitor PostgreSQL log files using file input module in
> > Rsyslog. PostgreSQL create log files based on date, so is it possible to
> > monitor file without using absolute path?
> >
> >
> >
> > Regards,
> >
> > * *
> >
> > *Sachin Sawant
> >
> > *
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From tbergfeld at hq.adiscon.com  Wed Jun 24 12:15:05 2009
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Wed, 24 Jun 2009 12:15:05 +0200
Subject: [rsyslog] rsyslog 4.3.2 (beta) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA31@GRFEXC.intern.adiscon.com>

Hi all,
The new rsyslog beta 4.3.2 has been released. This release consists of
several enhancements like the added generic network stream server for syslog,
capability to rebind the send socket of the UDP output section, a improved
testbench with a config switch and some bugfixes. This release is recommended
for all users of the beta branch.

Download:

http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-162.phtml

Changelog:

http://www.rsyslog.com/Article378.phtml

As always, feedback is appreciated.

Best regards,
Tom Bergfeld

Send this story to someone

--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From rgerhards at hq.adiscon.com  Wed Jun 24 14:31:04 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 24 Jun 2009 14:31:04 +0200
Subject: [rsyslog] segfault in v5-devel git branch
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>

Hi all,

I just wanted to let you know that the v5-devel git branch currently does not
work correctly when running with multiple worker threads. It will segfault
rather quickly. I am currently working on that and hope to resolve it soon.
Still, I thought it is better to tell you before someone begins to test with
that version.

Rainer


From rgerhards at hq.adiscon.com  Thu Jun 25 16:30:27 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 25 Jun 2009 16:30:27 +0200
Subject: [rsyslog] segfault in v5-devel git branch
References: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA5A@GRFEXC.intern.adiscon.com>

Hi all,

I just wanted to let you know that I hopefully have fixed this issue - at
least in my lab as well as under the threading debugging no problem shows up.
But of course, things like these require confirmation from the field.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Wednesday, June 24, 2009 2:31 PM
> To: rsyslog-users
> Subject: [rsyslog] segfault in v5-devel git branch
> 
> Hi all,
> 
> I just wanted to let you know that the v5-devel git branch currently does
not
> work correctly when running with multiple worker threads. It will segfault
> rather quickly. I am currently working on that and hope to resolve it soon.
> Still, I thought it is better to tell you before someone begins to test
with
> that version.
> 
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From kenneho.ndu at gmail.com  Fri Jun 26 14:30:09 2009
From: kenneho.ndu at gmail.com (Kenneth Holter)
Date: Fri, 26 Jun 2009 14:30:09 +0200
Subject: [rsyslog] rsyslogd:too many tcp sessions - dropping incoming request
Message-ID: <c25f25140906260530q7d102391k1154b1e5dca7a385@mail.gmail.com>

Hello list.


I'm running rsyslog 2.0.6, and have suddenly started to get this error
message:


rsyslogd:too many tcp sessions - dropping incoming request

I edited the /etc/sysconfig/rsyslog file and changed "-t200" to "-t200,500".
Is there any (performance) reasons or something that says that max number of
sessions should be 200 (default). If not, I'll stick with 500..


Regards,
Kenneth Holter


From rgerhards at hq.adiscon.com  Fri Jun 26 14:39:45 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 26 Jun 2009 14:39:45 +0200
Subject: [rsyslog] rsyslogd:too many tcp sessions - dropping incoming
	request
References: <c25f25140906260530q7d102391k1154b1e5dca7a385@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA68@GRFEXC.intern.adiscon.com>

No, I don't think so (in recent builds it is not). If I remember correctly,
the ",500" sets the accept queue but not the max # of sessions.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kenneth Holter
> Sent: Friday, June 26, 2009 2:30 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] rsyslogd:too many tcp sessions - dropping incoming
request
> 
> Hello list.
> 
> 
> I'm running rsyslog 2.0.6, and have suddenly started to get this error
> message:
> 
> 
> rsyslogd:too many tcp sessions - dropping incoming request
> 
> I edited the /etc/sysconfig/rsyslog file and changed "-t200" to
"-t200,500".
> Is there any (performance) reasons or something that says that max number
of
> sessions should be 200 (default). If not, I'll stick with 500..
> 
> 
> Regards,
> Kenneth Holter
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From mbiebl at gmail.com  Fri Jun 26 14:44:29 2009
From: mbiebl at gmail.com (Michael Biebl)
Date: Fri, 26 Jun 2009 14:44:29 +0200
Subject: [rsyslog] segfault in v5-devel git branch
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>
Message-ID: <aa76cdac0906260544l24ddecexb7c3b35436b05875@mail.gmail.com>

2009/6/24 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Hi all,
>
> I just wanted to let you know that the v5-devel git branch currently does not
> work correctly when running with multiple worker threads. It will segfault
> rather quickly. I am currently working on that and hope to resolve it soon.
> Still, I thought it is better to tell you before someone begins to test with
> that version.

I'm not even able to compile the v5-devel branch :-/

  CC    stringbuf.o
In file included from zlibw.h:28,
                 from stream.h:72,
                 from obj.h:50,
                 from rsyslog.h:441,
                 from stringbuf.c:39:
/usr/include/zlib.h:1366: error: expected ?=?, ?,?, ?;?, ?asm? or
?__attribute__? before ?gzseek64?
/usr/include/zlib.h:1367: error: expected ?=?, ?,?, ?;?, ?asm? or
?__attribute__? before ?gztell64?
/usr/include/zlib.h:1368: error: expected declaration specifiers or
?...? before ?off64_t?
/usr/include/zlib.h:1369: error: expected declaration specifiers or
?...? before ?off64_t?
make[2]: *** [librsyslog_la-stringbuf.lo] Fehler 1
make[1]: *** [all-recursive] Fehler 1
make: *** [all] Fehler 2
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Fri Jun 26 15:19:28 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 26 Jun 2009 15:19:28 +0200
Subject: [rsyslog] segfault in v5-devel git branch
References: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>
	<aa76cdac0906260544l24ddecexb7c3b35436b05875@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA6A@GRFEXC.intern.adiscon.com>

Can you please let me know what the offending line in your zlib.h include
file is? I guess there is a name conflict in one of the types used.

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Friday, June 26, 2009 2:44 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] segfault in v5-devel git branch
> 
> 2009/6/24 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Hi all,
> >
> > I just wanted to let you know that the v5-devel git branch currently does
> not
> > work correctly when running with multiple worker threads. It will
segfault
> > rather quickly. I am currently working on that and hope to resolve it
soon.
> > Still, I thought it is better to tell you before someone begins to test
with
> > that version.
> 
> I'm not even able to compile the v5-devel branch :-/
> 
>   CC    stringbuf.o
> In file included from zlibw.h:28,
>                  from stream.h:72,
>                  from obj.h:50,
>                  from rsyslog.h:441,
>                  from stringbuf.c:39:
> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or
> '__attribute__' before 'gzseek64'
> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or
> '__attribute__' before 'gztell64'
> /usr/include/zlib.h:1368: error: expected declaration specifiers or
> '...' before 'off64_t'
> /usr/include/zlib.h:1369: error: expected declaration specifiers or
> '...' before 'off64_t'
> make[2]: *** [librsyslog_la-stringbuf.lo] Fehler 1
> make[1]: *** [all-recursive] Fehler 1
> make: *** [all] Fehler 2
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From mbiebl at gmail.com  Fri Jun 26 15:36:18 2009
From: mbiebl at gmail.com (Michael Biebl)
Date: Fri, 26 Jun 2009 15:36:18 +0200
Subject: [rsyslog] segfault in v5-devel git branch
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA6A@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>
	<aa76cdac0906260544l24ddecexb7c3b35436b05875@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706FA6A@GRFEXC.intern.adiscon.com>
Message-ID: <aa76cdac0906260636ia9a3cc2gba9a4bfc6fecdec4@mail.gmail.com>

#ifdef _LARGEFILE64_SOURCE
   ZEXTERN gzFile ZEXPORT gzopen64 OF((const char *, const char *));
   ZEXTERN off64_t ZEXPORT gzseek64 OF((gzFile, off64_t, int));
   ZEXTERN off64_t ZEXPORT gztell64 OF((gzFile));
   ZEXTERN uLong ZEXPORT adler32_combine64 OF((uLong, uLong, off64_t));
   ZEXTERN uLong ZEXPORT crc32_combine64 OF((uLong, uLong, off64_t));
#endif


2009/6/26 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Can you please let me know what the offending line in your zlib.h include
> file is? I guess there is a name conflict in one of the types used.
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
>> Sent: Friday, June 26, 2009 2:44 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] segfault in v5-devel git branch
>>
>> 2009/6/24 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> > Hi all,
>> >
>> > I just wanted to let you know that the v5-devel git branch currently does
>> not
>> > work correctly when running with multiple worker threads. It will
> segfault
>> > rather quickly. I am currently working on that and hope to resolve it
> soon.
>> > Still, I thought it is better to tell you before someone begins to test
> with
>> > that version.
>>
>> I'm not even able to compile the v5-devel branch :-/
>>
>> ? CC ? ?stringbuf.o
>> In file included from zlibw.h:28,
>> ? ? ? ? ? ? ? ? ?from stream.h:72,
>> ? ? ? ? ? ? ? ? ?from obj.h:50,
>> ? ? ? ? ? ? ? ? ?from rsyslog.h:441,
>> ? ? ? ? ? ? ? ? ?from stringbuf.c:39:
>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or
>> '__attribute__' before 'gzseek64'
>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or
>> '__attribute__' before 'gztell64'
>> /usr/include/zlib.h:1368: error: expected declaration specifiers or
>> '...' before 'off64_t'
>> /usr/include/zlib.h:1369: error: expected declaration specifiers or
>> '...' before 'off64_t'
>> make[2]: *** [librsyslog_la-stringbuf.lo] Fehler 1
>> make[1]: *** [all-recursive] Fehler 1
>> make: *** [all] Fehler 2
>> --
>> Why is it that all of the instruments seeking intelligent life in the
>> universe are pointed away from Earth?
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>



-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From iamsayan at gmail.com  Mon Jun 29 18:25:18 2009
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Mon, 29 Jun 2009 12:25:18 -0400
Subject: [rsyslog] How to test rsyslog performance
Message-ID: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com>

Hi,
I saw the announcement of the 4.2.0 stable version of rsyslog. I was
wondering if there is a way I can test the performance of rsyslog with my
own rsyslog.conf.
The way I am trying to use rsyslog is on a local system, which has following
rules

logging different facilities to different files with different formats.
forwarding certain facilities to an external server.
receive messages from one(and only one) other machine over udp.

Are there any tool/scripts available which I can use to perform the tests
myself?
Basically what I am thinking of is running a script which writes is bunch of
messages to syslog, but then how do I make sure all those messages were
actually logged and not lost? It may be very painful to look through the
entire log file 1 message at a time :)
Is there a stat/counter based api which shows me how many messages were
logged?

Regards,
Sayan


From david at lang.hm  Mon Jun 29 18:46:53 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 29 Jun 2009 09:46:53 -0700 (PDT)
Subject: [rsyslog] How to test rsyslog performance
In-Reply-To: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com>
References: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906290943480.9159@asgard>

On Mon, 29 Jun 2009, Sayan Chowdhury wrote:

> Hi,
> I saw the announcement of the 4.2.0 stable version of rsyslog. I was
> wondering if there is a way I can test the performance of rsyslog with my
> own rsyslog.conf.
> The way I am trying to use rsyslog is on a local system, which has following
> rules
>
> logging different facilities to different files with different formats.
> forwarding certain facilities to an external server.
> receive messages from one(and only one) other machine over udp.
>
> Are there any tool/scripts available which I can use to perform the tests
> myself?
> Basically what I am thinking of is running a script which writes is bunch of
> messages to syslog, but then how do I make sure all those messages were
> actually logged and not lost? It may be very painful to look through the
> entire log file 1 message at a time :)
> Is there a stat/counter based api which shows me how many messages were
> logged?

what I do is to send a known quantity of messages and include a identifier 
string in the message ('rsyslog-test-message' for example), and then grep 
the resulting files to make sure that the right number of messages arrive.

to send messages to rsyslog I have used tcpreplay to replay UDP messages. 
I first setup a system to send the messages via UDP, start tcpdump, send 
the messages to the local box (which then sends them via UDP to a remote 
server), then I stop tcpdump and can use the resulting capture file with 
tcpreplay to replay the messages at different speeds.

David Lang


From iamsayan at gmail.com  Mon Jun 29 19:55:15 2009
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Mon, 29 Jun 2009 13:55:15 -0400
Subject: [rsyslog] How to test rsyslog performance
In-Reply-To: <alpine.DEB.1.10.0906290943480.9159@asgard>
References: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com>
	<alpine.DEB.1.10.0906290943480.9159@asgard>
Message-ID: <87a1ae540906291055n7625b369naf02a1576e8ee187@mail.gmail.com>

Hello David,
Thanks a bunch ..
I will try this, I was also thinking in similar lines, in the sense that I
will log 1 line of message, set the file size to be huge, and then count the
number of lines on the file.
I just read the numbers on Rainer's blog, and I think I will need the
tcprelay approach as well, there is no way I can generate so much load with
a shell script.
Thanks for the tip.
Regadrs,
Sayan


On Mon, Jun 29, 2009 at 12:46 PM, <david at lang.hm> wrote:

> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
>
> > Hi,
> > I saw the announcement of the 4.2.0 stable version of rsyslog. I was
> > wondering if there is a way I can test the performance of rsyslog with my
> > own rsyslog.conf.
> > The way I am trying to use rsyslog is on a local system, which has
> following
> > rules
> >
> > logging different facilities to different files with different formats.
> > forwarding certain facilities to an external server.
> > receive messages from one(and only one) other machine over udp.
> >
> > Are there any tool/scripts available which I can use to perform the tests
> > myself?
> > Basically what I am thinking of is running a script which writes is bunch
> of
> > messages to syslog, but then how do I make sure all those messages were
> > actually logged and not lost? It may be very painful to look through the
> > entire log file 1 message at a time :)
> > Is there a stat/counter based api which shows me how many messages were
> > logged?
>
> what I do is to send a known quantity of messages and include a identifier
> string in the message ('rsyslog-test-message' for example), and then grep
> the resulting files to make sure that the right number of messages arrive.
>
> to send messages to rsyslog I have used tcpreplay to replay UDP messages.
> I first setup a system to send the messages via UDP, start tcpdump, send
> the messages to the local box (which then sends them via UDP to a remote
> server), then I stop tcpdump and can use the resulting capture file with
> tcpreplay to replay the messages at different speeds.
>
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From david at lang.hm  Mon Jun 29 20:03:33 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 29 Jun 2009 11:03:33 -0700 (PDT)
Subject: [rsyslog] How to test rsyslog performance
In-Reply-To: <87a1ae540906291055n7625b369naf02a1576e8ee187@mail.gmail.com>
References: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com>
	<alpine.DEB.1.10.0906290943480.9159@asgard>
	<87a1ae540906291055n7625b369naf02a1576e8ee187@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906291100010.9159@asgard>

On Mon, 29 Jun 2009, Sayan Chowdhury wrote:

> Hello David,
> Thanks a bunch ..
> I will try this, I was also thinking in similar lines, in the sense that I
> will log 1 line of message, set the file size to be huge, and then count the
> number of lines on the file.
> I just read the numbers on Rainer's blog, and I think I will need the
> tcprelay approach as well, there is no way I can generate so much load with
> a shell script.
> Thanks for the tip.

with an earlier version of the 4.x series I was seeing numbers like the 
following

with 256 byte messages

recieving via UDP and storing in the queue, >300K/sec (wire speed, gig-E)

recieving via UDP and writing to disk, ~78K/sec

recieving via UDP, writing to disk and sending to another machine, ~30k/sec

since then there has been a lot of work to optimize the queue functions 
that appeared to be the bottleneck in my testing, and so I am very 
interested in seeing the new results (I expect to be testing shortly, but 
will be out much of this week, one of my 'hobbies' is doing professional 
fireworks, and so I will be busy preparing for the July 4th shows)

David Lang


> Regadrs,
> Sayan
>
>
> On Mon, Jun 29, 2009 at 12:46 PM, <david at lang.hm> wrote:
>
>> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
>>
>>> Hi,
>>> I saw the announcement of the 4.2.0 stable version of rsyslog. I was
>>> wondering if there is a way I can test the performance of rsyslog with my
>>> own rsyslog.conf.
>>> The way I am trying to use rsyslog is on a local system, which has
>> following
>>> rules
>>>
>>> logging different facilities to different files with different formats.
>>> forwarding certain facilities to an external server.
>>> receive messages from one(and only one) other machine over udp.
>>>
>>> Are there any tool/scripts available which I can use to perform the tests
>>> myself?
>>> Basically what I am thinking of is running a script which writes is bunch
>> of
>>> messages to syslog, but then how do I make sure all those messages were
>>> actually logged and not lost? It may be very painful to look through the
>>> entire log file 1 message at a time :)
>>> Is there a stat/counter based api which shows me how many messages were
>>> logged?
>>
>> what I do is to send a known quantity of messages and include a identifier
>> string in the message ('rsyslog-test-message' for example), and then grep
>> the resulting files to make sure that the right number of messages arrive.
>>
>> to send messages to rsyslog I have used tcpreplay to replay UDP messages.
>> I first setup a system to send the messages via UDP, start tcpdump, send
>> the messages to the local box (which then sends them via UDP to a remote
>> server), then I stop tcpdump and can use the resulting capture file with
>> tcpreplay to replay the messages at different speeds.
>>
>> David Lang
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Mon Jun 29 21:33:54 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 29 Jun 2009 21:33:54 +0200
Subject: [rsyslog] How to test rsyslog performance
References: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com><alpine.DEB.1.10.0906290943480.9159@asgard><87a1ae540906291055n7625b369naf02a1576e8ee187@mail.gmail.com>
	<alpine.DEB.1.10.0906291100010.9159@asgard>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA8A@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Monday, June 29, 2009 8:04 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] How to test rsyslog performance
> 
> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
> 
> > Hello David,
> > Thanks a bunch ..
> > I will try this, I was also thinking in similar lines, in 
> the sense that I
> > will log 1 line of message, set the file size to be huge, 
> and then count the
> > number of lines on the file.
> > I just read the numbers on Rainer's blog, and I think I 
> will need the
> > tcprelay approach as well, there is no way I can generate 
> so much load with
> > a shell script.
> > Thanks for the tip.
> 
> with an earlier version of the 4.x series I was seeing 
> numbers like the 
> following
> 
> with 256 byte messages
> 
> recieving via UDP and storing in the queue, >300K/sec (wire 
> speed, gig-E)
> 
> recieving via UDP and writing to disk, ~78K/sec
> 
> recieving via UDP, writing to disk and sending to another 
> machine, ~30k/sec
> 
> since then there has been a lot of work to optimize the queue 
> functions 
> that appeared to be the bottleneck in my testing,

In the mean time, I have identified at least one more bottleneck, and this
was alloc/free. These come with implicit synchroniation primitives (to
protect the heap) and it turned out that they cause quite some performance
loss as soon as we have enough threads running concurrently. I have removed
much of these alloc calls during the past two weeks, and the results - at
least in my non-high-end lab - are quite pleasant. This work is still not
100% finished, but I think I have reched the mark where further reduction of
alloc does not case any notable performance increase. I'll still continue to
finish up some things (so that the code base is consistent), but I don't
expect much performance increase from that. 

All in all, the "new v4" engine should perform around twice as fast as the
"old v4" engine (depending a bit on configuration). The v5 engine offers more
advanced optimizations and will receive even more of them. At this time, the
v5 engine in my lab (with TCP!) is at least twice as fast as the "new v4"
engine, probably a bit less with UDP.

There is still ample of tuning potential. Two things that immediately come to
my mind is that the inputs still use select() [originally used for
portability reasons] and the inputs themselves are single-threaded. For v5 I
can also envision a new malloc subsytem, that will do 1/1024th malloc per
message (right now, we do around 5 mallocs per message, tomorrow evening
probably only 3).

If someone is looking into performance, one should definitely look into the
v5 engine. This version is experimental, but if nobody tries it out, it can't
improve (everything I could do in my lab I have done, but I know from past
experience that the real world and high-end hardware is a different story).
But even the test with the threading debugger went quite well. I'll release a
new v5 shortly, and that one is definitely worth a try (if you need it
urgent, just let me know).

> and so I am very 
> interested in seeing the new results (I expect to be testing 
> shortly, but 
> will be out much of this week, one of my 'hobbies' is doing 
> professional 
> fireworks, and so I will be busy preparing for the July 4th shows)

Cool - wish I were there ;)

Rainer 

> David Lang
> 
> 
> > Regadrs,
> > Sayan
> >
> >
> > On Mon, Jun 29, 2009 at 12:46 PM, <david at lang.hm> wrote:
> >
> >> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
> >>
> >>> Hi,
> >>> I saw the announcement of the 4.2.0 stable version of 
> rsyslog. I was
> >>> wondering if there is a way I can test the performance of 
> rsyslog with my
> >>> own rsyslog.conf.
> >>> The way I am trying to use rsyslog is on a local system, which has
> >> following
> >>> rules
> >>>
> >>> logging different facilities to different files with 
> different formats.
> >>> forwarding certain facilities to an external server.
> >>> receive messages from one(and only one) other machine over udp.
> >>>
> >>> Are there any tool/scripts available which I can use to 
> perform the tests
> >>> myself?
> >>> Basically what I am thinking of is running a script which 
> writes is bunch
> >> of
> >>> messages to syslog, but then how do I make sure all those 
> messages were
> >>> actually logged and not lost? It may be very painful to 
> look through the
> >>> entire log file 1 message at a time :)
> >>> Is there a stat/counter based api which shows me how many 
> messages were
> >>> logged?
> >>
> >> what I do is to send a known quantity of messages and 
> include a identifier
> >> string in the message ('rsyslog-test-message' for 
> example), and then grep
> >> the resulting files to make sure that the right number of 
> messages arrive.
> >>
> >> to send messages to rsyslog I have used tcpreplay to 
> replay UDP messages.
> >> I first setup a system to send the messages via UDP, start 
> tcpdump, send
> >> the messages to the local box (which then sends them via 
> UDP to a remote
> >> server), then I stop tcpdump and can use the resulting 
> capture file with
> >> tcpreplay to replay the messages at different speeds.
> >>
> >> David Lang
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 


From david at lang.hm  Mon Jun 29 22:18:50 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 29 Jun 2009 13:18:50 -0700 (PDT)
Subject: [rsyslog] How to test rsyslog performance
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA8A@GRFEXC.intern.adiscon.com>
References: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com><alpine.DEB.1.10.0906290943480.9159@asgard><87a1ae540906291055n7625b369naf02a1576e8ee187@mail.gmail.com>
	<alpine.DEB.1.10.0906291100010.9159@asgard>
	<9B6E2A8877C38245BFB15CC491A11DA706FA8A@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906291317300.9159@asgard>

On Mon, 29 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
>>
>>> Hello David,
>>> Thanks a bunch ..
>>> I will try this, I was also thinking in similar lines, in
>> the sense that I
>>> will log 1 line of message, set the file size to be huge,
>> and then count the
>>> number of lines on the file.
>>> I just read the numbers on Rainer's blog, and I think I
>> will need the
>>> tcprelay approach as well, there is no way I can generate
>> so much load with
>>> a shell script.
>>> Thanks for the tip.
>>
>> with an earlier version of the 4.x series I was seeing
>> numbers like the
>> following
>>
>> with 256 byte messages
>>
>> recieving via UDP and storing in the queue, >300K/sec (wire
>> speed, gig-E)
>>
>> recieving via UDP and writing to disk, ~78K/sec
>>
>> recieving via UDP, writing to disk and sending to another
>> machine, ~30k/sec
>>
>> since then there has been a lot of work to optimize the queue
>> functions
>> that appeared to be the bottleneck in my testing,
>
> In the mean time, I have identified at least one more bottleneck, and this
> was alloc/free. These come with implicit synchroniation primitives (to
> protect the heap) and it turned out that they cause quite some performance
> loss as soon as we have enough threads running concurrently. I have removed
> much of these alloc calls during the past two weeks, and the results - at
> least in my non-high-end lab - are quite pleasant. This work is still not
> 100% finished, but I think I have reched the mark where further reduction of
> alloc does not case any notable performance increase. I'll still continue to
> finish up some things (so that the code base is consistent), but I don't
> expect much performance increase from that.
>
> All in all, the "new v4" engine should perform around twice as fast as the
> "old v4" engine (depending a bit on configuration). The v5 engine offers more
> advanced optimizations and will receive even more of them. At this time, the
> v5 engine in my lab (with TCP!) is at least twice as fast as the "new v4"
> engine, probably a bit less with UDP.
>
> There is still ample of tuning potential. Two things that immediately come to
> my mind is that the inputs still use select() [originally used for
> portability reasons] and the inputs themselves are single-threaded. For v5 I
> can also envision a new malloc subsytem, that will do 1/1024th malloc per
> message (right now, we do around 5 mallocs per message, tomorrow evening
> probably only 3).
>
> If someone is looking into performance, one should definitely look into the
> v5 engine. This version is experimental, but if nobody tries it out, it can't
> improve (everything I could do in my lab I have done, but I know from past
> experience that the real world and high-end hardware is a different story).
> But even the test with the threading debugger went quite well. I'll release a
> new v5 shortly, and that one is definitely worth a try (if you need it
> urgent, just let me know).

I tried to compile the v5 branch over the weekend, but got a compile 
error, so I just moved to the 4.2.0 release. on the ultra-reliable testbed 
it's getting ~8500/sec.

David Lang

>> and so I am very
>> interested in seeing the new results (I expect to be testing
>> shortly, but
>> will be out much of this week, one of my 'hobbies' is doing
>> professional
>> fireworks, and so I will be busy preparing for the July 4th shows)
>
> Cool - wish I were there ;)
>
> Rainer
>
>> David Lang
>>
>>
>>> Regadrs,
>>> Sayan
>>>
>>>
>>> On Mon, Jun 29, 2009 at 12:46 PM, <david at lang.hm> wrote:
>>>
>>>> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
>>>>
>>>>> Hi,
>>>>> I saw the announcement of the 4.2.0 stable version of
>> rsyslog. I was
>>>>> wondering if there is a way I can test the performance of
>> rsyslog with my
>>>>> own rsyslog.conf.
>>>>> The way I am trying to use rsyslog is on a local system, which has
>>>> following
>>>>> rules
>>>>>
>>>>> logging different facilities to different files with
>> different formats.
>>>>> forwarding certain facilities to an external server.
>>>>> receive messages from one(and only one) other machine over udp.
>>>>>
>>>>> Are there any tool/scripts available which I can use to
>> perform the tests
>>>>> myself?
>>>>> Basically what I am thinking of is running a script which
>> writes is bunch
>>>> of
>>>>> messages to syslog, but then how do I make sure all those
>> messages were
>>>>> actually logged and not lost? It may be very painful to
>> look through the
>>>>> entire log file 1 message at a time :)
>>>>> Is there a stat/counter based api which shows me how many
>> messages were
>>>>> logged?
>>>>
>>>> what I do is to send a known quantity of messages and
>> include a identifier
>>>> string in the message ('rsyslog-test-message' for
>> example), and then grep
>>>> the resulting files to make sure that the right number of
>> messages arrive.
>>>>
>>>> to send messages to rsyslog I have used tcpreplay to
>> replay UDP messages.
>>>> I first setup a system to send the messages via UDP, start
>> tcpdump, send
>>>> the messages to the local box (which then sends them via
>> UDP to a remote
>>>> server), then I stop tcpdump and can use the resulting
>> capture file with
>>>> tcpreplay to replay the messages at different speeds.
>>>>
>>>> David Lang
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From sfzhi at yahoo.com  Tue Jun 30 11:11:14 2009
From: sfzhi at yahoo.com (Sergei Zhirikov)
Date: Tue, 30 Jun 2009 11:11:14 +0200
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
Message-ID: <h2ckri$16j$1@ger.gmane.org>

Rainer Gerhards wrote:
> Hi all,
> 
> The most prominent use case is to size-limit files. $OutChannel provides the
> ability to execute a command when a certain file size is reached. The command
> may then rotate the file. After command execution, omfile checks the file
> size again. If it still is above the configured limit, this file will be
> disabled (no longer be written to until restart).
> 
> If you use this for size-limiting, please let me know how you do it. Please
> also let me know if you would like to see things handled differently.
> 
> Looking forward to your feedback.
> 
> Thanks,
> Rainer
> 

Hi Rainer,

I do use $OutChannel for size-limiting and it would be a disappointment 
to me if that feature disappeared. As a matter of fact, it was the main 
reason why I chose to use rsyslog in the first place.

I find the "classic" way of using cron+logrotate+SIGHUP quite ugly and 
in some cases unacceptable. I have a small home gateway with read-only 
main file system, so all the log files are written into a tmpfs mount. 
It happened a couple of times that some misconfigured or malfunctioning 
software produced a lot of logging and filled in most of the available 
RAM before the next scheduled run of logrotate. In this situation the 
size limiting feature of rsyslog is exactly what I need in order to be 
able to avoid such this kind of problems.

Kind regards,
Sergei.



From david at lang.hm  Tue Jun 30 12:35:58 2009
From: david at lang.hm (david at lang.hm)
Date: Tue, 30 Jun 2009 03:35:58 -0700 (PDT)
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <h2ckri$16j$1@ger.gmane.org>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
	<h2ckri$16j$1@ger.gmane.org>
Message-ID: <alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>

On Tue, 30 Jun 2009, Sergei Zhirikov wrote:

> Rainer Gerhards wrote:
>> Hi all,
>>
>> The most prominent use case is to size-limit files. $OutChannel provides the
>> ability to execute a command when a certain file size is reached. The command
>> may then rotate the file. After command execution, omfile checks the file
>> size again. If it still is above the configured limit, this file will be
>> disabled (no longer be written to until restart).
>>
>> If you use this for size-limiting, please let me know how you do it. Please
>> also let me know if you would like to see things handled differently.
>>
>> Looking forward to your feedback.
>>
>> Thanks,
>> Rainer
>>
>
> Hi Rainer,
>
> I do use $OutChannel for size-limiting and it would be a disappointment
> to me if that feature disappeared. As a matter of fact, it was the main
> reason why I chose to use rsyslog in the first place.
>
> I find the "classic" way of using cron+logrotate+SIGHUP quite ugly and
> in some cases unacceptable. I have a small home gateway with read-only
> main file system, so all the log files are written into a tmpfs mount.
> It happened a couple of times that some misconfigured or malfunctioning
> software produced a lot of logging and filled in most of the available
> RAM before the next scheduled run of logrotate. In this situation the
> size limiting feature of rsyslog is exactly what I need in order to be
> able to avoid such this kind of problems.

this is a very legitimate use. I suspect the biggest issue is how to give 
you this capability without slowing down the people who don't use it.

I wonder if rainerscript could be extended to where it could be used to do 
this?

something like:

for each message logged, add the length of the message to a counter, and 
have an if statement that calls the external program to roll the log and 
reset the counter.

David Lang


From rgerhards at hq.adiscon.com  Tue Jun 30 12:43:14 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 30 Jun 2009 12:43:14 +0200
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com><h2ckri$16j$1@ger.gmane.org>
	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, June 30, 2009 12:36 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] omfile redesign / $OutChannel will go away!
> 
> On Tue, 30 Jun 2009, Sergei Zhirikov wrote:
> 
> > Rainer Gerhards wrote:
> >> Hi all,
> >>
> >> The most prominent use case is to size-limit files. $OutChannel provides
> the
> >> ability to execute a command when a certain file size is reached. The
> command
> >> may then rotate the file. After command execution, omfile checks the
file
> >> size again. If it still is above the configured limit, this file will be
> >> disabled (no longer be written to until restart).
> >>
> >> If you use this for size-limiting, please let me know how you do it.
Please
> >> also let me know if you would like to see things handled differently.
> >>
> >> Looking forward to your feedback.
> >>
> >> Thanks,
> >> Rainer
> >>
> >
> > Hi Rainer,
> >
> > I do use $OutChannel for size-limiting and it would be a disappointment
> > to me if that feature disappeared. As a matter of fact, it was the main
> > reason why I chose to use rsyslog in the first place.
> >
> > I find the "classic" way of using cron+logrotate+SIGHUP quite ugly and
> > in some cases unacceptable. I have a small home gateway with read-only
> > main file system, so all the log files are written into a tmpfs mount.
> > It happened a couple of times that some misconfigured or malfunctioning
> > software produced a lot of logging and filled in most of the available
> > RAM before the next scheduled run of logrotate. In this situation the
> > size limiting feature of rsyslog is exactly what I need in order to be
> > able to avoid such this kind of problems.
> 
> this is a very legitimate use. I suspect the biggest issue is how to give
> you this capability without slowing down the people who don't use it.
> 
> I wonder if rainerscript could be extended to where it could be used to do
> this?
> 
> something like:
> 
> for each message logged, add the length of the message to a counter, and
> have an if statement that calls the external program to roll the log and
> reset the counter.

I probably should have phrased the initial question less agressive, but than
I'd probably not received any feedback. From the feedback I received so far,
there seem to be two use cases:

a) circular logs
b) stop logging when file hits max size (was that really what this post
described?)

In any case, these two seem to be the only things that are done with
outchannel. So when $outchannel goes away (and it will at some time), the
"regular" file writer will have options to support these use cases - probably
more elegant than before. For example, for case a), circular logs, I do not
need to involve an external process, so I can remove that feature without
harm, as long as I replace it with something suitable (and think that omfile
now uses the stream writer, which *already does* circular "logging" for disk
queues).

So functionality in actual need will not go away, as long as people tell what
they need.

If I look at the b) case, I suspect that this is not even the intent of the
original poster. I think he would be much better served with a circular log
that records as many log records as fit into a finite, operator-specified log
file size. Or am I wrong here? (If nobody tells me I am wrong, I'll probably
remove the b) use case in the long term!).

Rainer



From david at lang.hm  Tue Jun 30 13:03:54 2009
From: david at lang.hm (david at lang.hm)
Date: Tue, 30 Jun 2009 04:03:54 -0700 (PDT)
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com><h2ckri$16j$1@ger.gmane.org>
	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906300354370.11336@asgard.lang.hm>

On Tue, 30 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Tue, 30 Jun 2009, Sergei Zhirikov wrote:
>>
>>> Rainer Gerhards wrote:
>>>> Hi all,
>>>>
>>>> The most prominent use case is to size-limit files. $OutChannel provides
>> the
>>>> ability to execute a command when a certain file size is reached. The
>> command
>>>> may then rotate the file. After command execution, omfile checks the
> file
>>>> size again. If it still is above the configured limit, this file will be
>>>> disabled (no longer be written to until restart).
>>>>
>>>> If you use this for size-limiting, please let me know how you do it.
> Please
>>>> also let me know if you would like to see things handled differently.
>>>>
>>>> Looking forward to your feedback.
>>>>
>>>> Thanks,
>>>> Rainer
>>>>
>>>
>>> Hi Rainer,
>>>
>>> I do use $OutChannel for size-limiting and it would be a disappointment
>>> to me if that feature disappeared. As a matter of fact, it was the main
>>> reason why I chose to use rsyslog in the first place.
>>>
>>> I find the "classic" way of using cron+logrotate+SIGHUP quite ugly and
>>> in some cases unacceptable. I have a small home gateway with read-only
>>> main file system, so all the log files are written into a tmpfs mount.
>>> It happened a couple of times that some misconfigured or malfunctioning
>>> software produced a lot of logging and filled in most of the available
>>> RAM before the next scheduled run of logrotate. In this situation the
>>> size limiting feature of rsyslog is exactly what I need in order to be
>>> able to avoid such this kind of problems.
>>
>> this is a very legitimate use. I suspect the biggest issue is how to give
>> you this capability without slowing down the people who don't use it.
>>
>> I wonder if rainerscript could be extended to where it could be used to do
>> this?
>>
>> something like:
>>
>> for each message logged, add the length of the message to a counter, and
>> have an if statement that calls the external program to roll the log and
>> reset the counter.
>
> I probably should have phrased the initial question less agressive, but than
> I'd probably not received any feedback. From the feedback I received so far,
> there seem to be two use cases:
>
> a) circular logs
> b) stop logging when file hits max size (was that really what this post
> described?)

no, this is a third case
c) limit the size of the logfile that is being added to, and when it hits 
this limit roll it off to permanent storage (at which point the file 
becomes read-only)

this is a need at both the low-end and the high-end

for this poster, he needs it at the low end as his 'working' files are in 
ram (his /var/log/messages), but then he rolls it off to network storage.

at this high end you can have large volumes of logs arriving, and want to 
have ver fast (but expensive) storage to handle the high file modification 
rate, but want to make sure that you don't run out of space there by 
rolling the logs off to slower, but larger disks so that they don't 
fillthe fast drive and cause you to loose logs.

> In any case, these two seem to be the only things that are done with
> outchannel. So when $outchannel goes away (and it will at some time), the
> "regular" file writer will have options to support these use cases - probably
> more elegant than before. For example, for case a), circular logs, I do not
> need to involve an external process, so I can remove that feature without
> harm, as long as I replace it with something suitable (and think that omfile
> now uses the stream writer, which *already does* circular "logging" for disk
> queues).

that sounds reasonable

> So functionality in actual need will not go away, as long as people tell what
> they need.
>
> If I look at the b) case, I suspect that this is not even the intent of the
> original poster. I think he would be much better served with a circular log
> that records as many log records as fit into a finite, operator-specified log
> file size. Or am I wrong here? (If nobody tells me I am wrong, I'll probably
> remove the b) use case in the long term!).

no, it's not that total space for logs is limited to a fixed amount, it's 
that you have two spaces for logs, a small fast space and a large slow 
space, and you need to not run out of the fast space by moving logs to the 
slow space 'soon enough'


from a security point of view there are two arguments about option b

on the one hand, if a bad guy can run you out of space he can then do new 
things that aren't logged.

on the other hand, if you have a circular buffer, the bad buy can generate 
enough logs to cause you to overwrite the evidence of him breaking in to 
your system

there are places for both.

David Lang


From sfzhi at yahoo.com  Tue Jun 30 15:24:53 2009
From: sfzhi at yahoo.com (Sergei Zhirikov)
Date: Tue, 30 Jun 2009 15:24:53 +0200
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com><h2ckri$16j$1@ger.gmane.org>	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
Message-ID: <h2d3n6$cp5$1@ger.gmane.org>

> From the feedback I received so far, there seem to be two use cases:
> 
> a) circular logs
> b) stop logging when file hits max size (was that really what this post
> described?)
> 
> If I look at the b) case, I suspect that this is not even the intent of the
> original poster. I think he would be much better served with a circular log
> that records as many log records as fit into a finite, operator-specified log
> file size. Or am I wrong here? (If nobody tells me I am wrong, I'll probably
> remove the b) use case in the long term!).
> 
> Rainer
> 

In the case of log flood neither of the two options is perfect. The 
difference is only in what you lose, the oldest or the newest log 
entries. What I really like about the current rsyslog functionality is 
the possibility to call an external process to deal with the oversized 
logs. Stopping logging I consider only a safeguard in case that external 
process fails to do its job for whatever reason.

Having an external process makes the whole setup a lot more flexible, 
because it gives almost unlimited opportunities, for example:

   * gzip the older logs rather than delete them. In case of log flood 
gzipping is usually very efficient, because the same message is repeated 
in the log multiple times, so there is no need to delete anything.
   * filter the older logs based on conditions that involve analyzing 
more than one log entry at the same time. This can be useful, for 
example, do detect break-in attempts if desired.
   * measure the average speed of logging and send an email or generate 
another kind of alert when the logs are growing unusually fast.
   * make a decision whether to gzip the older logs, to move them to 
another location (e.g. a network storage, if available), or to delete 
them based on conditions outside of control of rsyslog (e.g. available 
system resources, availability of alternative storage, etc.)

The list can go on infinitely. I believe the possibility to call an 
external process to handle log rotation gives the flexibility that can 
not be provided within rsyslog itself, because there are so many 
different things one might want to do. And I would very much like to see 
this functionality stay in one or another form. Hope I'm not the only 
one who thinks that way.

Kind regards,
Sergei.



From epiphani at gmail.com  Tue Jun 30 15:38:11 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Tue, 30 Jun 2009 09:38:11 -0400
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <h2d3n6$cp5$1@ger.gmane.org>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
	<h2ckri$16j$1@ger.gmane.org>
	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
	<h2d3n6$cp5$1@ger.gmane.org>
Message-ID: <e7ca40f70906300638n2b7dd90eo49429daec8caddc2@mail.gmail.com>

On Tue, Jun 30, 2009 at 9:24 AM, Sergei Zhirikov<sfzhi at yahoo.com> wrote:
>
> ? * gzip the older logs rather than delete them. In case of log flood
> gzipping is usually very efficient, because the same message is repeated
> in the log multiple times, so there is no need to delete anything.

One thing worth noting is that the new omfile branch contains changes
that will allow gzipping of files automatically out of rsyslogd, which
removes the requirement to do it after writing to disk.  :)

-Aaron


From aoz.syn at gmail.com  Tue Jun 30 16:44:55 2009
From: aoz.syn at gmail.com (RB)
Date: Tue, 30 Jun 2009 08:44:55 -0600
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
	<h2ckri$16j$1@ger.gmane.org>
	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
Message-ID: <4255c2570906300744h5db55712m135c67aa2d15bcea@mail.gmail.com>

On Tue, Jun 30, 2009 at 04:43, Rainer Gerhards<rgerhards at hq.adiscon.com> wrote:
> a) circular logs

This.  I keep getting distracted with other projects, but have had on
my TODO list to add an 'omclog' output module that both handles the
*BSD clog format and extends it for some particular shortcomings.
Specifically, I'm thinking of embedded concerns here - compression,
ringing on number of records, and a few miscallaneous implementation
oversights it (clog) has.

> b) stop logging when file hits max size (was that really what this post
> described?)

As noted, most people want this use-case to cut off logs when they hit
a particular size so they may be archived/processed/etc.  I honestly
couldn't care less whether rsyslog has internal compression
capability, as long as it closes the handle and releases it to me to
deal with.  Maybe a hook to call a program when the size trigger is
hit, but even with [di]notify that becomes less necessary.


From sfzhi at yahoo.com  Tue Jun 30 18:12:28 2009
From: sfzhi at yahoo.com (Sergei Zhirikov)
Date: Tue, 30 Jun 2009 18:12:28 +0200
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <4255c2570906300744h5db55712m135c67aa2d15bcea@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>	<h2ckri$16j$1@ger.gmane.org>	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>	<9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
	<4255c2570906300744h5db55712m135c67aa2d15bcea@mail.gmail.com>
Message-ID: <h2ddhf$8q5$1@ger.gmane.org>

RB wrote:
> I honestly couldn't care less whether rsyslog has internal compression
> capability, as long as it closes the handle and releases it to me to
> deal with.  Maybe a hook to call a program when the size trigger is
> hit, but even with [di]notify that becomes less necessary.

I completely agree.

By the way, I use the inotify method to handle logs written by Samba 
server, which does have a log size limit, but does not call an external 
program when that limit is exceeded. So far I did not have any problems 
with that setup.

--
Sergei.



From david at lang.hm  Tue Jun 30 23:46:38 2009
From: david at lang.hm (david at lang.hm)
Date: Tue, 30 Jun 2009 14:46:38 -0700 (PDT)
Subject: [rsyslog] v5-devel won't compile
Message-ID: <alpine.DEB.1.10.0906301445160.9159@asgard>

ubuntu 9.04

zlib version
/* zlib.h -- interface of the 'zlib' general purpose compression library
   version 1.2.3.3, October 2nd, 2006

ii  zlib1g-dev                          1:1.2.3.3.dfsg-12 
compression library - development


error

In file included from zlibw.h:28,
                  from stream.h:72,
                  from obj.h:50,
                  from rsyslog.h:482,
                  from stringbuf.c:39:
/usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or 
'__attribute__' before 'gzseek64'
/usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or 
'__attribute__' before 'gztell64'
/usr/include/zlib.h:1368: error: expected declaration specifiers or '...' 
before 'off64_t'
/usr/include/zlib.h:1369: error: expected declaration specifiers or '...' 
before 'off64_t'
make[2]: *** [librsyslog_la-stringbuf.lo] Error 1
make[2]: Leaving directory `/usr/src/rsyslog/runtime'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/usr/src/rsyslog'
make: *** [all] Error 2


David Lang


From iamsayan at gmail.com  Tue Jun  2 23:42:11 2009
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Tue, 2 Jun 2009 17:42:11 -0400
Subject: [rsyslog] Property replacer manipulation question
Message-ID: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com>

Hello All,

This is my first post to this group, apologize if the question seems naive.

I am trying to change strings within the syslog message before forwarding it
to an external log server. For example, if my regular expression matches an
mac or an IP address present within the message, I want to replace it with a
string like this "<hidden ip>"
Another requirement is before forwarding the log to the external server, I
have to change the facility to a different number (say from local0 to
local7)

The approach I was taking is defining a template with the regular expression
in it, checking for the pattern and then replacing the mac/ip address.

However, I think this kind of pattern replacement is not supported with the
property replacer.
Is that correct?

I am using rsyslog version 3.21 to try this out and here is an example of
what I am trying to do

A sample log message

May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE|All elements Connected

I need to change it to

May 25 08:41:13. [Hidden MAC][12703]: 4|172|CORE|All elements Connected


Before it gets forwarded to the external machine.

Regards,
Sayan


From david at lang.hm  Tue Jun  2 23:50:40 2009
From: david at lang.hm (david at lang.hm)
Date: Tue, 2 Jun 2009 14:50:40 -0700 (PDT)
Subject: [rsyslog] Property replacer manipulation question
In-Reply-To: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com>
References: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906021445440.7953@asgard>

On Tue, 2 Jun 2009, Sayan Chowdhury wrote:

> Hello All,
>
> This is my first post to this group, apologize if the question seems naive.
>
> I am trying to change strings within the syslog message before forwarding it
> to an external log server. For example, if my regular expression matches an
> mac or an IP address present within the message, I want to replace it with a
> string like this "<hidden ip>"
> Another requirement is before forwarding the log to the external server, I
> have to change the facility to a different number (say from local0 to
> local7)
>
> The approach I was taking is defining a template with the regular expression
> in it, checking for the pattern and then replacing the mac/ip address.
>
> However, I think this kind of pattern replacement is not supported with the
> property replacer.
> Is that correct?
>
> I am using rsyslog version 3.21 to try this out and here is an example of
> what I am trying to do
>
> A sample log message
>
> May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE|All elements Connected
>
> I need to change it to
>
> May 25 08:41:13. [Hidden MAC][12703]: 4|172|CORE|All elements Connected
>
>
> Before it gets forwarded to the external machine.

in this case I believe that the MAC would show up in the programname (and 
syslog tag) fields.

so instead of the default format which looks something like
%timestamp% %hostname% %syslogtag%%msg%

you would do something like
%timestamp% %hostname% [Hidden MAC]%msg%

and setup the conditions so that if it matches your condition it sends it 
out with the second format, but if it doesn't it uses the first.

this isn't the search-and-replace functionality you were thinking of, but 
I think it will do the basic job (although the example I gave doesn't 
include the PID portion of the syslog tag, doing a substring on the syslog 
tag would let you add that back in if needed)

David Lang


From iamsayan at gmail.com  Wed Jun  3 01:56:37 2009
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Tue, 2 Jun 2009 19:56:37 -0400
Subject: [rsyslog] Property replacer manipulation question
In-Reply-To: <alpine.DEB.1.10.0906021445440.7953@asgard>
References: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com>
	<alpine.DEB.1.10.0906021445440.7953@asgard>
Message-ID: <87a1ae540906021656s34696amfc3f2e327a51b4be@mail.gmail.com>

Hello David,
Thanks for replying.

I think I should have made my template clearer as well ...
Actually these mac/ip addresses appear in the message portion. with
different kind of logging message that the system generates
so my rule should work for for a message like this as well


May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE| IP 10.0.0.87 assigned to
device DEVICE_0
should become something like


May 25 08:41:13. [HIDDEN MAC][12703]: 4|172|CORE| IP [HIDDEN IP] assigned to
device DEVICE_0


also, the other requirement I have is to change the facility Id in the
outgoing syslog message.

That's when I thought about a search and replace functionality with the
property replacer ...
Not sure whether it is possible though, so far I have been able to write
property replacers which can drop line feed, or replace / with an _ but not
something like this.

Regards,
Sayan


On Tue, Jun 2, 2009 at 5:50 PM, <david at lang.hm> wrote:

> On Tue, 2 Jun 2009, Sayan Chowdhury wrote:
>
> > Hello All,
> >
> > This is my first post to this group, apologize if the question seems
> naive.
> >
> > I am trying to change strings within the syslog message before forwarding
> it
> > to an external log server. For example, if my regular expression matches
> an
> > mac or an IP address present within the message, I want to replace it
> with a
> > string like this "<hidden ip>"
> > Another requirement is before forwarding the log to the external server,
> I
> > have to change the facility to a different number (say from local0 to
> > local7)
> >
> > The approach I was taking is defining a template with the regular
> expression
> > in it, checking for the pattern and then replacing the mac/ip address.
> >
> > However, I think this kind of pattern replacement is not supported with
> the
> > property replacer.
> > Is that correct?
> >
> > I am using rsyslog version 3.21 to try this out and here is an example of
> > what I am trying to do
> >
> > A sample log message
> >
> > May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE|All elements Connected
> >
> > I need to change it to
> >
> > May 25 08:41:13. [Hidden MAC][12703]: 4|172|CORE|All elements Connected
> >
> >
> > Before it gets forwarded to the external machine.
>
> in this case I believe that the MAC would show up in the programname (and
> syslog tag) fields.
>
> so instead of the default format which looks something like
> %timestamp% %hostname% %syslogtag%%msg%
>
> you would do something like
> %timestamp% %hostname% [Hidden MAC]%msg%
>
> and setup the conditions so that if it matches your condition it sends it
> out with the second format, but if it doesn't it uses the first.
>
> this isn't the search-and-replace functionality you were thinking of, but
> I think it will do the basic job (although the example I gave doesn't
> include the PID portion of the syslog tag, doing a substring on the syslog
> tag would let you add that back in if needed)
>
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From david at lang.hm  Wed Jun  3 04:04:16 2009
From: david at lang.hm (david at lang.hm)
Date: Tue, 2 Jun 2009 19:04:16 -0700 (PDT)
Subject: [rsyslog] Property replacer manipulation question
In-Reply-To: <87a1ae540906021656s34696amfc3f2e327a51b4be@mail.gmail.com>
References: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com>
	<alpine.DEB.1.10.0906021445440.7953@asgard>
	<87a1ae540906021656s34696amfc3f2e327a51b4be@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906021901220.7953@asgard>

On Tue, 2 Jun 2009, Sayan Chowdhury wrote:

> Hello David,
> Thanks for replying.
>
> I think I should have made my template clearer as well ...
> Actually these mac/ip addresses appear in the message portion. with
> different kind of logging message that the system generates
> so my rule should work for for a message like this as well
>
>
> May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE| IP 10.0.0.87 assigned to
> device DEVICE_0
> should become something like
>
>
> May 25 08:41:13. [HIDDEN MAC][12703]: 4|172|CORE| IP [HIDDEN IP] assigned to
> device DEVICE_0
>
>
> also, the other requirement I have is to change the facility Id in the
> outgoing syslog message.

there are few enough facility Ids that you can put them into the template

> That's when I thought about a search and replace functionality with the
> property replacer ...
> Not sure whether it is possible though, so far I have been able to write
> property replacers which can drop line feed, or replace / with an _ but not
> something like this.

I haven't seen any ability to do arbatrary search-and-replace in rsyslog.

David Lang


> Regards,
> Sayan
>
>
> On Tue, Jun 2, 2009 at 5:50 PM, <david at lang.hm> wrote:
>
>> On Tue, 2 Jun 2009, Sayan Chowdhury wrote:
>>
>>> Hello All,
>>>
>>> This is my first post to this group, apologize if the question seems
>> naive.
>>>
>>> I am trying to change strings within the syslog message before forwarding
>> it
>>> to an external log server. For example, if my regular expression matches
>> an
>>> mac or an IP address present within the message, I want to replace it
>> with a
>>> string like this "<hidden ip>"
>>> Another requirement is before forwarding the log to the external server,
>> I
>>> have to change the facility to a different number (say from local0 to
>>> local7)
>>>
>>> The approach I was taking is defining a template with the regular
>> expression
>>> in it, checking for the pattern and then replacing the mac/ip address.
>>>
>>> However, I think this kind of pattern replacement is not supported with
>> the
>>> property replacer.
>>> Is that correct?
>>>
>>> I am using rsyslog version 3.21 to try this out and here is an example of
>>> what I am trying to do
>>>
>>> A sample log message
>>>
>>> May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE|All elements Connected
>>>
>>> I need to change it to
>>>
>>> May 25 08:41:13. [Hidden MAC][12703]: 4|172|CORE|All elements Connected
>>>
>>>
>>> Before it gets forwarded to the external machine.
>>
>> in this case I believe that the MAC would show up in the programname (and
>> syslog tag) fields.
>>
>> so instead of the default format which looks something like
>> %timestamp% %hostname% %syslogtag%%msg%
>>
>> you would do something like
>> %timestamp% %hostname% [Hidden MAC]%msg%
>>
>> and setup the conditions so that if it matches your condition it sends it
>> out with the second format, but if it doesn't it uses the first.
>>
>> this isn't the search-and-replace functionality you were thinking of, but
>> I think it will do the basic job (although the example I gave doesn't
>> include the PID portion of the syslog tag, doing a substring on the syslog
>> tag would let you add that back in if needed)
>>
>> David Lang
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Wed Jun  3 07:36:49 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Jun 2009 07:36:49 +0200
Subject: [rsyslog] Property replacer manipulation question
References: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com><alpine.DEB.1.10.0906021445440.7953@asgard><87a1ae540906021656s34696amfc3f2e327a51b4be@mail.gmail.com>
	<alpine.DEB.1.10.0906021901220.7953@asgard>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F853@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Wednesday, June 03, 2009 4:04 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Property replacer manipulation question
> 
> On Tue, 2 Jun 2009, Sayan Chowdhury wrote:
> 
> > Hello David,
> > Thanks for replying.
> >
> > I think I should have made my template clearer as well ...
> > Actually these mac/ip addresses appear in the message portion. with
> > different kind of logging message that the system generates
> > so my rule should work for for a message like this as well
> >
> >
> > May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE| IP 10.0.0.87 assigned
to
> > device DEVICE_0
> > should become something like
> >
> >
> > May 25 08:41:13. [HIDDEN MAC][12703]: 4|172|CORE| IP [HIDDEN IP] assigned
to
> > device DEVICE_0
> >
> >
> > also, the other requirement I have is to change the facility Id in the
> > outgoing syslog message.
> 
> there are few enough facility Ids that you can put them into the template
> 
> > That's when I thought about a search and replace functionality with the
> > property replacer ...
> > Not sure whether it is possible though, so far I have been able to write
> > property replacers which can drop line feed, or replace / with an _ but
not
> > something like this.
> 
> I haven't seen any ability to do arbatrary search-and-replace in rsyslog.

This is not available, but sounds like a good idea. I suggest to add a
feature enhancement ticket to the bug tracker. Be warned, though, that I
currently simply have no time to implement it. But it may happen as a
by-activity when I change things in the property replacer, so it would be
good not to forget about it.

Rainer


From rgerhards at hq.adiscon.com  Wed Jun  3 18:33:19 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Jun 2009 18:33:19 +0200
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>

Hi all,

I have begun to redesign the pretty old omfile output plugin. Large parts of
this work are covered by a sponsorship.

I will probably post a number of questions during the redesign process.
Please speak up if you have any opinion on them. Once I have finished the new
design, it is not likely to change during the next three to five years. For
the same reason, please speak up now if you are desperately missing something
in omfile.

A concrete question I have is on the $OutChannel directive[1]. It is
scheduled for replacement for quite a while, and now may be the time that it
finally goes away. If you use $OutChannel, please let me know why.

The most prominent use case is to size-limit files. $OutChannel provides the
ability to execute a command when a certain file size is reached. The command
may then rotate the file. After command execution, omfile checks the file
size again. If it still is above the configured limit, this file will be
disabled (no longer be written to until restart).

If you use this for size-limiting, please let me know how you do it. Please
also let me know if you would like to see things handled differently.

Looking forward to your feedback.

Thanks,
Rainer


[1] http://www.rsyslog.com/doc-rsyslog_conf_output.html


From iamsayan at gmail.com  Wed Jun  3 19:28:24 2009
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Wed, 3 Jun 2009 13:28:24 -0400
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
Message-ID: <87a1ae540906031028s3584fbdat428ca35a17f638ef@mail.gmail.com>

Hello Rainer,

I do use this facility. This is how I do it

$outchannel log_rotation_16,/var/private/log/logsip,
10000000,/opt/platform/bin/log_rotate.pl

if $syslogfacility == '16' and $syslogseverity <= '6' then
$log_rotation_16;precise

The log_rotate.pl perl script that I have rotates the files
and maintains a set of 10 files. When the current file reaches the 10 mb
limit, the older file is zipped and copied over to a different directory.
I am on release 3.21 now

I would definitely like to have the feature be present in the future
releases as well. I am happy as long as this feature of file size based
rotation is there and I can migrate my rules/scripts to whatever new
mechanism that get implemented.

Regards,
Sayan






On Wed, Jun 3, 2009 at 12:33 PM, Rainer Gerhards
<rgerhards at hq.adiscon.com>wrote:

> Hi all,
>
> I have begun to redesign the pretty old omfile output plugin. Large parts
> of
> this work are covered by a sponsorship.
>
> I will probably post a number of questions during the redesign process.
> Please speak up if you have any opinion on them. Once I have finished the
> new
> design, it is not likely to change during the next three to five years. For
> the same reason, please speak up now if you are desperately missing
> something
> in omfile.
>
> A concrete question I have is on the $OutChannel directive[1]. It is
> scheduled for replacement for quite a while, and now may be the time that
> it
> finally goes away. If you use $OutChannel, please let me know why.
>
> The most prominent use case is to size-limit files. $OutChannel provides
> the
> ability to execute a command when a certain file size is reached. The
> command
> may then rotate the file. After command execution, omfile checks the file
> size again. If it still is above the configured limit, this file will be
> disabled (no longer be written to until restart).
>
> If you use this for size-limiting, please let me know how you do it. Please
> also let me know if you would like to see things handled differently.
>
> Looking forward to your feedback.
>
> Thanks,
> Rainer
>
>
> [1] http://www.rsyslog.com/doc-rsyslog_conf_output.html
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Thu Jun  4 11:01:46 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 4 Jun 2009 11:01:46 +0200
Subject: [rsyslog] request for feedback: dynafile cache size
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F879@GRFEXC.intern.adiscon.com>

Hi all,

as you probably know, dynafiles keep a cache of the n most recently used
files open. This is done in order to speed up processing. I wonder how many
files you usually expect to be open. The root of my question is that I think
about the algorithm that finds cache entries. So far, a simple linked list is
used, but I wonder if something more efficient (like a binary tree) would be
useful. On the other hand, if the cache size is really small, there is not
much difference between O(n) and O(log n), so it may not justify added code
complexity.

Feedback appreciated.

Rainer


From david at lang.hm  Thu Jun  4 13:50:42 2009
From: david at lang.hm (david at lang.hm)
Date: Thu, 4 Jun 2009 04:50:42 -0700 (PDT)
Subject: [rsyslog] request for feedback: dynafile cache size
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F879@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F879@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906040448240.11064@asgard>

On Thu, 4 Jun 2009, Rainer Gerhards wrote:

> Hi all,
>
> as you probably know, dynafiles keep a cache of the n most recently used
> files open. This is done in order to speed up processing. I wonder how many
> files you usually expect to be open. The root of my question is that I think
> about the algorithm that finds cache entries. So far, a simple linked list is
> used, but I wonder if something more efficient (like a binary tree) would be
> useful. On the other hand, if the cache size is really small, there is not
> much difference between O(n) and O(log n), so it may not justify added code
> complexity.
>
> Feedback appreciated.

I have a use case where it could be several thousand files

I would expect the normal worst case to be dozens to a few hundred files.

David Lang


From rgerhards at hq.adiscon.com  Thu Jun  4 13:52:18 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 4 Jun 2009 13:52:18 +0200
Subject: [rsyslog] request for feedback: dynafile cache size
References: <9B6E2A8877C38245BFB15CC491A11DA706F879@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906040448240.11064@asgard>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F888@GRFEXC.intern.adiscon.com>

Thanks for the feedback. That definitely asks for a binary tree search. So I
am convinced now (I'll not do it immediately, but probably towards the end of
my redesign work).

Raienr

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Thursday, June 04, 2009 1:51 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] request for feedback: dynafile cache size
> 
> On Thu, 4 Jun 2009, Rainer Gerhards wrote:
> 
> > Hi all,
> >
> > as you probably know, dynafiles keep a cache of the n most recently used
> > files open. This is done in order to speed up processing. I wonder how
many
> > files you usually expect to be open. The root of my question is that I
think
> > about the algorithm that finds cache entries. So far, a simple linked
list
> is
> > used, but I wonder if something more efficient (like a binary tree) would
be
> > useful. On the other hand, if the cache size is really small, there is
not
> > much difference between O(n) and O(log n), so it may not justify added
code
> > complexity.
> >
> > Feedback appreciated.
> 
> I have a use case where it could be several thousand files
> 
> I would expect the normal worst case to be dozens to a few hundred files.
> 
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From nathan at gossamer-threads.com  Sat Jun  6 00:09:16 2009
From: nathan at gossamer-threads.com (Nathan March)
Date: Fri, 05 Jun 2009 15:09:16 -0700
Subject: [rsyslog] Possible bug? 100% cpu on a server thread and
 the	client stops sending data
In-Reply-To: <4A1A57A0.1060903@net-m.de>
References: <4A15999D.3000909@gossamer-threads.com>
	<4A1A37DE.8050004@net-m.de>	<9B6E2A8877C38245BFB15CC491A11DA702B0FB@GRFEXC.intern.adiscon.com>
	<4A1A57A0.1060903@net-m.de>
Message-ID: <4A29978C.9080505@gossamer-threads.com>

Any hope of this bug getting some attention soon? =)

Thanks,
Nathan


Patrick Shen wrote:
> Hi Rainer,
>
> Thanks for the quick reply.
>
> My environment is like below:
>
> CLIENT ---
>          |
>          | (TCP)
>          --------> SERVER ----
>                              |
>                              | (RELP Relay)
>                              --------------> SERVER Standby
>
> I've tested, even if the standby server is under high load, one of our
> clients won't accept any new network connections.
>
> Just for your information.
>
> Best regards,
> Patrick
>
> Rainer Gerhards wrote:
>   
>> Hi all,
>>
>> thanks for the detailed info. Will look at it asap, but I am currently tied
>> up with some other work...
>>
>> Please let me know anything else that may be relevant.
>>
>> Thanks,
>> Rainer
>>
>>     
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of Patrick Shen
>>> Sent: Monday, May 25, 2009 8:17 AM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server thread and the
>>> client stops sending data
>>>
>>> Hi Nathan,
>>>
>>> In my company, we have the similar symptom. We have 50+ servers which
>>> send logs to the central logging servers. Both clients and server use
>>> rsyslog v3.20.5 right now. In the past when we use v3.20.0, the symptom
>>> is more common when server is under high load average.
>>>
>>> It's also weird for us, not all of clients were lost of connections. We
>>> have 2-3 clients which is running the same application will looks like
>>> hanging up when server is high-load. New ssh connection is very slow at
>>> that time. But the connected ssh session works normal (We have some
>>> prepared screen session on other clients which connect to them via ssh).
>>>
>>> When the clients are lost of connections. So far our solution is to
>>> restart rsyslog both on affected clients and server. Then the clients
>>> get back and look normal.
>>>
>>> We've suffered it for a while, but I didn't take chance to analyze it
>>> like you.
>>>
>>> Best regards,
>>> Patrick
>>>
>>> Nathan March wrote:
>>>       
>>>> Having a weird issue here..... Testing out a new rsyslog deployment and
>>>> I've got around 30 servers logging to one machine. On one of the clients
>>>> we had an issue where ssh was mysteriously very slow to login and
>>>> tracked it down to rsyslog causing issues. Simultaneously, one of the
>>>> threads on the logging server spiked up to 100%. New log data from the
>>>> client would stop showing up on the server at this poitn.
>>>>
>>>> On the client after attempting to gracefully stop rsyslog:
>>>>
>>>> vanguard etc # ps aux | grep -i rsyslog
>>>> root      3764  0.0  0.1  5092 2564 ?        S    15:06   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f /etc/rsyslog.conf
>>>> root      3766  0.0  0.1  5092 2564 ?        S    15:06   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f /etc/rsyslog.conf
>>>> root      6203  0.0  0.0  1512  524 pts/3    S    15:09   0:00 grep -i
>>>> rsyslog
>>>> vanguard etc # strace -p 3764
>>>> Process 3764 attached - interrupt to quit
>>>> send(11, "\25\3\2\1\0uE\254\304\n\311\377\204}p\237O\211\322\211"...,
>>>> 261, 0 <unfinished ...>
>>>> Process 3764 detached
>>>>
>>>> FD 11 being the socket to the logging server, it never does anything
>>>> asides from that send.
>>>>
>>>> On the server:
>>>>
>>>> ldap ~ # ps aux | grep rsyslog
>>>> root      7370  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7379  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7380  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7381  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7382  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7383  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7384  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7385 28.3  0.0  83936  3380 ?        R    14:54   5:07
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> Stracing the 7385 pid just shows this repeating over and over (FD 4
>>>> being the network socket again):
>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 EAGAIN (Resource
>>>> temporarily unavailable)
>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 EAGAIN (Resource
>>>> temporarily unavailable)
>>>>
>>>> It happened 3 times within a relatively short period of time, sometimes
>>>> within minutes of me restarting rsyslog on both machines. I enabled
>>>> debugging and it took around 8 hours to pop up again.
>>>>
>>>> With debugging, I pulled this from the rsyslog server:
>>>>
>>>> 6640.408108000:imtcp.c: New connect on NSD 0x59ddc0.
>>>> 6640.408129000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: pszRcvBuf (nil),
>>>> lenRcvBuf 0
>>>> 6640.408144000:imtcp.c: hasRcvInBuffer on nsd 0x59dcd0: pszRcvBuf
>>>> 0x5c11c0, lenRcvBuf -1
>>>> 6640.408158000:imtcp.c: hasRcvInBuffer on nsd 0x59dc10: pszRcvBuf
>>>> 0x5cb510, lenRcvBuf -1
>>>> 6640.408171000:imtcp.c: hasRcvInBuffer on nsd 0x5c3d70: pszRcvBuf
>>>> 0x5d45c0, lenRcvBuf -1
>>>> 6640.408185000:imtcp.c: hasRcvInBuffer on nsd 0x5cf460: pszRcvBuf
>>>> 0x5dd010, lenRcvBuf -1
>>>> 6640.408199000:imtcp.c: hasRcvInBuffer on nsd 0x5d8600: pszRcvBuf
>>>> 0x5e58b0, lenRcvBuf -1
>>>> 6640.408213000:imtcp.c: hasRcvInBuffer on nsd 0x59d690: pszRcvBuf
>>>> 0x5ee150, lenRcvBuf -1
>>>> 6640.408227000:imtcp.c: hasRcvInBuffer on nsd 0x59d2f0: pszRcvBuf
>>>> 0x5f7320, lenRcvBuf -1
>>>> 6640.408241000:imtcp.c: hasRcvInBuffer on nsd 0x5e8840: pszRcvBuf
>>>> 0x5fe0a0, lenRcvBuf -1
>>>> 6640.408255000:imtcp.c: hasRcvInBuffer on nsd 0x6027b0: pszRcvBuf
>>>> 0x608350, lenRcvBuf -1
>>>> 6640.408269000:imtcp.c: hasRcvInBuffer on nsd 0x5c3660: pszRcvBuf
>>>> 0x612110, lenRcvBuf -1
>>>> 6640.408283000:imtcp.c: hasRcvInBuffer on nsd 0x602690: pszRcvBuf
>>>> 0x615040, lenRcvBuf -1
>>>> 6640.408296000:imtcp.c: hasRcvInBuffer on nsd 0x5c3b60: pszRcvBuf
>>>> 0x6218f0, lenRcvBuf -1
>>>> 6640.408310000:imtcp.c: hasRcvInBuffer on nsd 0x60c150: pszRcvBuf
>>>> 0x62a190, lenRcvBuf -1
>>>> 6640.408324000:imtcp.c: hasRcvInBuffer on nsd 0x625420: pszRcvBuf
>>>> 0x634fe0, lenRcvBuf -1
>>>> 6640.408338000:imtcp.c: hasRcvInBuffer on nsd 0x6117b0: pszRcvBuf
>>>> 0x63b000, lenRcvBuf -1
>>>> 6640.408352000:imtcp.c: hasRcvInBuffer on nsd 0x603990: pszRcvBuf
>>>> 0x643c70, lenRcvBuf -1
>>>> 6640.408365000:imtcp.c: hasRcvInBuffer on nsd 0x633c00: pszRcvBuf
>>>> 0x64dbb0, lenRcvBuf -1
>>>> 6640.408379000:imtcp.c: hasRcvInBuffer on nsd 0x62cfa0: pszRcvBuf
>>>> 0x650960, lenRcvBuf -1
>>>> 6640.408393000:imtcp.c: hasRcvInBuffer on nsd 0x633d10: pszRcvBuf
>>>> 0x65dbc0, lenRcvBuf -1
>>>> 6640.408407000:imtcp.c: hasRcvInBuffer on nsd 0x5f2860: pszRcvBuf
>>>> 0x666020, lenRcvBuf -1
>>>> 6640.408421000:imtcp.c: hasRcvInBuffer on nsd 0x64d250: pszRcvBuf
>>>> 0x66e480, lenRcvBuf -1
>>>> 6640.408435000:imtcp.c: hasRcvInBuffer on nsd 0x659b30: pszRcvBuf
>>>> 0x676d20, lenRcvBuf 78
>>>> 6640.408448000:imtcp.c: hasRcvInBuffer on nsd 0x669fb0: pszRcvBuf
>>>> 0x67fdd0, lenRcvBuf -1
>>>> 6640.408462000:imtcp.c: hasRcvInBuffer on nsd 0x59a630: pszRcvBuf
>>>> 0x687f10, lenRcvBuf -1
>>>> 6640.408476000:imtcp.c: hasRcvInBuffer on nsd 0x6614c0: pszRcvBuf
>>>> 0x6907b0, lenRcvBuf -1
>>>> 6640.408490000:imtcp.c: hasRcvInBuffer on nsd 0x60c2d0: pszRcvBuf
>>>> 0x699050, lenRcvBuf -1
>>>> 6640.408504000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: pszRcvBuf (nil),
>>>> lenRcvBuf 0
>>>>
>>>> This repeats enough to generate around 29 million lines in the debug log
>>>> file.
>>>>
>>>> In the client log, there's plenty of these which seems somewhat normal:
>>>>
>>>> 6509.743164000:imuxsock.c: --------imuxsock calling select, active file
>>>> descriptors (max 11): 11
>>>> 6509.743231000:imuxsock.c: Message from UNIX socket: #11
>>>> 6509.743262000:imuxsock.c: logmsg: flags 4, from 'vanguard', msg May 20
>>>> 23:15:09 bin/qmail-local[23198]: using .qmail .qmail-cfsc-forum
>>>> 6509.743281000:imuxsock.c: Message has legacy syslog format.
>>>> 6509.743302000:imuxsock.c: main queue: entry added, size now 579 entries
>>>> 6509.743325000:imuxsock.c: wtpAdviseMaxWorkers signals busy
>>>> 6509.743343000:imuxsock.c: main queue: EnqueueMsg advised worker start
>>>>
>>>> However comparing it to an earlier point in the log  there's none of the
>>>> associated tcp queue entries like this:
>>>>
>>>> 3587.234178000:main queue:Reg/w0: TCP sent 107 bytes, requested 107
>>>> 3587.234204000:main queue:Reg/w0: main queue: entering rate limiter
>>>> 3587.234229000:main queue:Reg/w0: main queue: entry deleted, state 0,
>>>> size now 0 entries
>>>> 3587.234255000:main queue:Reg/w0: Called action, logging to builtin-file
>>>> 3587.234285000:main queue:Reg/w0:  (/var/log/maillog)
>>>> 3587.234314000:main queue:Reg/w0: Called action, logging to builtin-fwd
>>>> 3587.234340000:main queue:Reg/w0:  ldap.nmsrv.com
>>>> 3587.234359000:main queue:Reg/w0:  ldap.nmsrv.com:10514/tcp
>>>> 3587.234425000:main queue:Reg/w0: TCP sent 78 bytes, requested 78
>>>> 3587.234446000:main queue:Reg/w0: main queue: entering rate limiter
>>>> 3587.234471000:main queue:Reg/w0: main queue:Reg/w0: worker IDLE,
>>>> waiting for work.
>>>>
>>>> Anyone able to shed some light on this situation? I've got the full
>>>> debug logs if it's useful to anyone to take a closer look.
>>>>
>>>> Thanks!
>>>>
>>>> - Nathan
>>>>
>>>>
>>>>
>>>>         
>> ----------------------------------------------------------------------------
>>     
>>> --------
>>>       
>>>> Server config file
>>>>
>>>>         
>> ----------------------------------------------------------------------------
>>     
>>> --------
>>>       
>>>> $ModLoad immark.so # provides --MARK-- message capability
>>>> $ModLoad imuxsock.so # provides support for local system logging (e.g.
>>>> via logger command)
>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
>>>>
>>>> $umask 0137
>>>> $DirCreateMode 0640
>>>> $FileCreateMode 0640
>>>> $FileOwner root
>>>> $FileGroup admin
>>>>
>>>> $template DynFile,"/var/log/rsyslog/%HOSTNAME%/messages"
>>>> $template DynFileCron,"/var/log/rsyslog/%HOSTNAME%/cron"
>>>> $template DynFileSecure,"/var/log/rsyslog/%HOSTNAME%/secure"
>>>> $template DynFileMail,"/var/log/rsyslog/%HOSTNAME%/maillog"
>>>> $template DynFileSpool,"/var/log/rsyslog/%HOSTNAME%/spooler"
>>>> $template DynFileBoot,"/var/log/rsyslog/%HOSTNAME%/boot"
>>>> $template DynFileSyslog,"/var/log/rsyslog/%HOSTNAME%/syslog"
>>>>
>>>> :msg, contains, "no keys found for"            ~
>>>> :msg, contains, "session opened for user"        ~
>>>>
>>>> # Log all kernel messages to the console.
>>>> # Logging much else clutters up the screen.
>>>> kern.*                                                 /dev/console
>>>>
>>>> # Log anything (except mail) of level info or higher.
>>>> # Don't log private authentication messages!
>>>> *.info;mail.none;cron.none                     ?DynFile
>>>>
>>>> *.warn;\
>>>>         authpriv.none;cron.none;mail.none;news.none     ?DynFileSyslog
>>>>
>>>> # Log all the mail messages in one place.
>>>> mail.*                                                  ?DynFileMail
>>>>
>>>> # Log cron stuff
>>>> cron.*                                                  ?DynFileCron
>>>>
>>>> # Everybody gets emergency messages
>>>> *.emerg                                                 *
>>>>
>>>> # Save news errors of level crit and higher in a special file
>>>> uucp,news.crit                                          ?DynFileSpool
>>>>
>>>> # Save boot messages also to boot.log
>>>> local7.*                                                ?DynFileBoot
>>>>
>>>> # Relp config for PCI
>>>> $ModLoad imrelp
>>>> $InputRELPServerRun 2515
>>>>
>>>> # make gtls driver the default
>>>> $DefaultNetstreamDriver gtls
>>>>
>>>> # certificate files
>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
>>>> $DefaultNetstreamDriverCertFile /etc/ssl/servercrt.pem
>>>> $DefaultNetstreamDriverKeyFile /etc/ssl/serverkey.pem
>>>>
>>>> $ModLoad imtcp.so
>>>>
>>>> $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
>>>> $InputTCPServerStreamDriverAuthMode anon # client is NOT authenticated
>>>> $InputTCPServerRun 10514 # start up listener at port 10514
>>>> $InputTCPMaxSessions 500
>>>>
>>>>
>>>>         
>> ----------------------------------------------------------------------------
>>     
>>> --------
>>>       
>>>> Client config file
>>>>
>>>>         
>> ----------------------------------------------------------------------------
>>     
>>> --------
>>>       
>>>> $ModLoad imuxsock.so # provides support for local system logging (e.g.
>>>> via logger command)
>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
>>>>
>>>> $umask 0137
>>>> $DirCreateMode 0640
>>>> $FileCreateMode 0640
>>>> $FileOwner root
>>>> $FileGroup admin
>>>>
>>>>
>>>> # Log anything 'info' or higher, but lower than 'warn'.
>>>> # Exclude authpriv, cron, mail, and news.  These are logged elsewhere.
>>>> *.info;*.!warn;\
>>>>         authpriv.none;cron.none;mail.none;news.none     /var/log/messages
>>>>
>>>> # Log anything 'warn' or higher.
>>>> # Exclude authpriv, cron, mail, and news.  These are logged elsewhere.
>>>> *.warn;\
>>>>         authpriv.none;cron.none;mail.none;news.none     /var/log/syslog
>>>>
>>>> # Debugging information is logged here.
>>>> *.=debug                                                /var/log/debug
>>>>
>>>> # Private authentication message logging:
>>>> authpriv.*                                              /var/log/secure
>>>>
>>>> # Cron related logs:
>>>> cron.*                                                  /var/log/cron
>>>>
>>>> # Mail related logs:
>>>> mail.*                                                  /var/log/maillog
>>>>
>>>> # Emergency level messages go to all users:
>>>> *.emerg                                                 *
>>>>
>>>>
>>>> ## Uncomment these lines to use RELP instead for PCI compliance (stunnel
>>>> required)
>>>> #$ModLoad omrelp
>>>> #*.*;mail.none :omrelp:localhost:2515;RSYSLOG_ForwardFormat
>>>>
>>>> # certificate files - just CA for a client
>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
>>>>
>>>> # set up the action
>>>> $DefaultNetstreamDriver gtls # use gtls netstream driver
>>>> $ActionSendStreamDriverMode 1 # require TLS for the connection
>>>> $ActionSendStreamDriverAuthMode anon # server is NOT authenticated
>>>> *.* @@(o)ldap.nmsrv.com:10514 # send (all) messages
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.comoften
>>>>         
>>> --
>>> Patrick Shen
>>> Operations Engineer
>>>
>>> net mobile AG Shanghai office
>>>
>>> T: +86  21 61350900 - 222
>>> F: +86  21 61350906
>>> M: +86  13122245730
>>>
>>> FE55 6A7F 3192 F359 C1DF  12F6 9078 58B4 57AF 0BE3
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>       
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>     
>
>
>   


From rgerhards at hq.adiscon.com  Sun Jun  7 20:46:54 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 7 Jun 2009 20:46:54 +0200
Subject: [rsyslog] Possible bug? 100% cpu on a server thread and
	the	client stops sending data
References: <4A15999D.3000909@gossamer-threads.com><4A1A37DE.8050004@net-m.de>	<9B6E2A8877C38245BFB15CC491A11DA702B0FB@GRFEXC.intern.adiscon.com><4A1A57A0.1060903@net-m.de>
	<4A29978C.9080505@gossamer-threads.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8BA@GRFEXC.intern.adiscon.com>

Please give v4 a try, the bug is potentially fixed there. If it actually is,
this gives me a clue of where to look at.

Rainer 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Nathan March
> Sent: Saturday, June 06, 2009 12:09 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server 
> thread and the client stops sending data
> 
> Any hope of this bug getting some attention soon? =)
> 
> Thanks,
> Nathan
> 
> 
> Patrick Shen wrote:
> > Hi Rainer,
> >
> > Thanks for the quick reply.
> >
> > My environment is like below:
> >
> > CLIENT ---
> >          |
> >          | (TCP)
> >          --------> SERVER ----
> >                              |
> >                              | (RELP Relay)
> >                              --------------> SERVER Standby
> >
> > I've tested, even if the standby server is under high load, 
> one of our
> > clients won't accept any new network connections.
> >
> > Just for your information.
> >
> > Best regards,
> > Patrick
> >
> > Rainer Gerhards wrote:
> >   
> >> Hi all,
> >>
> >> thanks for the detailed info. Will look at it asap, but I 
> am currently tied
> >> up with some other work...
> >>
> >> Please let me know anything else that may be relevant.
> >>
> >> Thanks,
> >> Rainer
> >>
> >>     
> >>> -----Original Message-----
> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>> bounces at lists.adiscon.com] On Behalf Of Patrick Shen
> >>> Sent: Monday, May 25, 2009 8:17 AM
> >>> To: rsyslog-users
> >>> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server 
> thread and the
> >>> client stops sending data
> >>>
> >>> Hi Nathan,
> >>>
> >>> In my company, we have the similar symptom. We have 50+ 
> servers which
> >>> send logs to the central logging servers. Both clients 
> and server use
> >>> rsyslog v3.20.5 right now. In the past when we use 
> v3.20.0, the symptom
> >>> is more common when server is under high load average.
> >>>
> >>> It's also weird for us, not all of clients were lost of 
> connections. We
> >>> have 2-3 clients which is running the same application 
> will looks like
> >>> hanging up when server is high-load. New ssh connection 
> is very slow at
> >>> that time. But the connected ssh session works normal (We 
> have some
> >>> prepared screen session on other clients which connect to 
> them via ssh).
> >>>
> >>> When the clients are lost of connections. So far our 
> solution is to
> >>> restart rsyslog both on affected clients and server. Then 
> the clients
> >>> get back and look normal.
> >>>
> >>> We've suffered it for a while, but I didn't take chance 
> to analyze it
> >>> like you.
> >>>
> >>> Best regards,
> >>> Patrick
> >>>
> >>> Nathan March wrote:
> >>>       
> >>>> Having a weird issue here..... Testing out a new rsyslog 
> deployment and
> >>>> I've got around 30 servers logging to one machine. On 
> one of the clients
> >>>> we had an issue where ssh was mysteriously very slow to login and
> >>>> tracked it down to rsyslog causing issues. 
> Simultaneously, one of the
> >>>> threads on the logging server spiked up to 100%. New log 
> data from the
> >>>> client would stop showing up on the server at this poitn.
> >>>>
> >>>> On the client after attempting to gracefully stop rsyslog:
> >>>>
> >>>> vanguard etc # ps aux | grep -i rsyslog
> >>>> root      3764  0.0  0.1  5092 2564 ?        S    15:06   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f 
> /etc/rsyslog.conf
> >>>> root      3766  0.0  0.1  5092 2564 ?        S    15:06   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f 
> /etc/rsyslog.conf
> >>>> root      6203  0.0  0.0  1512  524 pts/3    S    15:09  
>  0:00 grep -i
> >>>> rsyslog
> >>>> vanguard etc # strace -p 3764
> >>>> Process 3764 attached - interrupt to quit
> >>>> send(11, 
> "\25\3\2\1\0uE\254\304\n\311\377\204}p\237O\211\322\211"...,
> >>>> 261, 0 <unfinished ...>
> >>>> Process 3764 detached
> >>>>
> >>>> FD 11 being the socket to the logging server, it never 
> does anything
> >>>> asides from that send.
> >>>>
> >>>> On the server:
> >>>>
> >>>> ldap ~ # ps aux | grep rsyslog
> >>>> root      7370  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7379  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7380  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7381  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7382  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7383  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7384  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7385 28.3  0.0  83936  3380 ?        R    14:54   5:07
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> Stracing the 7385 pid just shows this repeating over and 
> over (FD 4
> >>>> being the network socket again):
> >>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 
> EAGAIN (Resource
> >>>> temporarily unavailable)
> >>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 
> EAGAIN (Resource
> >>>> temporarily unavailable)
> >>>>
> >>>> It happened 3 times within a relatively short period of 
> time, sometimes
> >>>> within minutes of me restarting rsyslog on both 
> machines. I enabled
> >>>> debugging and it took around 8 hours to pop up again.
> >>>>
> >>>> With debugging, I pulled this from the rsyslog server:
> >>>>
> >>>> 6640.408108000:imtcp.c: New connect on NSD 0x59ddc0.
> >>>> 6640.408129000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: 
> pszRcvBuf (nil),
> >>>> lenRcvBuf 0
> >>>> 6640.408144000:imtcp.c: hasRcvInBuffer on nsd 0x59dcd0: pszRcvBuf
> >>>> 0x5c11c0, lenRcvBuf -1
> >>>> 6640.408158000:imtcp.c: hasRcvInBuffer on nsd 0x59dc10: pszRcvBuf
> >>>> 0x5cb510, lenRcvBuf -1
> >>>> 6640.408171000:imtcp.c: hasRcvInBuffer on nsd 0x5c3d70: pszRcvBuf
> >>>> 0x5d45c0, lenRcvBuf -1
> >>>> 6640.408185000:imtcp.c: hasRcvInBuffer on nsd 0x5cf460: pszRcvBuf
> >>>> 0x5dd010, lenRcvBuf -1
> >>>> 6640.408199000:imtcp.c: hasRcvInBuffer on nsd 0x5d8600: pszRcvBuf
> >>>> 0x5e58b0, lenRcvBuf -1
> >>>> 6640.408213000:imtcp.c: hasRcvInBuffer on nsd 0x59d690: pszRcvBuf
> >>>> 0x5ee150, lenRcvBuf -1
> >>>> 6640.408227000:imtcp.c: hasRcvInBuffer on nsd 0x59d2f0: pszRcvBuf
> >>>> 0x5f7320, lenRcvBuf -1
> >>>> 6640.408241000:imtcp.c: hasRcvInBuffer on nsd 0x5e8840: pszRcvBuf
> >>>> 0x5fe0a0, lenRcvBuf -1
> >>>> 6640.408255000:imtcp.c: hasRcvInBuffer on nsd 0x6027b0: pszRcvBuf
> >>>> 0x608350, lenRcvBuf -1
> >>>> 6640.408269000:imtcp.c: hasRcvInBuffer on nsd 0x5c3660: pszRcvBuf
> >>>> 0x612110, lenRcvBuf -1
> >>>> 6640.408283000:imtcp.c: hasRcvInBuffer on nsd 0x602690: pszRcvBuf
> >>>> 0x615040, lenRcvBuf -1
> >>>> 6640.408296000:imtcp.c: hasRcvInBuffer on nsd 0x5c3b60: pszRcvBuf
> >>>> 0x6218f0, lenRcvBuf -1
> >>>> 6640.408310000:imtcp.c: hasRcvInBuffer on nsd 0x60c150: pszRcvBuf
> >>>> 0x62a190, lenRcvBuf -1
> >>>> 6640.408324000:imtcp.c: hasRcvInBuffer on nsd 0x625420: pszRcvBuf
> >>>> 0x634fe0, lenRcvBuf -1
> >>>> 6640.408338000:imtcp.c: hasRcvInBuffer on nsd 0x6117b0: pszRcvBuf
> >>>> 0x63b000, lenRcvBuf -1
> >>>> 6640.408352000:imtcp.c: hasRcvInBuffer on nsd 0x603990: pszRcvBuf
> >>>> 0x643c70, lenRcvBuf -1
> >>>> 6640.408365000:imtcp.c: hasRcvInBuffer on nsd 0x633c00: pszRcvBuf
> >>>> 0x64dbb0, lenRcvBuf -1
> >>>> 6640.408379000:imtcp.c: hasRcvInBuffer on nsd 0x62cfa0: pszRcvBuf
> >>>> 0x650960, lenRcvBuf -1
> >>>> 6640.408393000:imtcp.c: hasRcvInBuffer on nsd 0x633d10: pszRcvBuf
> >>>> 0x65dbc0, lenRcvBuf -1
> >>>> 6640.408407000:imtcp.c: hasRcvInBuffer on nsd 0x5f2860: pszRcvBuf
> >>>> 0x666020, lenRcvBuf -1
> >>>> 6640.408421000:imtcp.c: hasRcvInBuffer on nsd 0x64d250: pszRcvBuf
> >>>> 0x66e480, lenRcvBuf -1
> >>>> 6640.408435000:imtcp.c: hasRcvInBuffer on nsd 0x659b30: pszRcvBuf
> >>>> 0x676d20, lenRcvBuf 78
> >>>> 6640.408448000:imtcp.c: hasRcvInBuffer on nsd 0x669fb0: pszRcvBuf
> >>>> 0x67fdd0, lenRcvBuf -1
> >>>> 6640.408462000:imtcp.c: hasRcvInBuffer on nsd 0x59a630: pszRcvBuf
> >>>> 0x687f10, lenRcvBuf -1
> >>>> 6640.408476000:imtcp.c: hasRcvInBuffer on nsd 0x6614c0: pszRcvBuf
> >>>> 0x6907b0, lenRcvBuf -1
> >>>> 6640.408490000:imtcp.c: hasRcvInBuffer on nsd 0x60c2d0: pszRcvBuf
> >>>> 0x699050, lenRcvBuf -1
> >>>> 6640.408504000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: 
> pszRcvBuf (nil),
> >>>> lenRcvBuf 0
> >>>>
> >>>> This repeats enough to generate around 29 million lines 
> in the debug log
> >>>> file.
> >>>>
> >>>> In the client log, there's plenty of these which seems 
> somewhat normal:
> >>>>
> >>>> 6509.743164000:imuxsock.c: --------imuxsock calling 
> select, active file
> >>>> descriptors (max 11): 11
> >>>> 6509.743231000:imuxsock.c: Message from UNIX socket: #11
> >>>> 6509.743262000:imuxsock.c: logmsg: flags 4, from 
> 'vanguard', msg May 20
> >>>> 23:15:09 bin/qmail-local[23198]: using .qmail .qmail-cfsc-forum
> >>>> 6509.743281000:imuxsock.c: Message has legacy syslog format.
> >>>> 6509.743302000:imuxsock.c: main queue: entry added, size 
> now 579 entries
> >>>> 6509.743325000:imuxsock.c: wtpAdviseMaxWorkers signals busy
> >>>> 6509.743343000:imuxsock.c: main queue: EnqueueMsg 
> advised worker start
> >>>>
> >>>> However comparing it to an earlier point in the log  
> there's none of the
> >>>> associated tcp queue entries like this:
> >>>>
> >>>> 3587.234178000:main queue:Reg/w0: TCP sent 107 bytes, 
> requested 107
> >>>> 3587.234204000:main queue:Reg/w0: main queue: entering 
> rate limiter
> >>>> 3587.234229000:main queue:Reg/w0: main queue: entry 
> deleted, state 0,
> >>>> size now 0 entries
> >>>> 3587.234255000:main queue:Reg/w0: Called action, logging 
> to builtin-file
> >>>> 3587.234285000:main queue:Reg/w0:  (/var/log/maillog)
> >>>> 3587.234314000:main queue:Reg/w0: Called action, logging 
> to builtin-fwd
> >>>> 3587.234340000:main queue:Reg/w0:  ldap.nmsrv.com
> >>>> 3587.234359000:main queue:Reg/w0:  ldap.nmsrv.com:10514/tcp
> >>>> 3587.234425000:main queue:Reg/w0: TCP sent 78 bytes, requested 78
> >>>> 3587.234446000:main queue:Reg/w0: main queue: entering 
> rate limiter
> >>>> 3587.234471000:main queue:Reg/w0: main queue:Reg/w0: worker IDLE,
> >>>> waiting for work.
> >>>>
> >>>> Anyone able to shed some light on this situation? I've 
> got the full
> >>>> debug logs if it's useful to anyone to take a closer look.
> >>>>
> >>>> Thanks!
> >>>>
> >>>> - Nathan
> >>>>
> >>>>
> >>>>
> >>>>         
> >> 
> --------------------------------------------------------------
> --------------
> >>     
> >>> --------
> >>>       
> >>>> Server config file
> >>>>
> >>>>         
> >> 
> --------------------------------------------------------------
> --------------
> >>     
> >>> --------
> >>>       
> >>>> $ModLoad immark.so # provides --MARK-- message capability
> >>>> $ModLoad imuxsock.so # provides support for local system 
> logging (e.g.
> >>>> via logger command)
> >>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> >>>>
> >>>> $umask 0137
> >>>> $DirCreateMode 0640
> >>>> $FileCreateMode 0640
> >>>> $FileOwner root
> >>>> $FileGroup admin
> >>>>
> >>>> $template DynFile,"/var/log/rsyslog/%HOSTNAME%/messages"
> >>>> $template DynFileCron,"/var/log/rsyslog/%HOSTNAME%/cron"
> >>>> $template DynFileSecure,"/var/log/rsyslog/%HOSTNAME%/secure"
> >>>> $template DynFileMail,"/var/log/rsyslog/%HOSTNAME%/maillog"
> >>>> $template DynFileSpool,"/var/log/rsyslog/%HOSTNAME%/spooler"
> >>>> $template DynFileBoot,"/var/log/rsyslog/%HOSTNAME%/boot"
> >>>> $template DynFileSyslog,"/var/log/rsyslog/%HOSTNAME%/syslog"
> >>>>
> >>>> :msg, contains, "no keys found for"            ~
> >>>> :msg, contains, "session opened for user"        ~
> >>>>
> >>>> # Log all kernel messages to the console.
> >>>> # Logging much else clutters up the screen.
> >>>> kern.*                                                 
> /dev/console
> >>>>
> >>>> # Log anything (except mail) of level info or higher.
> >>>> # Don't log private authentication messages!
> >>>> *.info;mail.none;cron.none                     ?DynFile
> >>>>
> >>>> *.warn;\
> >>>>         authpriv.none;cron.none;mail.none;news.none     
> ?DynFileSyslog
> >>>>
> >>>> # Log all the mail messages in one place.
> >>>> mail.*                                                  
> ?DynFileMail
> >>>>
> >>>> # Log cron stuff
> >>>> cron.*                                                  
> ?DynFileCron
> >>>>
> >>>> # Everybody gets emergency messages
> >>>> *.emerg                                                 *
> >>>>
> >>>> # Save news errors of level crit and higher in a special file
> >>>> uucp,news.crit                                          
> ?DynFileSpool
> >>>>
> >>>> # Save boot messages also to boot.log
> >>>> local7.*                                                
> ?DynFileBoot
> >>>>
> >>>> # Relp config for PCI
> >>>> $ModLoad imrelp
> >>>> $InputRELPServerRun 2515
> >>>>
> >>>> # make gtls driver the default
> >>>> $DefaultNetstreamDriver gtls
> >>>>
> >>>> # certificate files
> >>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
> >>>> $DefaultNetstreamDriverCertFile /etc/ssl/servercrt.pem
> >>>> $DefaultNetstreamDriverKeyFile /etc/ssl/serverkey.pem
> >>>>
> >>>> $ModLoad imtcp.so
> >>>>
> >>>> $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
> >>>> $InputTCPServerStreamDriverAuthMode anon # client is NOT 
> authenticated
> >>>> $InputTCPServerRun 10514 # start up listener at port 10514
> >>>> $InputTCPMaxSessions 500
> >>>>
> >>>>
> >>>>         
> >> 
> --------------------------------------------------------------
> --------------
> >>     
> >>> --------
> >>>       
> >>>> Client config file
> >>>>
> >>>>         
> >> 
> --------------------------------------------------------------
> --------------
> >>     
> >>> --------
> >>>       
> >>>> $ModLoad imuxsock.so # provides support for local system 
> logging (e.g.
> >>>> via logger command)
> >>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> >>>>
> >>>> $umask 0137
> >>>> $DirCreateMode 0640
> >>>> $FileCreateMode 0640
> >>>> $FileOwner root
> >>>> $FileGroup admin
> >>>>
> >>>>
> >>>> # Log anything 'info' or higher, but lower than 'warn'.
> >>>> # Exclude authpriv, cron, mail, and news.  These are 
> logged elsewhere.
> >>>> *.info;*.!warn;\
> >>>>         authpriv.none;cron.none;mail.none;news.none     
> /var/log/messages
> >>>>
> >>>> # Log anything 'warn' or higher.
> >>>> # Exclude authpriv, cron, mail, and news.  These are 
> logged elsewhere.
> >>>> *.warn;\
> >>>>         authpriv.none;cron.none;mail.none;news.none     
> /var/log/syslog
> >>>>
> >>>> # Debugging information is logged here.
> >>>> *.=debug                                                
> /var/log/debug
> >>>>
> >>>> # Private authentication message logging:
> >>>> authpriv.*                                              
> /var/log/secure
> >>>>
> >>>> # Cron related logs:
> >>>> cron.*                                                  
> /var/log/cron
> >>>>
> >>>> # Mail related logs:
> >>>> mail.*                                                  
> /var/log/maillog
> >>>>
> >>>> # Emergency level messages go to all users:
> >>>> *.emerg                                                 *
> >>>>
> >>>>
> >>>> ## Uncomment these lines to use RELP instead for PCI 
> compliance (stunnel
> >>>> required)
> >>>> #$ModLoad omrelp
> >>>> #*.*;mail.none :omrelp:localhost:2515;RSYSLOG_ForwardFormat
> >>>>
> >>>> # certificate files - just CA for a client
> >>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
> >>>>
> >>>> # set up the action
> >>>> $DefaultNetstreamDriver gtls # use gtls netstream driver
> >>>> $ActionSendStreamDriverMode 1 # require TLS for the connection
> >>>> $ActionSendStreamDriverAuthMode anon # server is NOT 
> authenticated
> >>>> *.* @@(o)ldap.nmsrv.com:10514 # send (all) messages
> >>>>
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.comoften
> >>>>         
> >>> --
> >>> Patrick Shen
> >>> Operations Engineer
> >>>
> >>> net mobile AG Shanghai office
> >>>
> >>> T: +86  21 61350900 - 222
> >>> F: +86  21 61350906
> >>> M: +86  13122245730
> >>>
> >>> FE55 6A7F 3192 F359 C1DF  12F6 9078 58B4 57AF 0BE3
> >>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>       
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>     
> >
> >
> >   
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 


From rgerhards at hq.adiscon.com  Mon Jun  8 17:33:00 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 8 Jun 2009 17:33:00 +0200
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>

Hi all,

I am working on the ultra-reliable file writing capability. I have revisited
the doc and am considering fsync() vs. fdatasync(). From the man pages, it
looks like fdatasync() is sufficient if I just need to ensure that any data
written can also be read. The advantage of fdatasync() is that it is somewhat
faster as not all meta data is necessarily written.

I would appreciate if someone could comment on this issue.

Thanks,
Rainer


From david at lang.hm  Mon Jun  8 18:09:56 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 8 Jun 2009 09:09:56 -0700 (PDT)
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906080908030.32468@asgard>

On Mon, 8 Jun 2009, Rainer Gerhards wrote:

> Hi all,
>
> I am working on the ultra-reliable file writing capability. I have revisited
> the doc and am considering fsync() vs. fdatasync(). From the man pages, it
> looks like fdatasync() is sufficient if I just need to ensure that any data
> written can also be read. The advantage of fdatasync() is that it is somewhat
> faster as not all meta data is necessarily written.
>
> I would appreciate if someone could comment on this issue.

my understanding is that if you are growing the file you need to do a 
fsync of the directory, but if you are re-writing part of an already 
allocated file fdatasync is the better thing to use.

David Lang


From rgerhards at hq.adiscon.com  Mon Jun  8 18:18:12 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 8 Jun 2009 18:18:12 +0200
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906080908030.32468@asgard>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8C6@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Monday, June 08, 2009 6:10 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
> 
> On Mon, 8 Jun 2009, Rainer Gerhards wrote:
> 
> > Hi all,
> >
> > I am working on the ultra-reliable file writing capability. I have
revisited
> > the doc and am considering fsync() vs. fdatasync(). From the man pages,
it
> > looks like fdatasync() is sufficient if I just need to ensure that any
data
> > written can also be read. The advantage of fdatasync() is that it is
> somewhat
> > faster as not all meta data is necessarily written.
> >
> > I would appreciate if someone could comment on this issue.
> 
> my understanding is that if you are growing the file you need to do a
> fsync of the directory, but if you are re-writing part of an already
> allocated file fdatasync is the better thing to use.

The man page[1] is somewhat vague. It tells:

"Calling fsync() does not necessarily ensure that the entry in the directory
containing the file has also reached disk. For that an explicit fsync() on a
file descriptor for the directory is also needed."

But also

"fdatasync() is similar to fsync(), but does not flush modified metadata
unless that metadata is needed in order to allow a subsequent data retrieval
to be correctly handled. For example, changes to st_atime or st_mtime
(respectively, time of last access and time of last modification; see
stat(2)) do not not require flushing because they are not necessary for a
subsequent data read to be handled correctly. On the other hand, a change to
the file size (st_size, as made by say ftruncate(2)), would require a
metadata flush.
"

So my current understanding is that fdatasync() is always safe, but there
still exist situations where a sync on the directory is needed. What I can
think of is a newly created file.

I have to admit that I am a bit puzzled of what it takes to do the "explicit
fsync on the directory". Most importantly, I need to have the file handle
open for writing, else the fsync will potentially fail. I don't think this is
such an excellent idea on directories.. I am right now working on this, and
my search has not brought up many good hints. So what I currently plan to do
is an open(dirname, O_RDWR) on the directory and then do an fsync on it
later. 

And I guess even after the fsync() on the directeroy, there potentially is an
issue with newly created files inside newly create directories, but I don't
think I will address this immediately. But I may be useful/necessary to add a
tree walker who actually walks the full path name and calls the fsyncs on all
of them...

Comments are highly appreciated.

Rainer

[1] http://linux.die.net/man/2/fsync


From rgerhards at hq.adiscon.com  Mon Jun  8 18:31:28 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 8 Jun 2009 18:31:28 +0200
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906080908030.32468@asgard>
	<9B6E2A8877C38245BFB15CC491A11DA706F8C6@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8C7@GRFEXC.intern.adiscon.com>

> my search has not brought up many good hints. So what I currently plan to
do
> is an open(dirname, O_RDWR) on the directory and then do an fsync on it
> later.

... which does not work, because I receive EISDIR ;)

No I do an O_RDONLY, which works and then even fsync() on it works as well.
So this is what I will now be doing, but with a somewhat bad feeling...

Rainer


From epiphani at gmail.com  Mon Jun  8 18:12:13 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 8 Jun 2009 12:12:13 -0400
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
In-Reply-To: <alpine.DEB.1.10.0906080908030.32468@asgard>
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906080908030.32468@asgard>
Message-ID: <e7ca40f70906080912i189153a6m1fe4633008048c12@mail.gmail.com>

On Mon, Jun 8, 2009 at 12:09 PM, <david at lang.hm> wrote:
>> I am working on the ultra-reliable file writing capability. I have revisited
>> the doc and am considering fsync() vs. fdatasync(). From the man pages, it
>> looks like fdatasync() is sufficient if I just need to ensure that any data
>> written can also be read. The advantage of fdatasync() is that it is somewhat
>> faster as not all meta data is necessarily written.
>>
>> I would appreciate if someone could comment on this issue.
>
> my understanding is that if you are growing the file you need to do a
> fsync of the directory, but if you are re-writing part of an already
> allocated file fdatasync is the better thing to use.

While following this path, it would also be a good idea to check NFS
compatibility, as I would bet that is a common configuration.  NFS can
have its own unique quirks....

-Aaron


From rgerhards at hq.adiscon.com  Mon Jun  8 18:53:18 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 8 Jun 2009 18:53:18 +0200
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906080908030.32468@asgard>
	<e7ca40f70906080912i189153a6m1fe4633008048c12@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8C8@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> Sent: Monday, June 08, 2009 6:12 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
> 
> On Mon, Jun 8, 2009 at 12:09 PM, <david at lang.hm> wrote:
> >> I am working on the ultra-reliable file writing capability. I have
> revisited
> >> the doc and am considering fsync() vs. fdatasync(). From the man pages,
it
> >> looks like fdatasync() is sufficient if I just need to ensure that any
data
> >> written can also be read. The advantage of fdatasync() is that it is
> somewhat
> >> faster as not all meta data is necessarily written.
> >>
> >> I would appreciate if someone could comment on this issue.
> >
> > my understanding is that if you are growing the file you need to do a
> > fsync of the directory, but if you are re-writing part of an already
> > allocated file fdatasync is the better thing to use.
> 
> While following this path, it would also be a good idea to check NFS
> compatibility, as I would bet that is a common configuration.  NFS can
> have its own unique quirks....

>From what I have seen during my research, I would expect that NFS does not
always correctly fsync(). It seems to depend a lot on the actual
implementation, maybe version of it. What I currently do is ignore an error
during fsync() (fdatasync to be more precise). The reason is that there is
not much I could do if I see the error. Aborting the run would probably be
much more fatal than ignoring the error. The same goes if the fsync() for the
directory fails or if I can not open the directory.

I would assume that if a truly reliable system is requested, local disks must
be used.

Comments welcome,
Rainer


From david at lang.hm  Mon Jun  8 19:07:35 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 8 Jun 2009 10:07:35 -0700 (PDT)
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F8C6@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906080908030.32468@asgard>
	<9B6E2A8877C38245BFB15CC491A11DA706F8C6@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906080956030.9159@asgard>

On Mon, 8 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Mon, 8 Jun 2009, Rainer Gerhards wrote:
>>
>>> Hi all,
>>>
>>> I am working on the ultra-reliable file writing capability. I have
> revisited
>>> the doc and am considering fsync() vs. fdatasync(). From the man pages,
> it
>>> looks like fdatasync() is sufficient if I just need to ensure that any
> data
>>> written can also be read. The advantage of fdatasync() is that it is
>> somewhat
>>> faster as not all meta data is necessarily written.
>>>
>>> I would appreciate if someone could comment on this issue.
>>
>> my understanding is that if you are growing the file you need to do a
>> fsync of the directory, but if you are re-writing part of an already
>> allocated file fdatasync is the better thing to use.
>
> The man page[1] is somewhat vague. It tells:
>
> "Calling fsync() does not necessarily ensure that the entry in the directory
> containing the file has also reached disk. For that an explicit fsync() on a
> file descriptor for the directory is also needed."
>
> But also
>
> "fdatasync() is similar to fsync(), but does not flush modified metadata
> unless that metadata is needed in order to allow a subsequent data retrieval
> to be correctly handled. For example, changes to st_atime or st_mtime
> (respectively, time of last access and time of last modification; see
> stat(2)) do not not require flushing because they are not necessary for a
> subsequent data read to be handled correctly. On the other hand, a change to
> the file size (st_size, as made by say ftruncate(2)), would require a
> metadata flush.
> "
>
> So my current understanding is that fdatasync() is always safe, but there
> still exist situations where a sync on the directory is needed. What I can
> think of is a newly created file.

any time the file size changes (so that the metadata about the disk 
allocation is safe)

> I have to admit that I am a bit puzzled of what it takes to do the "explicit
> fsync on the directory". Most importantly, I need to have the file handle
> open for writing, else the fsync will potentially fail. I don't think this is
> such an excellent idea on directories.. I am right now working on this, and
> my search has not brought up many good hints. So what I currently plan to do
> is an open(dirname, O_RDWR) on the directory and then do an fsync on it
> later.

I believe that that's correct.

> And I guess even after the fsync() on the directeroy, there potentially is an
> issue with newly created files inside newly create directories, but I don't
> think I will address this immediately. But I may be useful/necessary to add a
> tree walker who actually walks the full path name and calls the fsyncs on all
> of them...

in an ideal world yes, but in practice you would only need to do this when 
you create a new file and that new directory entry caused the filesystem 
to allocate more space on disk to hole the directory entry, so it's almost 
never an issue.

another advantage of fdatasync is that it tells the OS exactly what 
portion of the file it needs to work on, with a fsync it needs to search 
all buffers related to that file to figure out what needs to be written 
(not a big hit, but a hit)

David Lang


From nathan at gossamer-threads.com  Tue Jun  9 18:57:49 2009
From: nathan at gossamer-threads.com (Nathan March)
Date: Tue, 09 Jun 2009 09:57:49 -0700
Subject: [rsyslog] Possible bug? 100% cpu on a server thread
 and	the	client stops sending data
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F8BA@GRFEXC.intern.adiscon.com>
References: <4A15999D.3000909@gossamer-threads.com><4A1A37DE.8050004@net-m.de>	<9B6E2A8877C38245BFB15CC491A11DA702B0FB@GRFEXC.intern.adiscon.com><4A1A57A0.1060903@net-m.de>	<4A29978C.9080505@gossamer-threads.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F8BA@GRFEXC.intern.adiscon.com>
Message-ID: <4A2E948D.4060809@gossamer-threads.com>

Confirmed as still happening on 4.1.7

- Nathan

Rainer Gerhards wrote:
> Please give v4 a try, the bug is potentially fixed there. If it actually is,
> this gives me a clue of where to look at.
>
> Rainer 
>
>   
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com 
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Nathan March
>> Sent: Saturday, June 06, 2009 12:09 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server 
>> thread and the client stops sending data
>>
>> Any hope of this bug getting some attention soon? =)
>>
>> Thanks,
>> Nathan
>>
>>
>> Patrick Shen wrote:
>>     
>>> Hi Rainer,
>>>
>>> Thanks for the quick reply.
>>>
>>> My environment is like below:
>>>
>>> CLIENT ---
>>>          |
>>>          | (TCP)
>>>          --------> SERVER ----
>>>                              |
>>>                              | (RELP Relay)
>>>                              --------------> SERVER Standby
>>>
>>> I've tested, even if the standby server is under high load, 
>>>       
>> one of our
>>     
>>> clients won't accept any new network connections.
>>>
>>> Just for your information.
>>>
>>> Best regards,
>>> Patrick
>>>
>>> Rainer Gerhards wrote:
>>>   
>>>       
>>>> Hi all,
>>>>
>>>> thanks for the detailed info. Will look at it asap, but I 
>>>>         
>> am currently tied
>>     
>>>> up with some other work...
>>>>
>>>> Please let me know anything else that may be relevant.
>>>>
>>>> Thanks,
>>>> Rainer
>>>>
>>>>     
>>>>         
>>>>> -----Original Message-----
>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>> bounces at lists.adiscon.com] On Behalf Of Patrick Shen
>>>>> Sent: Monday, May 25, 2009 8:17 AM
>>>>> To: rsyslog-users
>>>>> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server 
>>>>>           
>> thread and the
>>     
>>>>> client stops sending data
>>>>>
>>>>> Hi Nathan,
>>>>>
>>>>> In my company, we have the similar symptom. We have 50+ 
>>>>>           
>> servers which
>>     
>>>>> send logs to the central logging servers. Both clients 
>>>>>           
>> and server use
>>     
>>>>> rsyslog v3.20.5 right now. In the past when we use 
>>>>>           
>> v3.20.0, the symptom
>>     
>>>>> is more common when server is under high load average.
>>>>>
>>>>> It's also weird for us, not all of clients were lost of 
>>>>>           
>> connections. We
>>     
>>>>> have 2-3 clients which is running the same application 
>>>>>           
>> will looks like
>>     
>>>>> hanging up when server is high-load. New ssh connection 
>>>>>           
>> is very slow at
>>     
>>>>> that time. But the connected ssh session works normal (We 
>>>>>           
>> have some
>>     
>>>>> prepared screen session on other clients which connect to 
>>>>>           
>> them via ssh).
>>     
>>>>> When the clients are lost of connections. So far our 
>>>>>           
>> solution is to
>>     
>>>>> restart rsyslog both on affected clients and server. Then 
>>>>>           
>> the clients
>>     
>>>>> get back and look normal.
>>>>>
>>>>> We've suffered it for a while, but I didn't take chance 
>>>>>           
>> to analyze it
>>     
>>>>> like you.
>>>>>
>>>>> Best regards,
>>>>> Patrick
>>>>>
>>>>> Nathan March wrote:
>>>>>       
>>>>>           
>>>>>> Having a weird issue here..... Testing out a new rsyslog 
>>>>>>             
>> deployment and
>>     
>>>>>> I've got around 30 servers logging to one machine. On 
>>>>>>             
>> one of the clients
>>     
>>>>>> we had an issue where ssh was mysteriously very slow to login and
>>>>>> tracked it down to rsyslog causing issues. 
>>>>>>             
>> Simultaneously, one of the
>>     
>>>>>> threads on the logging server spiked up to 100%. New log 
>>>>>>             
>> data from the
>>     
>>>>>> client would stop showing up on the server at this poitn.
>>>>>>
>>>>>> On the client after attempting to gracefully stop rsyslog:
>>>>>>
>>>>>> vanguard etc # ps aux | grep -i rsyslog
>>>>>> root      3764  0.0  0.1  5092 2564 ?        S    15:06   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f 
>>>>>>             
>> /etc/rsyslog.conf
>>     
>>>>>> root      3766  0.0  0.1  5092 2564 ?        S    15:06   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f 
>>>>>>             
>> /etc/rsyslog.conf
>>     
>>>>>> root      6203  0.0  0.0  1512  524 pts/3    S    15:09  
>>>>>>             
>>  0:00 grep -i
>>     
>>>>>> rsyslog
>>>>>> vanguard etc # strace -p 3764
>>>>>> Process 3764 attached - interrupt to quit
>>>>>> send(11, 
>>>>>>             
>> "\25\3\2\1\0uE\254\304\n\311\377\204}p\237O\211\322\211"...,
>>     
>>>>>> 261, 0 <unfinished ...>
>>>>>> Process 3764 detached
>>>>>>
>>>>>> FD 11 being the socket to the logging server, it never 
>>>>>>             
>> does anything
>>     
>>>>>> asides from that send.
>>>>>>
>>>>>> On the server:
>>>>>>
>>>>>> ldap ~ # ps aux | grep rsyslog
>>>>>> root      7370  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7379  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7380  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7381  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7382  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7383  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7384  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7385 28.3  0.0  83936  3380 ?        R    14:54   5:07
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> Stracing the 7385 pid just shows this repeating over and 
>>>>>>             
>> over (FD 4
>>     
>>>>>> being the network socket again):
>>>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 
>>>>>>             
>> EAGAIN (Resource
>>     
>>>>>> temporarily unavailable)
>>>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 
>>>>>>             
>> EAGAIN (Resource
>>     
>>>>>> temporarily unavailable)
>>>>>>
>>>>>> It happened 3 times within a relatively short period of 
>>>>>>             
>> time, sometimes
>>     
>>>>>> within minutes of me restarting rsyslog on both 
>>>>>>             
>> machines. I enabled
>>     
>>>>>> debugging and it took around 8 hours to pop up again.
>>>>>>
>>>>>> With debugging, I pulled this from the rsyslog server:
>>>>>>
>>>>>> 6640.408108000:imtcp.c: New connect on NSD 0x59ddc0.
>>>>>> 6640.408129000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: 
>>>>>>             
>> pszRcvBuf (nil),
>>     
>>>>>> lenRcvBuf 0
>>>>>> 6640.408144000:imtcp.c: hasRcvInBuffer on nsd 0x59dcd0: pszRcvBuf
>>>>>> 0x5c11c0, lenRcvBuf -1
>>>>>> 6640.408158000:imtcp.c: hasRcvInBuffer on nsd 0x59dc10: pszRcvBuf
>>>>>> 0x5cb510, lenRcvBuf -1
>>>>>> 6640.408171000:imtcp.c: hasRcvInBuffer on nsd 0x5c3d70: pszRcvBuf
>>>>>> 0x5d45c0, lenRcvBuf -1
>>>>>> 6640.408185000:imtcp.c: hasRcvInBuffer on nsd 0x5cf460: pszRcvBuf
>>>>>> 0x5dd010, lenRcvBuf -1
>>>>>> 6640.408199000:imtcp.c: hasRcvInBuffer on nsd 0x5d8600: pszRcvBuf
>>>>>> 0x5e58b0, lenRcvBuf -1
>>>>>> 6640.408213000:imtcp.c: hasRcvInBuffer on nsd 0x59d690: pszRcvBuf
>>>>>> 0x5ee150, lenRcvBuf -1
>>>>>> 6640.408227000:imtcp.c: hasRcvInBuffer on nsd 0x59d2f0: pszRcvBuf
>>>>>> 0x5f7320, lenRcvBuf -1
>>>>>> 6640.408241000:imtcp.c: hasRcvInBuffer on nsd 0x5e8840: pszRcvBuf
>>>>>> 0x5fe0a0, lenRcvBuf -1
>>>>>> 6640.408255000:imtcp.c: hasRcvInBuffer on nsd 0x6027b0: pszRcvBuf
>>>>>> 0x608350, lenRcvBuf -1
>>>>>> 6640.408269000:imtcp.c: hasRcvInBuffer on nsd 0x5c3660: pszRcvBuf
>>>>>> 0x612110, lenRcvBuf -1
>>>>>> 6640.408283000:imtcp.c: hasRcvInBuffer on nsd 0x602690: pszRcvBuf
>>>>>> 0x615040, lenRcvBuf -1
>>>>>> 6640.408296000:imtcp.c: hasRcvInBuffer on nsd 0x5c3b60: pszRcvBuf
>>>>>> 0x6218f0, lenRcvBuf -1
>>>>>> 6640.408310000:imtcp.c: hasRcvInBuffer on nsd 0x60c150: pszRcvBuf
>>>>>> 0x62a190, lenRcvBuf -1
>>>>>> 6640.408324000:imtcp.c: hasRcvInBuffer on nsd 0x625420: pszRcvBuf
>>>>>> 0x634fe0, lenRcvBuf -1
>>>>>> 6640.408338000:imtcp.c: hasRcvInBuffer on nsd 0x6117b0: pszRcvBuf
>>>>>> 0x63b000, lenRcvBuf -1
>>>>>> 6640.408352000:imtcp.c: hasRcvInBuffer on nsd 0x603990: pszRcvBuf
>>>>>> 0x643c70, lenRcvBuf -1
>>>>>> 6640.408365000:imtcp.c: hasRcvInBuffer on nsd 0x633c00: pszRcvBuf
>>>>>> 0x64dbb0, lenRcvBuf -1
>>>>>> 6640.408379000:imtcp.c: hasRcvInBuffer on nsd 0x62cfa0: pszRcvBuf
>>>>>> 0x650960, lenRcvBuf -1
>>>>>> 6640.408393000:imtcp.c: hasRcvInBuffer on nsd 0x633d10: pszRcvBuf
>>>>>> 0x65dbc0, lenRcvBuf -1
>>>>>> 6640.408407000:imtcp.c: hasRcvInBuffer on nsd 0x5f2860: pszRcvBuf
>>>>>> 0x666020, lenRcvBuf -1
>>>>>> 6640.408421000:imtcp.c: hasRcvInBuffer on nsd 0x64d250: pszRcvBuf
>>>>>> 0x66e480, lenRcvBuf -1
>>>>>> 6640.408435000:imtcp.c: hasRcvInBuffer on nsd 0x659b30: pszRcvBuf
>>>>>> 0x676d20, lenRcvBuf 78
>>>>>> 6640.408448000:imtcp.c: hasRcvInBuffer on nsd 0x669fb0: pszRcvBuf
>>>>>> 0x67fdd0, lenRcvBuf -1
>>>>>> 6640.408462000:imtcp.c: hasRcvInBuffer on nsd 0x59a630: pszRcvBuf
>>>>>> 0x687f10, lenRcvBuf -1
>>>>>> 6640.408476000:imtcp.c: hasRcvInBuffer on nsd 0x6614c0: pszRcvBuf
>>>>>> 0x6907b0, lenRcvBuf -1
>>>>>> 6640.408490000:imtcp.c: hasRcvInBuffer on nsd 0x60c2d0: pszRcvBuf
>>>>>> 0x699050, lenRcvBuf -1
>>>>>> 6640.408504000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: 
>>>>>>             
>> pszRcvBuf (nil),
>>     
>>>>>> lenRcvBuf 0
>>>>>>
>>>>>> This repeats enough to generate around 29 million lines 
>>>>>>             
>> in the debug log
>>     
>>>>>> file.
>>>>>>
>>>>>> In the client log, there's plenty of these which seems 
>>>>>>             
>> somewhat normal:
>>     
>>>>>> 6509.743164000:imuxsock.c: --------imuxsock calling 
>>>>>>             
>> select, active file
>>     
>>>>>> descriptors (max 11): 11
>>>>>> 6509.743231000:imuxsock.c: Message from UNIX socket: #11
>>>>>> 6509.743262000:imuxsock.c: logmsg: flags 4, from 
>>>>>>             
>> 'vanguard', msg May 20
>>     
>>>>>> 23:15:09 bin/qmail-local[23198]: using .qmail .qmail-cfsc-forum
>>>>>> 6509.743281000:imuxsock.c: Message has legacy syslog format.
>>>>>> 6509.743302000:imuxsock.c: main queue: entry added, size 
>>>>>>             
>> now 579 entries
>>     
>>>>>> 6509.743325000:imuxsock.c: wtpAdviseMaxWorkers signals busy
>>>>>> 6509.743343000:imuxsock.c: main queue: EnqueueMsg 
>>>>>>             
>> advised worker start
>>     
>>>>>> However comparing it to an earlier point in the log  
>>>>>>             
>> there's none of the
>>     
>>>>>> associated tcp queue entries like this:
>>>>>>
>>>>>> 3587.234178000:main queue:Reg/w0: TCP sent 107 bytes, 
>>>>>>             
>> requested 107
>>     
>>>>>> 3587.234204000:main queue:Reg/w0: main queue: entering 
>>>>>>             
>> rate limiter
>>     
>>>>>> 3587.234229000:main queue:Reg/w0: main queue: entry 
>>>>>>             
>> deleted, state 0,
>>     
>>>>>> size now 0 entries
>>>>>> 3587.234255000:main queue:Reg/w0: Called action, logging 
>>>>>>             
>> to builtin-file
>>     
>>>>>> 3587.234285000:main queue:Reg/w0:  (/var/log/maillog)
>>>>>> 3587.234314000:main queue:Reg/w0: Called action, logging 
>>>>>>             
>> to builtin-fwd
>>     
>>>>>> 3587.234340000:main queue:Reg/w0:  ldap.nmsrv.com
>>>>>> 3587.234359000:main queue:Reg/w0:  ldap.nmsrv.com:10514/tcp
>>>>>> 3587.234425000:main queue:Reg/w0: TCP sent 78 bytes, requested 78
>>>>>> 3587.234446000:main queue:Reg/w0: main queue: entering 
>>>>>>             
>> rate limiter
>>     
>>>>>> 3587.234471000:main queue:Reg/w0: main queue:Reg/w0: worker IDLE,
>>>>>> waiting for work.
>>>>>>
>>>>>> Anyone able to shed some light on this situation? I've 
>>>>>>             
>> got the full
>>     
>>>>>> debug logs if it's useful to anyone to take a closer look.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> - Nathan
>>>>>>
>>>>>>
>>>>>>
>>>>>>         
>>>>>>             
>> --------------------------------------------------------------
>> --------------
>>     
>>>>     
>>>>         
>>>>> --------
>>>>>       
>>>>>           
>>>>>> Server config file
>>>>>>
>>>>>>         
>>>>>>             
>> --------------------------------------------------------------
>> --------------
>>     
>>>>     
>>>>         
>>>>> --------
>>>>>       
>>>>>           
>>>>>> $ModLoad immark.so # provides --MARK-- message capability
>>>>>> $ModLoad imuxsock.so # provides support for local system 
>>>>>>             
>> logging (e.g.
>>     
>>>>>> via logger command)
>>>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
>>>>>>
>>>>>> $umask 0137
>>>>>> $DirCreateMode 0640
>>>>>> $FileCreateMode 0640
>>>>>> $FileOwner root
>>>>>> $FileGroup admin
>>>>>>
>>>>>> $template DynFile,"/var/log/rsyslog/%HOSTNAME%/messages"
>>>>>> $template DynFileCron,"/var/log/rsyslog/%HOSTNAME%/cron"
>>>>>> $template DynFileSecure,"/var/log/rsyslog/%HOSTNAME%/secure"
>>>>>> $template DynFileMail,"/var/log/rsyslog/%HOSTNAME%/maillog"
>>>>>> $template DynFileSpool,"/var/log/rsyslog/%HOSTNAME%/spooler"
>>>>>> $template DynFileBoot,"/var/log/rsyslog/%HOSTNAME%/boot"
>>>>>> $template DynFileSyslog,"/var/log/rsyslog/%HOSTNAME%/syslog"
>>>>>>
>>>>>> :msg, contains, "no keys found for"            ~
>>>>>> :msg, contains, "session opened for user"        ~
>>>>>>
>>>>>> # Log all kernel messages to the console.
>>>>>> # Logging much else clutters up the screen.
>>>>>> kern.*                                                 
>>>>>>             
>> /dev/console
>>     
>>>>>> # Log anything (except mail) of level info or higher.
>>>>>> # Don't log private authentication messages!
>>>>>> *.info;mail.none;cron.none                     ?DynFile
>>>>>>
>>>>>> *.warn;\
>>>>>>         authpriv.none;cron.none;mail.none;news.none     
>>>>>>             
>> ?DynFileSyslog
>>     
>>>>>> # Log all the mail messages in one place.
>>>>>> mail.*                                                  
>>>>>>             
>> ?DynFileMail
>>     
>>>>>> # Log cron stuff
>>>>>> cron.*                                                  
>>>>>>             
>> ?DynFileCron
>>     
>>>>>> # Everybody gets emergency messages
>>>>>> *.emerg                                                 *
>>>>>>
>>>>>> # Save news errors of level crit and higher in a special file
>>>>>> uucp,news.crit                                          
>>>>>>             
>> ?DynFileSpool
>>     
>>>>>> # Save boot messages also to boot.log
>>>>>> local7.*                                                
>>>>>>             
>> ?DynFileBoot
>>     
>>>>>> # Relp config for PCI
>>>>>> $ModLoad imrelp
>>>>>> $InputRELPServerRun 2515
>>>>>>
>>>>>> # make gtls driver the default
>>>>>> $DefaultNetstreamDriver gtls
>>>>>>
>>>>>> # certificate files
>>>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
>>>>>> $DefaultNetstreamDriverCertFile /etc/ssl/servercrt.pem
>>>>>> $DefaultNetstreamDriverKeyFile /etc/ssl/serverkey.pem
>>>>>>
>>>>>> $ModLoad imtcp.so
>>>>>>
>>>>>> $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
>>>>>> $InputTCPServerStreamDriverAuthMode anon # client is NOT 
>>>>>>             
>> authenticated
>>     
>>>>>> $InputTCPServerRun 10514 # start up listener at port 10514
>>>>>> $InputTCPMaxSessions 500
>>>>>>
>>>>>>
>>>>>>         
>>>>>>             
>> --------------------------------------------------------------
>> --------------
>>     
>>>>     
>>>>         
>>>>> --------
>>>>>       
>>>>>           
>>>>>> Client config file
>>>>>>
>>>>>>         
>>>>>>             
>> --------------------------------------------------------------
>> --------------
>>     
>>>>     
>>>>         
>>>>> --------
>>>>>       
>>>>>           
>>>>>> $ModLoad imuxsock.so # provides support for local system 
>>>>>>             
>> logging (e.g.
>>     
>>>>>> via logger command)
>>>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
>>>>>>
>>>>>> $umask 0137
>>>>>> $DirCreateMode 0640
>>>>>> $FileCreateMode 0640
>>>>>> $FileOwner root
>>>>>> $FileGroup admin
>>>>>>
>>>>>>
>>>>>> # Log anything 'info' or higher, but lower than 'warn'.
>>>>>> # Exclude authpriv, cron, mail, and news.  These are 
>>>>>>             
>> logged elsewhere.
>>     
>>>>>> *.info;*.!warn;\
>>>>>>         authpriv.none;cron.none;mail.none;news.none     
>>>>>>             
>> /var/log/messages
>>     
>>>>>> # Log anything 'warn' or higher.
>>>>>> # Exclude authpriv, cron, mail, and news.  These are 
>>>>>>             
>> logged elsewhere.
>>     
>>>>>> *.warn;\
>>>>>>         authpriv.none;cron.none;mail.none;news.none     
>>>>>>             
>> /var/log/syslog
>>     
>>>>>> # Debugging information is logged here.
>>>>>> *.=debug                                                
>>>>>>             
>> /var/log/debug
>>     
>>>>>> # Private authentication message logging:
>>>>>> authpriv.*                                              
>>>>>>             
>> /var/log/secure
>>     
>>>>>> # Cron related logs:
>>>>>> cron.*                                                  
>>>>>>             
>> /var/log/cron
>>     
>>>>>> # Mail related logs:
>>>>>> mail.*                                                  
>>>>>>             
>> /var/log/maillog
>>     
>>>>>> # Emergency level messages go to all users:
>>>>>> *.emerg                                                 *
>>>>>>
>>>>>>
>>>>>> ## Uncomment these lines to use RELP instead for PCI 
>>>>>>             
>> compliance (stunnel
>>     
>>>>>> required)
>>>>>> #$ModLoad omrelp
>>>>>> #*.*;mail.none :omrelp:localhost:2515;RSYSLOG_ForwardFormat
>>>>>>
>>>>>> # certificate files - just CA for a client
>>>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
>>>>>>
>>>>>> # set up the action
>>>>>> $DefaultNetstreamDriver gtls # use gtls netstream driver
>>>>>> $ActionSendStreamDriverMode 1 # require TLS for the connection
>>>>>> $ActionSendStreamDriverAuthMode anon # server is NOT 
>>>>>>             
>> authenticated
>>     
>>>>>> *.* @@(o)ldap.nmsrv.com:10514 # send (all) messages
>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.comoften
>>>>>>         
>>>>>>             
>>>>> --
>>>>> Patrick Shen
>>>>> Operations Engineer
>>>>>
>>>>> net mobile AG Shanghai office
>>>>>
>>>>> T: +86  21 61350900 - 222
>>>>> F: +86  21 61350906
>>>>> M: +86  13122245730
>>>>>
>>>>> FE55 6A7F 3192 F359 C1DF  12F6 9078 58B4 57AF 0BE3
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>>       
>>>>>           
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>     
>>>>         
>>>   
>>>       
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
>>     
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>   


From rgerhards at hq.adiscon.com  Wed Jun 10 18:12:38 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 10 Jun 2009 18:12:38 +0200
Subject: [rsyslog] Possible bug? 100% cpu on a server thread
	and	the	client stops sending data
References: <4A15999D.3000909@gossamer-threads.com><4A1A37DE.8050004@net-m.de>	<9B6E2A8877C38245BFB15CC491A11DA702B0FB@GRFEXC.intern.adiscon.com><4A1A57A0.1060903@net-m.de>	<4A29978C.9080505@gossamer-threads.com><9B6E2A8877C38245BFB15CC491A11DA706F8BA@GRFEXC.intern.adiscon.com>
	<4A2E948D.4060809@gossamer-threads.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8FD@GRFEXC.intern.adiscon.com>

Args, sorry.. I hadn't thought about the beta. Could you retry with the
current devel version? This is the one that potentially has a fix.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Nathan March
> Sent: Tuesday, June 09, 2009 6:58 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server thread and the
> client stops sending data
> 
> Confirmed as still happening on 4.1.7
> 
> - Nathan
> 
> Rainer Gerhards wrote:
> > Please give v4 a try, the bug is potentially fixed there. If it actually
is,
> > this gives me a clue of where to look at.
> >
> > Rainer
> >
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Nathan March
> >> Sent: Saturday, June 06, 2009 12:09 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server
> >> thread and the client stops sending data
> >>
> >> Any hope of this bug getting some attention soon? =)
> >>
> >> Thanks,
> >> Nathan
> >>
> >>
> >> Patrick Shen wrote:
> >>
> >>> Hi Rainer,
> >>>
> >>> Thanks for the quick reply.
> >>>
> >>> My environment is like below:
> >>>
> >>> CLIENT ---
> >>>          |
> >>>          | (TCP)
> >>>          --------> SERVER ----
> >>>                              |
> >>>                              | (RELP Relay)
> >>>                              --------------> SERVER Standby
> >>>
> >>> I've tested, even if the standby server is under high load,
> >>>
> >> one of our
> >>
> >>> clients won't accept any new network connections.
> >>>
> >>> Just for your information.
> >>>
> >>> Best regards,
> >>> Patrick
> >>>
> >>> Rainer Gerhards wrote:
> >>>
> >>>
> >>>> Hi all,
> >>>>
> >>>> thanks for the detailed info. Will look at it asap, but I
> >>>>
> >> am currently tied
> >>
> >>>> up with some other work...
> >>>>
> >>>> Please let me know anything else that may be relevant.
> >>>>
> >>>> Thanks,
> >>>> Rainer
> >>>>
> >>>>
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>>> bounces at lists.adiscon.com] On Behalf Of Patrick Shen
> >>>>> Sent: Monday, May 25, 2009 8:17 AM
> >>>>> To: rsyslog-users
> >>>>> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server
> >>>>>
> >> thread and the
> >>
> >>>>> client stops sending data
> >>>>>
> >>>>> Hi Nathan,
> >>>>>
> >>>>> In my company, we have the similar symptom. We have 50+
> >>>>>
> >> servers which
> >>
> >>>>> send logs to the central logging servers. Both clients
> >>>>>
> >> and server use
> >>
> >>>>> rsyslog v3.20.5 right now. In the past when we use
> >>>>>
> >> v3.20.0, the symptom
> >>
> >>>>> is more common when server is under high load average.
> >>>>>
> >>>>> It's also weird for us, not all of clients were lost of
> >>>>>
> >> connections. We
> >>
> >>>>> have 2-3 clients which is running the same application
> >>>>>
> >> will looks like
> >>
> >>>>> hanging up when server is high-load. New ssh connection
> >>>>>
> >> is very slow at
> >>
> >>>>> that time. But the connected ssh session works normal (We
> >>>>>
> >> have some
> >>
> >>>>> prepared screen session on other clients which connect to
> >>>>>
> >> them via ssh).
> >>
> >>>>> When the clients are lost of connections. So far our
> >>>>>
> >> solution is to
> >>
> >>>>> restart rsyslog both on affected clients and server. Then
> >>>>>
> >> the clients
> >>
> >>>>> get back and look normal.
> >>>>>
> >>>>> We've suffered it for a while, but I didn't take chance
> >>>>>
> >> to analyze it
> >>
> >>>>> like you.
> >>>>>
> >>>>> Best regards,
> >>>>> Patrick
> >>>>>
> >>>>> Nathan March wrote:
> >>>>>
> >>>>>
> >>>>>> Having a weird issue here..... Testing out a new rsyslog
> >>>>>>
> >> deployment and
> >>
> >>>>>> I've got around 30 servers logging to one machine. On
> >>>>>>
> >> one of the clients
> >>
> >>>>>> we had an issue where ssh was mysteriously very slow to login and
> >>>>>> tracked it down to rsyslog causing issues.
> >>>>>>
> >> Simultaneously, one of the
> >>
> >>>>>> threads on the logging server spiked up to 100%. New log
> >>>>>>
> >> data from the
> >>
> >>>>>> client would stop showing up on the server at this poitn.
> >>>>>>
> >>>>>> On the client after attempting to gracefully stop rsyslog:
> >>>>>>
> >>>>>> vanguard etc # ps aux | grep -i rsyslog
> >>>>>> root      3764  0.0  0.1  5092 2564 ?        S    15:06   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >> /etc/rsyslog.conf
> >>
> >>>>>> root      3766  0.0  0.1  5092 2564 ?        S    15:06   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >> /etc/rsyslog.conf
> >>
> >>>>>> root      6203  0.0  0.0  1512  524 pts/3    S    15:09
> >>>>>>
> >>  0:00 grep -i
> >>
> >>>>>> rsyslog
> >>>>>> vanguard etc # strace -p 3764
> >>>>>> Process 3764 attached - interrupt to quit
> >>>>>> send(11,
> >>>>>>
> >> "\25\3\2\1\0uE\254\304\n\311\377\204}p\237O\211\322\211"...,
> >>
> >>>>>> 261, 0 <unfinished ...>
> >>>>>> Process 3764 detached
> >>>>>>
> >>>>>> FD 11 being the socket to the logging server, it never
> >>>>>>
> >> does anything
> >>
> >>>>>> asides from that send.
> >>>>>>
> >>>>>> On the server:
> >>>>>>
> >>>>>> ldap ~ # ps aux | grep rsyslog
> >>>>>> root      7370  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7379  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7380  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7381  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7382  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7383  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7384  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7385 28.3  0.0  83936  3380 ?        R    14:54   5:07
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> Stracing the 7385 pid just shows this repeating over and
> >>>>>>
> >> over (FD 4
> >>
> >>>>>> being the network socket again):
> >>>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1
> >>>>>>
> >> EAGAIN (Resource
> >>
> >>>>>> temporarily unavailable)
> >>>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1
> >>>>>>
> >> EAGAIN (Resource
> >>
> >>>>>> temporarily unavailable)
> >>>>>>
> >>>>>> It happened 3 times within a relatively short period of
> >>>>>>
> >> time, sometimes
> >>
> >>>>>> within minutes of me restarting rsyslog on both
> >>>>>>
> >> machines. I enabled
> >>
> >>>>>> debugging and it took around 8 hours to pop up again.
> >>>>>>
> >>>>>> With debugging, I pulled this from the rsyslog server:
> >>>>>>
> >>>>>> 6640.408108000:imtcp.c: New connect on NSD 0x59ddc0.
> >>>>>> 6640.408129000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550:
> >>>>>>
> >> pszRcvBuf (nil),
> >>
> >>>>>> lenRcvBuf 0
> >>>>>> 6640.408144000:imtcp.c: hasRcvInBuffer on nsd 0x59dcd0: pszRcvBuf
> >>>>>> 0x5c11c0, lenRcvBuf -1
> >>>>>> 6640.408158000:imtcp.c: hasRcvInBuffer on nsd 0x59dc10: pszRcvBuf
> >>>>>> 0x5cb510, lenRcvBuf -1
> >>>>>> 6640.408171000:imtcp.c: hasRcvInBuffer on nsd 0x5c3d70: pszRcvBuf
> >>>>>> 0x5d45c0, lenRcvBuf -1
> >>>>>> 6640.408185000:imtcp.c: hasRcvInBuffer on nsd 0x5cf460: pszRcvBuf
> >>>>>> 0x5dd010, lenRcvBuf -1
> >>>>>> 6640.408199000:imtcp.c: hasRcvInBuffer on nsd 0x5d8600: pszRcvBuf
> >>>>>> 0x5e58b0, lenRcvBuf -1
> >>>>>> 6640.408213000:imtcp.c: hasRcvInBuffer on nsd 0x59d690: pszRcvBuf
> >>>>>> 0x5ee150, lenRcvBuf -1
> >>>>>> 6640.408227000:imtcp.c: hasRcvInBuffer on nsd 0x59d2f0: pszRcvBuf
> >>>>>> 0x5f7320, lenRcvBuf -1
> >>>>>> 6640.408241000:imtcp.c: hasRcvInBuffer on nsd 0x5e8840: pszRcvBuf
> >>>>>> 0x5fe0a0, lenRcvBuf -1
> >>>>>> 6640.408255000:imtcp.c: hasRcvInBuffer on nsd 0x6027b0: pszRcvBuf
> >>>>>> 0x608350, lenRcvBuf -1
> >>>>>> 6640.408269000:imtcp.c: hasRcvInBuffer on nsd 0x5c3660: pszRcvBuf
> >>>>>> 0x612110, lenRcvBuf -1
> >>>>>> 6640.408283000:imtcp.c: hasRcvInBuffer on nsd 0x602690: pszRcvBuf
> >>>>>> 0x615040, lenRcvBuf -1
> >>>>>> 6640.408296000:imtcp.c: hasRcvInBuffer on nsd 0x5c3b60: pszRcvBuf
> >>>>>> 0x6218f0, lenRcvBuf -1
> >>>>>> 6640.408310000:imtcp.c: hasRcvInBuffer on nsd 0x60c150: pszRcvBuf
> >>>>>> 0x62a190, lenRcvBuf -1
> >>>>>> 6640.408324000:imtcp.c: hasRcvInBuffer on nsd 0x625420: pszRcvBuf
> >>>>>> 0x634fe0, lenRcvBuf -1
> >>>>>> 6640.408338000:imtcp.c: hasRcvInBuffer on nsd 0x6117b0: pszRcvBuf
> >>>>>> 0x63b000, lenRcvBuf -1
> >>>>>> 6640.408352000:imtcp.c: hasRcvInBuffer on nsd 0x603990: pszRcvBuf
> >>>>>> 0x643c70, lenRcvBuf -1
> >>>>>> 6640.408365000:imtcp.c: hasRcvInBuffer on nsd 0x633c00: pszRcvBuf
> >>>>>> 0x64dbb0, lenRcvBuf -1
> >>>>>> 6640.408379000:imtcp.c: hasRcvInBuffer on nsd 0x62cfa0: pszRcvBuf
> >>>>>> 0x650960, lenRcvBuf -1
> >>>>>> 6640.408393000:imtcp.c: hasRcvInBuffer on nsd 0x633d10: pszRcvBuf
> >>>>>> 0x65dbc0, lenRcvBuf -1
> >>>>>> 6640.408407000:imtcp.c: hasRcvInBuffer on nsd 0x5f2860: pszRcvBuf
> >>>>>> 0x666020, lenRcvBuf -1
> >>>>>> 6640.408421000:imtcp.c: hasRcvInBuffer on nsd 0x64d250: pszRcvBuf
> >>>>>> 0x66e480, lenRcvBuf -1
> >>>>>> 6640.408435000:imtcp.c: hasRcvInBuffer on nsd 0x659b30: pszRcvBuf
> >>>>>> 0x676d20, lenRcvBuf 78
> >>>>>> 6640.408448000:imtcp.c: hasRcvInBuffer on nsd 0x669fb0: pszRcvBuf
> >>>>>> 0x67fdd0, lenRcvBuf -1
> >>>>>> 6640.408462000:imtcp.c: hasRcvInBuffer on nsd 0x59a630: pszRcvBuf
> >>>>>> 0x687f10, lenRcvBuf -1
> >>>>>> 6640.408476000:imtcp.c: hasRcvInBuffer on nsd 0x6614c0: pszRcvBuf
> >>>>>> 0x6907b0, lenRcvBuf -1
> >>>>>> 6640.408490000:imtcp.c: hasRcvInBuffer on nsd 0x60c2d0: pszRcvBuf
> >>>>>> 0x699050, lenRcvBuf -1
> >>>>>> 6640.408504000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550:
> >>>>>>
> >> pszRcvBuf (nil),
> >>
> >>>>>> lenRcvBuf 0
> >>>>>>
> >>>>>> This repeats enough to generate around 29 million lines
> >>>>>>
> >> in the debug log
> >>
> >>>>>> file.
> >>>>>>
> >>>>>> In the client log, there's plenty of these which seems
> >>>>>>
> >> somewhat normal:
> >>
> >>>>>> 6509.743164000:imuxsock.c: --------imuxsock calling
> >>>>>>
> >> select, active file
> >>
> >>>>>> descriptors (max 11): 11
> >>>>>> 6509.743231000:imuxsock.c: Message from UNIX socket: #11
> >>>>>> 6509.743262000:imuxsock.c: logmsg: flags 4, from
> >>>>>>
> >> 'vanguard', msg May 20
> >>
> >>>>>> 23:15:09 bin/qmail-local[23198]: using .qmail .qmail-cfsc-forum
> >>>>>> 6509.743281000:imuxsock.c: Message has legacy syslog format.
> >>>>>> 6509.743302000:imuxsock.c: main queue: entry added, size
> >>>>>>
> >> now 579 entries
> >>
> >>>>>> 6509.743325000:imuxsock.c: wtpAdviseMaxWorkers signals busy
> >>>>>> 6509.743343000:imuxsock.c: main queue: EnqueueMsg
> >>>>>>
> >> advised worker start
> >>
> >>>>>> However comparing it to an earlier point in the log
> >>>>>>
> >> there's none of the
> >>
> >>>>>> associated tcp queue entries like this:
> >>>>>>
> >>>>>> 3587.234178000:main queue:Reg/w0: TCP sent 107 bytes,
> >>>>>>
> >> requested 107
> >>
> >>>>>> 3587.234204000:main queue:Reg/w0: main queue: entering
> >>>>>>
> >> rate limiter
> >>
> >>>>>> 3587.234229000:main queue:Reg/w0: main queue: entry
> >>>>>>
> >> deleted, state 0,
> >>
> >>>>>> size now 0 entries
> >>>>>> 3587.234255000:main queue:Reg/w0: Called action, logging
> >>>>>>
> >> to builtin-file
> >>
> >>>>>> 3587.234285000:main queue:Reg/w0:  (/var/log/maillog)
> >>>>>> 3587.234314000:main queue:Reg/w0: Called action, logging
> >>>>>>
> >> to builtin-fwd
> >>
> >>>>>> 3587.234340000:main queue:Reg/w0:  ldap.nmsrv.com
> >>>>>> 3587.234359000:main queue:Reg/w0:  ldap.nmsrv.com:10514/tcp
> >>>>>> 3587.234425000:main queue:Reg/w0: TCP sent 78 bytes, requested 78
> >>>>>> 3587.234446000:main queue:Reg/w0: main queue: entering
> >>>>>>
> >> rate limiter
> >>
> >>>>>> 3587.234471000:main queue:Reg/w0: main queue:Reg/w0: worker IDLE,
> >>>>>> waiting for work.
> >>>>>>
> >>>>>> Anyone able to shed some light on this situation? I've
> >>>>>>
> >> got the full
> >>
> >>>>>> debug logs if it's useful to anyone to take a closer look.
> >>>>>>
> >>>>>> Thanks!
> >>>>>>
> >>>>>> - Nathan
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >> --------------------------------------------------------------
> >> --------------
> >>
> >>>>
> >>>>
> >>>>> --------
> >>>>>
> >>>>>
> >>>>>> Server config file
> >>>>>>
> >>>>>>
> >>>>>>
> >> --------------------------------------------------------------
> >> --------------
> >>
> >>>>
> >>>>
> >>>>> --------
> >>>>>
> >>>>>
> >>>>>> $ModLoad immark.so # provides --MARK-- message capability
> >>>>>> $ModLoad imuxsock.so # provides support for local system
> >>>>>>
> >> logging (e.g.
> >>
> >>>>>> via logger command)
> >>>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> >>>>>>
> >>>>>> $umask 0137
> >>>>>> $DirCreateMode 0640
> >>>>>> $FileCreateMode 0640
> >>>>>> $FileOwner root
> >>>>>> $FileGroup admin
> >>>>>>
> >>>>>> $template DynFile,"/var/log/rsyslog/%HOSTNAME%/messages"
> >>>>>> $template DynFileCron,"/var/log/rsyslog/%HOSTNAME%/cron"
> >>>>>> $template DynFileSecure,"/var/log/rsyslog/%HOSTNAME%/secure"
> >>>>>> $template DynFileMail,"/var/log/rsyslog/%HOSTNAME%/maillog"
> >>>>>> $template DynFileSpool,"/var/log/rsyslog/%HOSTNAME%/spooler"
> >>>>>> $template DynFileBoot,"/var/log/rsyslog/%HOSTNAME%/boot"
> >>>>>> $template DynFileSyslog,"/var/log/rsyslog/%HOSTNAME%/syslog"
> >>>>>>
> >>>>>> :msg, contains, "no keys found for"            ~
> >>>>>> :msg, contains, "session opened for user"        ~
> >>>>>>
> >>>>>> # Log all kernel messages to the console.
> >>>>>> # Logging much else clutters up the screen.
> >>>>>> kern.*
> >>>>>>
> >> /dev/console
> >>
> >>>>>> # Log anything (except mail) of level info or higher.
> >>>>>> # Don't log private authentication messages!
> >>>>>> *.info;mail.none;cron.none                     ?DynFile
> >>>>>>
> >>>>>> *.warn;\
> >>>>>>         authpriv.none;cron.none;mail.none;news.none
> >>>>>>
> >> ?DynFileSyslog
> >>
> >>>>>> # Log all the mail messages in one place.
> >>>>>> mail.*
> >>>>>>
> >> ?DynFileMail
> >>
> >>>>>> # Log cron stuff
> >>>>>> cron.*
> >>>>>>
> >> ?DynFileCron
> >>
> >>>>>> # Everybody gets emergency messages
> >>>>>> *.emerg                                                 *
> >>>>>>
> >>>>>> # Save news errors of level crit and higher in a special file
> >>>>>> uucp,news.crit
> >>>>>>
> >> ?DynFileSpool
> >>
> >>>>>> # Save boot messages also to boot.log
> >>>>>> local7.*
> >>>>>>
> >> ?DynFileBoot
> >>
> >>>>>> # Relp config for PCI
> >>>>>> $ModLoad imrelp
> >>>>>> $InputRELPServerRun 2515
> >>>>>>
> >>>>>> # make gtls driver the default
> >>>>>> $DefaultNetstreamDriver gtls
> >>>>>>
> >>>>>> # certificate files
> >>>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
> >>>>>> $DefaultNetstreamDriverCertFile /etc/ssl/servercrt.pem
> >>>>>> $DefaultNetstreamDriverKeyFile /etc/ssl/serverkey.pem
> >>>>>>
> >>>>>> $ModLoad imtcp.so
> >>>>>>
> >>>>>> $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
> >>>>>> $InputTCPServerStreamDriverAuthMode anon # client is NOT
> >>>>>>
> >> authenticated
> >>
> >>>>>> $InputTCPServerRun 10514 # start up listener at port 10514
> >>>>>> $InputTCPMaxSessions 500
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >> --------------------------------------------------------------
> >> --------------
> >>
> >>>>
> >>>>
> >>>>> --------
> >>>>>
> >>>>>
> >>>>>> Client config file
> >>>>>>
> >>>>>>
> >>>>>>
> >> --------------------------------------------------------------
> >> --------------
> >>
> >>>>
> >>>>
> >>>>> --------
> >>>>>
> >>>>>
> >>>>>> $ModLoad imuxsock.so # provides support for local system
> >>>>>>
> >> logging (e.g.
> >>
> >>>>>> via logger command)
> >>>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> >>>>>>
> >>>>>> $umask 0137
> >>>>>> $DirCreateMode 0640
> >>>>>> $FileCreateMode 0640
> >>>>>> $FileOwner root
> >>>>>> $FileGroup admin
> >>>>>>
> >>>>>>
> >>>>>> # Log anything 'info' or higher, but lower than 'warn'.
> >>>>>> # Exclude authpriv, cron, mail, and news.  These are
> >>>>>>
> >> logged elsewhere.
> >>
> >>>>>> *.info;*.!warn;\
> >>>>>>         authpriv.none;cron.none;mail.none;news.none
> >>>>>>
> >> /var/log/messages
> >>
> >>>>>> # Log anything 'warn' or higher.
> >>>>>> # Exclude authpriv, cron, mail, and news.  These are
> >>>>>>
> >> logged elsewhere.
> >>
> >>>>>> *.warn;\
> >>>>>>         authpriv.none;cron.none;mail.none;news.none
> >>>>>>
> >> /var/log/syslog
> >>
> >>>>>> # Debugging information is logged here.
> >>>>>> *.=debug
> >>>>>>
> >> /var/log/debug
> >>
> >>>>>> # Private authentication message logging:
> >>>>>> authpriv.*
> >>>>>>
> >> /var/log/secure
> >>
> >>>>>> # Cron related logs:
> >>>>>> cron.*
> >>>>>>
> >> /var/log/cron
> >>
> >>>>>> # Mail related logs:
> >>>>>> mail.*
> >>>>>>
> >> /var/log/maillog
> >>
> >>>>>> # Emergency level messages go to all users:
> >>>>>> *.emerg                                                 *
> >>>>>>
> >>>>>>
> >>>>>> ## Uncomment these lines to use RELP instead for PCI
> >>>>>>
> >> compliance (stunnel
> >>
> >>>>>> required)
> >>>>>> #$ModLoad omrelp
> >>>>>> #*.*;mail.none :omrelp:localhost:2515;RSYSLOG_ForwardFormat
> >>>>>>
> >>>>>> # certificate files - just CA for a client
> >>>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
> >>>>>>
> >>>>>> # set up the action
> >>>>>> $DefaultNetstreamDriver gtls # use gtls netstream driver
> >>>>>> $ActionSendStreamDriverMode 1 # require TLS for the connection
> >>>>>> $ActionSendStreamDriverAuthMode anon # server is NOT
> >>>>>>
> >> authenticated
> >>
> >>>>>> *.* @@(o)ldap.nmsrv.com:10514 # send (all) messages
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> rsyslog mailing list
> >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>> http://www.rsyslog.comoften
> >>>>>>
> >>>>>>
> >>>>> --
> >>>>> Patrick Shen
> >>>>> Operations Engineer
> >>>>>
> >>>>> net mobile AG Shanghai office
> >>>>>
> >>>>> T: +86  21 61350900 - 222
> >>>>> F: +86  21 61350906
> >>>>> M: +86  13122245730
> >>>>>
> >>>>> FE55 6A7F 3192 F359 C1DF  12F6 9078 58B4 57AF 0BE3
> >>>>>
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>>>
> >>>>>
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>>>
> >>>>
> >>>
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Mon Jun 15 16:48:23 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 16:48:23 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>

Hi all,

I am having a hard time finding a suitable-fast but tickles solution to
obtaining low-precision time API under Linux. Please read here:

http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html

If I don't find a better solution, I'll probably be forced to run rsyslogd on
a tick, which would not be a good thing from a power consumption point of
view.

Comments and suggestions are highly appreciated.

Thanks,
Rainer


From epiphani at gmail.com  Mon Jun 15 16:55:57 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 15 Jun 2009 10:55:57 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
Message-ID: <e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>

I'm slightly confused - whats wrong with gettimeofday()?  It may be
higher precision, but it still
gives you the ability to find the nearest second with a few extra operations.

-Aaron

On Mon, Jun 15, 2009 at 10:48 AM, Rainer
Gerhards<rgerhards at hq.adiscon.com> wrote:
> Hi all,
>
> I am having a hard time finding a suitable-fast but tickles solution to
> obtaining low-precision time API under Linux. Please read here:
>
> http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html
>
> If I don't find a better solution, I'll probably be forced to run rsyslogd on
> a tick, which would not be a good thing from a power consumption point of
> view.
>
> Comments and suggestions are highly appreciated.
>
> Thanks,
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Mon Jun 15 16:59:21 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 16:59:21 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F937@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> Sent: Monday, June 15, 2009 4:56 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> I'm slightly confused - whats wrong with gettimeofday()?  It may be
> higher precision, but it still
> gives you the ability to find the nearest second with a few extra
operations.
> 
> -Aaron

Much too slow. For example, you can often write a sector quicker to disk than
gettimeofday() returns...

If I would base buffer-writing optimizations on gettimeofday(), the
gettimeofday() would eat up much more time than I can save with the
optimization.

Rainer

> 
> On Mon, Jun 15, 2009 at 10:48 AM, Rainer
> Gerhards<rgerhards at hq.adiscon.com> wrote:
> > Hi all,
> >
> > I am having a hard time finding a suitable-fast but tickles solution to
> > obtaining low-precision time API under Linux. Please read here:
> >
> >
http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html
> >
> > If I don't find a better solution, I'll probably be forced to run
rsyslogd
> on
> > a tick, which would not be a good thing from a power consumption point of
> > view.
> >
> > Comments and suggestions are highly appreciated.
> >
> > Thanks,
> > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Mon Jun 15 17:10:20 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 17:10:20 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>

I have updated the blogpost to hopefully provide a better description of the
situation:

http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> Sent: Monday, June 15, 2009 4:56 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> I'm slightly confused - whats wrong with gettimeofday()?  It may be
> higher precision, but it still
> gives you the ability to find the nearest second with a few extra
operations.
> 
> -Aaron
> 
> On Mon, Jun 15, 2009 at 10:48 AM, Rainer
> Gerhards<rgerhards at hq.adiscon.com> wrote:
> > Hi all,
> >
> > I am having a hard time finding a suitable-fast but tickles solution to
> > obtaining low-precision time API under Linux. Please read here:
> >
> >
http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html
> >
> > If I don't find a better solution, I'll probably be forced to run
rsyslogd
> on
> > a tick, which would not be a good thing from a power consumption point of
> > view.
> >
> > Comments and suggestions are highly appreciated.
> >
> > Thanks,
> > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From jmoyer at redhat.com  Mon Jun 15 17:13:16 2009
From: jmoyer at redhat.com (Jeff Moyer)
Date: Mon, 15 Jun 2009 11:13:16 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F937@GRFEXC.intern.adiscon.com>
	(Rainer Gerhards's message of "Mon, 15 Jun 2009 16:59:21 +0200")
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F937@GRFEXC.intern.adiscon.com>
Message-ID: <x49y6rtlecz.fsf@segfault.boston.devel.redhat.com>

"Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
>> Sent: Monday, June 15, 2009 4:56 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] high-performance, low-precision time API under
> linux?
>> 
>> I'm slightly confused - whats wrong with gettimeofday()?  It may be
>> higher precision, but it still
>> gives you the ability to find the nearest second with a few extra
> operations.
>> 
>> -Aaron
>
> Much too slow. For example, you can often write a sector quicker to disk than
> gettimeofday() returns...

Really?  That doesn't sound right to me.  Writing a sector to disk can
take 8ms or more.  One would hope gettimeofday would be faster than
that, given that it reports microseconds.  ;)  The exact resolution, of
course, depends on how your kernel is configured.  Here are the
measurements (in CPU cycles) I get running each call (time(NULL) and
gettimeofday(&tv, NULL)) 1000 times:

$ ./some-cycle-costs 
                                    fewest cycles  avg cycles
syscall(NR_time)                    2134           2721.99
syscall(NR_gettimeofday)            2156           2622.87

If I understand your original question properly, you don't need to
always have a timer running.  You can only schedule a timer when there
is I/O buffered.  I'm not sure whether your code lends itself to that or
not.

I would be interested in hearing more about what conditions cause
gettimeofday to be so slow.

Cheers,
Jeff

>> On Mon, Jun 15, 2009 at 10:48 AM, Rainer
>> Gerhards<rgerhards at hq.adiscon.com> wrote:
>> > Hi all,
>> >
>> > I am having a hard time finding a suitable-fast but tickles solution to
>> > obtaining low-precision time API under Linux. Please read here:
>> >
>> >
> http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html
>> >
>> > If I don't find a better solution, I'll probably be forced to run
> rsyslogd
>> on
>> > a tick, which would not be a good thing from a power consumption point of
>> > view.
>> >
>> > Comments and suggestions are highly appreciated.
>> >
>> > Thanks,
>> > Rainer


From epiphani at gmail.com  Mon Jun 15 17:32:13 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 15 Jun 2009 11:32:13 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
Message-ID: <e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>

I went over your post...

You shouldn't be seeing a 4ms response time on a gettimeofday() call -
this is my test:

gettimeofday({1245079355, 445434}, NULL) = 0 <0.000004>
gettimeofday({1245079355, 445503}, NULL) = 0 <0.000004>
gettimeofday({1245079355, 445560}, NULL) = 0 <0.000003>
gettimeofday({1245079355, 445629}, NULL) = 0 <0.000004>

That's 3-4 microseconds.

For the sake of argument, I checked time() as well, which is obsoleted
by gettimeofday()...

ime(NULL)                              = 1245079501 <0.000008>
time(NULL)                              = 1245079501 <0.000007>
time(NULL)                              = 1245079501 <0.000004>
time(NULL)                              = 1245079501 <0.000008>
time(NULL)                              = 1245079501 <0.000004>

About half the speed, but still in the microsecond range.

gettimeofday() is one of the fastest syscalls you can issue.
Obviously you don't want to beat on it, but I wouldn't worry about
calling it a few dozen times per second.

-Aaron


From rgerhards at hq.adiscon.com  Mon Jun 15 17:36:22 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 17:36:22 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F937@GRFEXC.intern.adiscon.com>
	<x49y6rtlecz.fsf@segfault.boston.devel.redhat.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F93B@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jeff Moyer
> Sent: Monday, June 15, 2009 5:13 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> "Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:
> 
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> >> Sent: Monday, June 15, 2009 4:56 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] high-performance, low-precision time API under
> > linux?
> >>
> >> I'm slightly confused - whats wrong with gettimeofday()?  It may be
> >> higher precision, but it still
> >> gives you the ability to find the nearest second with a few extra
> > operations.
> >>
> >> -Aaron
> >
> > Much too slow. For example, you can often write a sector quicker to disk
> than
> > gettimeofday() returns...
> 
> Really?  That doesn't sound right to me.  Writing a sector to disk can
> take 8ms or more.  One would hope gettimeofday would be faster than
> that, given that it reports microseconds.  ;)  The exact resolution, of
> course, depends on how your kernel is configured.  Here are the
> measurements (in CPU cycles) I get running each call (time(NULL) and
> gettimeofday(&tv, NULL)) 1000 times:
> 
> $ ./some-cycle-costs
>                                     fewest cycles  avg cycles
> syscall(NR_time)                    2134           2721.99
> syscall(NR_gettimeofday)            2156           2622.87

I've updated the blog post with my test program and some numbers:

http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html

(sorry for keeping this on the blog, but that makes it far easier to also ask
at other places...)

> If I understand your original question properly, you don't need to
> always have a timer running.  You can only schedule a timer when there
> is I/O buffered.  I'm not sure whether your code lends itself to that or
> not.

The code in question would not just be used for buffered writes. The problem
is also that I need to frequently re-set the timer, because it must not write
data if any new has arrived. In order to do this re-set handling properly, I
also need to know the system time.

Of course, I can schedule a specific timer while data is unwritten. But that
is just another tick processing if I don't reset the timer when there is no
need. But... all suggestions are welcome. The overall idea is that I would
like to implement something like the original callouot table. However,
callout worked on a tick. I intended to replace that with timers (more
precisely their pthreads-equivalent, which would give me some aids in awaking
the system in unexpected cases). That was the plan, but I overlooked that it
required many of the costly time() calls.

Rainer

> I would be interested in hearing more about what conditions cause
> gettimeofday to be so slow.
> 
> Cheers,
> Jeff
> 
> >> On Mon, Jun 15, 2009 at 10:48 AM, Rainer
> >> Gerhards<rgerhards at hq.adiscon.com> wrote:
> >> > Hi all,
> >> >
> >> > I am having a hard time finding a suitable-fast but tickles solution
to
> >> > obtaining low-precision time API under Linux. Please read here:
> >> >
> >> >
> >
http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html
> >> >
> >> > If I don't find a better solution, I'll probably be forced to run
> > rsyslogd
> >> on
> >> > a tick, which would not be a good thing from a power consumption point
of
> >> > view.
> >> >
> >> > Comments and suggestions are highly appreciated.
> >> >
> >> > Thanks,
> >> > Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From epiphani at gmail.com  Mon Jun 15 17:36:29 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 15 Jun 2009 11:36:29 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
Message-ID: <e7ca40f70906150836h38d3580fnb42aeef002145199@mail.gmail.com>

As more followup...

I would suggest running a timer per thread.  I assume every thread
would contain some type of
event-trigger loop, and I would make your timer calls in that loop,
caching for the remainder of
that iteration.  I'm not quite sure how well this would work, but you
can most certainly get away
with a fair bit of drift, especially if high-precision timers are not
required for log output.

-Aaron

On Mon, Jun 15, 2009 at 11:32 AM, Aaron Wiebe<epiphani at gmail.com> wrote:
> I went over your post...
>
> You shouldn't be seeing a 4ms response time on a gettimeofday() call -
> this is my test:
>
> gettimeofday({1245079355, 445434}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445503}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445560}, NULL) = 0 <0.000003>
> gettimeofday({1245079355, 445629}, NULL) = 0 <0.000004>
>
> That's 3-4 microseconds.
>
> For the sake of argument, I checked time() as well, which is obsoleted
> by gettimeofday()...
>
> ime(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000008>
> time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000007>
> time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000004>
> time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000008>
> time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000004>
>
> About half the speed, but still in the microsecond range.
>
> gettimeofday() is one of the fastest syscalls you can issue.
> Obviously you don't want to beat on it, but I wouldn't worry about
> calling it a few dozen times per second.
>
> -Aaron
>


From rgerhards at hq.adiscon.com  Mon Jun 15 17:45:55 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 17:45:55 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
	<e7ca40f70906150836h38d3580fnb42aeef002145199@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F93C@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> Sent: Monday, June 15, 2009 5:36 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> As more followup...
> 
> I would suggest running a timer per thread.  I assume every thread
> would contain some type of
> event-trigger loop, and I would make your timer calls in that loop,
> caching for the remainder of
> that iteration.  I'm not quite sure how well this would work, but you
> can most certainly get away
> with a fair bit of drift, especially if high-precision timers are not
> required for log output.

But that makes me running on that tick...

(more below)

> 
> -Aaron
> 
> On Mon, Jun 15, 2009 at 11:32 AM, Aaron Wiebe<epiphani at gmail.com> wrote:
> > I went over your post...
> >
> > You shouldn't be seeing a 4ms response time on a gettimeofday() call -
> > this is my test:
> >
> > gettimeofday({1245079355, 445434}, NULL) = 0 <0.000004>
> > gettimeofday({1245079355, 445503}, NULL) = 0 <0.000004>
> > gettimeofday({1245079355, 445560}, NULL) = 0 <0.000003>
> > gettimeofday({1245079355, 445629}, NULL) = 0 <0.000004>
> >
> > That's 3-4 microseconds.

Oops, sorry, yes, that is what I meant...

> > For the sake of argument, I checked time() as well, which is obsoleted
> > by gettimeofday()...
> >
> > ime(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000008>
> > time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000007>
> > time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000004>
> > time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000008>
> > time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000004>
> >
> > About half the speed, but still in the microsecond range.
> >
> > gettimeofday() is one of the fastest syscalls you can issue.
> > Obviously you don't want to beat on it, but I wouldn't worry about
> > calling it a few dozen times per second.

It's not about a few dozen times. I need to re-set the timer when I logically
write something to the file. The problem is that I don't know how long it was
since I did this the last time. So I would end up with calling time once per
logical write, that is a in the range of tenthousand to hunderedthousand
times per second.

A solution for this case may be to not reset any started timer and simply let
it fire whenever that be. So if there is a time started, don't care about the
other writes but do this only once the timer is done. That, of course, comes
at the cost of thread synchronization. Again, atomics may help here. And
while it is tempting to assume atomic writes for simple 32 bit values, this
is not always the case. So in the extreme case, and without (reliable)
atomics, one may end up with timers not being started where they should be.
Using sync primitives, however, is also very costly...

Rainer
> >
> > -Aaron
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From epiphani at gmail.com  Mon Jun 15 17:57:38 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 15 Jun 2009 11:57:38 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F93C@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
	<e7ca40f70906150836h38d3580fnb42aeef002145199@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F93C@GRFEXC.intern.adiscon.com>
Message-ID: <e7ca40f70906150857r53cd1b00yc0a748ba226c3a7c@mail.gmail.com>

On Mon, Jun 15, 2009 at 11:45 AM, Rainer
Gerhards<rgerhards at hq.adiscon.com> wrote:
>
> It's not about a few dozen times. I need to re-set the timer when I logically
> write something to the file. The problem is that I don't know how long it was
> since I did this the last time. So I would end up with calling time once per
> logical write, that is a in the range of tenthousand to hunderedthousand
> times per second.

I'll suggest this first, since it has the possibility of being the
best solution:  posix aio.  Have a look at `man 3 aio_write'.  That
should reduce your write() latency enough to not worry about IO
blocking and messing with the clock reasonably well.  I've used this
method in the past, and it generally works wonders - plus its not too
hard to implement.  The downsize is that it isn't very portable.

Otherwise, you're looking at batched writes and timer resets after the
batch completes.  There is nothing required in terms of delivering
messages to a file within a few seconds, so you could batch up writes
up to a high watermark / specified time prior to starting a write
cycle.

-Aaron


From rgerhards at hq.adiscon.com  Mon Jun 15 18:10:40 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 18:10:40 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com><e7ca40f70906150836h38d3580fnb42aeef002145199@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F93C@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150857r53cd1b00yc0a748ba226c3a7c@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F93D@GRFEXC.intern.adiscon.com>

OK, I've also had some other discussions. Looks like there is no quicker way
to do it. So I think I'll ditch the approach to have a generic capability to
do timeout processing and rather do specific code for each use case. As you
write, this is no problem for the file writer. I'll probably go to write
every 10 seconds or so (if there is a buffer flush request pending). I'll
evaluate the other use cases when I come along them.

Thanks everyone for commenting :)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> Sent: Monday, June 15, 2009 5:58 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> On Mon, Jun 15, 2009 at 11:45 AM, Rainer
> Gerhards<rgerhards at hq.adiscon.com> wrote:
> >
> > It's not about a few dozen times. I need to re-set the timer when I
> logically
> > write something to the file. The problem is that I don't know how long it
> was
> > since I did this the last time. So I would end up with calling time once
per
> > logical write, that is a in the range of tenthousand to hunderedthousand
> > times per second.
> 
> I'll suggest this first, since it has the possibility of being the
> best solution:  posix aio.  Have a look at `man 3 aio_write'.  That
> should reduce your write() latency enough to not worry about IO
> blocking and messing with the clock reasonably well.  I've used this
> method in the past, and it generally works wonders - plus its not too
> hard to implement.  The downsize is that it isn't very portable.
> 
> Otherwise, you're looking at batched writes and timer resets after the
> batch completes.  There is nothing required in terms of delivering
> messages to a file within a few seconds, so you could batch up writes
> up to a high watermark / specified time prior to starting a write
> cycle.
> 
> -Aaron
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From david at lang.hm  Mon Jun 15 19:01:01 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 15 Jun 2009 10:01:01 -0700 (PDT)
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906150956580.9159@asgard>

On Mon, 15 Jun 2009, Aaron Wiebe wrote:

> I went over your post...
>
> You shouldn't be seeing a 4ms response time on a gettimeofday() call -
> this is my test:
>
> gettimeofday({1245079355, 445434}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445503}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445560}, NULL) = 0 <0.000003>
> gettimeofday({1245079355, 445629}, NULL) = 0 <0.000004>
>
> That's 3-4 microseconds.
>
> For the sake of argument, I checked time() as well, which is obsoleted
> by gettimeofday()...
>
> ime(NULL)                              = 1245079501 <0.000008>
> time(NULL)                              = 1245079501 <0.000007>
> time(NULL)                              = 1245079501 <0.000004>
> time(NULL)                              = 1245079501 <0.000008>
> time(NULL)                              = 1245079501 <0.000004>
>
> About half the speed, but still in the microsecond range.
>
> gettimeofday() is one of the fastest syscalls you can issue.
> Obviously you don't want to beat on it, but I wouldn't worry about
> calling it a few dozen times per second.

this depends greatly on your specific system. linux has some wonderful 
speedups available (basicly it internally does updates only when time 
changes and the system call gets optimized to a read from memory), but 
this is not available on all systems and requires support from glibc. I 
don't remember if the limit is the CPU, 32 bit address space, kernel 
config options, or something else.

I remember seeing that without this kernel optimization a very fast 
pentium (3GHz+) took more than 1 microsecond to do a gettimeofday() call 
(tested by doing 1m calls and it took more than a second)

David Lang


From david at lang.hm  Mon Jun 15 19:12:01 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 15 Jun 2009 10:12:01 -0700 (PDT)
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906151007200.9159@asgard>

On Mon, 15 Jun 2009, Aaron Wiebe wrote:

> I went over your post...
>
> You shouldn't be seeing a 4ms response time on a gettimeofday() call -
> this is my test:
>
> gettimeofday({1245079355, 445434}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445503}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445560}, NULL) = 0 <0.000003>
> gettimeofday({1245079355, 445629}, NULL) = 0 <0.000004>
>
> That's 3-4 microseconds.
>
> For the sake of argument, I checked time() as well, which is obsoleted
> by gettimeofday()...
>
> ime(NULL)                              = 1245079501 <0.000008>
> time(NULL)                              = 1245079501 <0.000007>
> time(NULL)                              = 1245079501 <0.000004>
> time(NULL)                              = 1245079501 <0.000008>
> time(NULL)                              = 1245079501 <0.000004>
>
> About half the speed, but still in the microsecond range.
>
> gettimeofday() is one of the fastest syscalls you can issue.
> Obviously you don't want to beat on it, but I wouldn't worry about
> calling it a few dozen times per second.

Aaron,

rsyslog is able to recieve (not fully process, but recieve into the queue) 
over 300,000 messages/second from UDP without dropping any packets (256 
byte messages over gig ethernet), without maxing out the CPU on the thread 
recieving the messages. In the next week or so I hope to be able to do 
some tests with 10G ethernet ;-)

with 4 microseconds per gettimeofday() call, you can only do 250,000 
gettimeofday() calls per second. just doing a single call per message will 
take more time than you have.

David Lang


From epiphani at gmail.com  Mon Jun 15 19:19:49 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 15 Jun 2009 13:19:49 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <alpine.DEB.1.10.0906151007200.9159@asgard>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
	<alpine.DEB.1.10.0906151007200.9159@asgard>
Message-ID: <e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com>

Greetings David,

On Mon, Jun 15, 2009 at 1:12 PM, <david at lang.hm> wrote:

>> gettimeofday() is one of the fastest syscalls you can issue.
>> Obviously you don't want to beat on it, but I wouldn't worry about
>> calling it a few dozen times per second.
>
> Aaron,
>
> rsyslog is able to recieve (not fully process, but recieve into the queue)
> over 300,000 messages/second from UDP without dropping any packets (256
> byte messages over gig ethernet), without maxing out the CPU on the thread
> recieving the messages. In the next week or so I hope to be able to do
> some tests with 10G ethernet ;-)
>
> with 4 microseconds per gettimeofday() call, you can only do 250,000
> gettimeofday() calls per second. just doing a single call per message will
> take more time than you have.

I never meant to suggest that doing a gettimeofday() on every received
message made sense.  My point was that a gettimeofday() call is a very
lightweight system call - one of the lightest.  Using a light system
call is of course no replacement for proper application design, and
Rainers' question was a valid one.  Unfortunately, there isn't much in
the way of changes to the system calls being used that would improve
the execution.  (Though dropping in gettimeofday() instead of time()
would probably be a good idea, since time() is documented as being
obsolete to gettimeofday().)

-Aaron


From Thomas.Mieslinger at 1und1.de  Mon Jun 15 19:57:43 2009
From: Thomas.Mieslinger at 1und1.de (Thomas Mieslinger)
Date: Mon, 15 Jun 2009 19:57:43 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com><alpine.DEB.1.10.0906151007200.9159@asgard>
	<e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com>
Message-ID: <3DBDD4313FFFEA4F86E79AEF498136C3079B636D@WEBEXCHANGE.webde.local>

BTW: In the Informix DBMS the Software Architects decided to implement an OS independent timer thread to work arround (very costly in the late 90s) gettimeofday() calls.

I've run the code on some Solaris 10 boxes, because they were known for their slow gettimeofday() implementation

S11 snv_105 Pentium 4 3GHz 
time ./a.out 100000
real    0m0.087s
user    0m0.082s
sys     0m0.004s

S10 Generic_138889-03 Athlon 2,2GHz
time ./a.out 100000
real    0m0.028s
user    0m0.026s
sys     0m0.002s

ok, lets try this on an older boxes...
suse70 2.2.21 libc-2.1.3-141 2xP3 800
time ./a.out 100000

real    0m0.100s
user    0m0.080s
sys     0m0.020s

So, what are you using as a development box?


From rgerhards at hq.adiscon.com  Mon Jun 15 20:23:17 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 20:23:17 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com><alpine.DEB.1.10.0906151007200.9159@asgard><e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com>
	<3DBDD4313FFFEA4F86E79AEF498136C3079B636D@WEBEXCHANGE.webde.local>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F93E@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Thomas Mieslinger
> Sent: Monday, June 15, 2009 7:58 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> BTW: In the Informix DBMS the Software Architects decided to implement an
OS
> independent timer thread to work arround (very costly in the late 90s)
> gettimeofday() calls.
> 
> I've run the code on some Solaris 10 boxes, because they were known for
their
> slow gettimeofday() implementation
> 
> S11 snv_105 Pentium 4 3GHz
> time ./a.out 100000
> real    0m0.087s
> user    0m0.082s
> sys     0m0.004s
> 
> S10 Generic_138889-03 Athlon 2,2GHz
> time ./a.out 100000
> real    0m0.028s
> user    0m0.026s
> sys     0m0.002s
> 
> ok, lets try this on an older boxes...
> suse70 2.2.21 libc-2.1.3-141 2xP3 800
> time ./a.out 100000
> 
> real    0m0.100s
> user    0m0.080s
> sys     0m0.020s
> 
> So, what are you using as a development box?

I should have mentioned that I am typically running inside a virtualized
environment as I routinely need to run three to four machines in parallel for
testing.

If anyone is up for hardware contributions, just let me know ;)

Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From david at lang.hm  Mon Jun 15 20:27:15 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 15 Jun 2009 11:27:15 -0700 (PDT)
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F93E@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com><alpine.DEB.1.10.0906151007200.9159@asgard><e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com>
	<3DBDD4313FFFEA4F86E79AEF498136C3079B636D@WEBEXCHANGE.webde.local>
	<9B6E2A8877C38245BFB15CC491A11DA706F93E@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906151126180.9159@asgard>

On Mon, 15 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Thomas Mieslinger
>> Sent: Monday, June 15, 2009 7:58 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] high-performance, low-precision time API under
> linux?
>>
>> BTW: In the Informix DBMS the Software Architects decided to implement an
> OS
>> independent timer thread to work arround (very costly in the late 90s)
>> gettimeofday() calls.
>>
>> I've run the code on some Solaris 10 boxes, because they were known for
> their
>> slow gettimeofday() implementation
>>
>> S11 snv_105 Pentium 4 3GHz
>> time ./a.out 100000
>> real    0m0.087s
>> user    0m0.082s
>> sys     0m0.004s
>>
>> S10 Generic_138889-03 Athlon 2,2GHz
>> time ./a.out 100000
>> real    0m0.028s
>> user    0m0.026s
>> sys     0m0.002s
>>
>> ok, lets try this on an older boxes...
>> suse70 2.2.21 libc-2.1.3-141 2xP3 800
>> time ./a.out 100000
>>
>> real    0m0.100s
>> user    0m0.080s
>> sys     0m0.020s
>>
>> So, what are you using as a development box?
>
> I should have mentioned that I am typically running inside a virtualized
> environment as I routinely need to run three to four machines in parallel for
> testing.

ahh, virtualized boxes are especially bad for gettimeofday().
which virtualization are you using?

David Lang

> If anyone is up for hardware contributions, just let me know ;)
>
> Rainer
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Mon Jun 15 20:31:42 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 20:31:42 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com><alpine.DEB.1.10.0906151007200.9159@asgard><e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com><3DBDD4313FFFEA4F86E79AEF498136C3079B636D@WEBEXCHANGE.webde.local><9B6E2A8877C38245BFB15CC491A11DA706F93E@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906151126180.9159@asgard>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F93F@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Monday, June 15, 2009 8:27 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> On Mon, 15 Jun 2009, Rainer Gerhards wrote:
> 
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Thomas Mieslinger
> >> Sent: Monday, June 15, 2009 7:58 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] high-performance, low-precision time API under
> > linux?
> >>
> >> BTW: In the Informix DBMS the Software Architects decided to implement
an
> > OS
> >> independent timer thread to work arround (very costly in the late 90s)
> >> gettimeofday() calls.
> >>
> >> I've run the code on some Solaris 10 boxes, because they were known for
> > their
> >> slow gettimeofday() implementation
> >>
> >> S11 snv_105 Pentium 4 3GHz
> >> time ./a.out 100000
> >> real    0m0.087s
> >> user    0m0.082s
> >> sys     0m0.004s
> >>
> >> S10 Generic_138889-03 Athlon 2,2GHz
> >> time ./a.out 100000
> >> real    0m0.028s
> >> user    0m0.026s
> >> sys     0m0.002s
> >>
> >> ok, lets try this on an older boxes...
> >> suse70 2.2.21 libc-2.1.3-141 2xP3 800
> >> time ./a.out 100000
> >>
> >> real    0m0.100s
> >> user    0m0.080s
> >> sys     0m0.020s
> >>
> >> So, what are you using as a development box?
> >
> > I should have mentioned that I am typically running inside a virtualized
> > environment as I routinely need to run three to four machines in parallel
> for
> > testing.
> 
> ahh, virtualized boxes are especially bad for gettimeofday().
> which virtualization are you using?

This one is using vmware, but that doesn't really matter. Even on a real box,
as you said, the toll to pay for gettimeofday() is high compared to the rest
of the workload. So I actually do not care how much the absolute time is but
rather how it goes into the overall mix. The context switch itself is
expensive enough, so this makes everything that goes to kernel space rather
expensive.

As I had written earlier, I will probably give up the approach of a
generalized timeout handler in favor of many targeted implementations. It
would have been nice to have this callout-tablish facility, but it looks like
it is too expensive to implement.

Rainer

> 
> David Lang
> 
> > If anyone is up for hardware contributions, just let me know ;)
> >
> > Rainer
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Wed Jun 17 09:25:20 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 17 Jun 2009 09:25:20 +0200
Subject: [rsyslog] General availability of atomic operations?
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F96E@GRFEXC.intern.adiscon.com>

Hi all,

I am currently facing a situation where I can re-structure message object
represenation in a very efficient way if I have atomic operations available.
However, if they are not available, the very same implementation would have a
heavy hit on concurrency and overhead.

What I need is atomic increments and decrements (for reference counting). So
I wonder if support for these operations is available on a sufficiently large
number of platforms. I would appreciate feedback on this issue, including
pointers to place where else I could put this question.

Thanks,
Rainer	


From thomas.mieslinger at 1und1.de  Wed Jun 17 10:15:18 2009
From: thomas.mieslinger at 1und1.de (Thomas Mieslinger)
Date: Wed, 17 Jun 2009 10:15:18 +0200
Subject: [rsyslog] General availability of atomic operations?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F96E@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F96E@GRFEXC.intern.adiscon.com>
Message-ID: <4A38A616.1010408@1und1.de>

Rainer Gerhards wrote:
> What I need is atomic increments and decrements (for reference counting). So
> I wonder if support for these operations is available on a sufficiently large
> number of platforms. I would appreciate feedback on this issue, including
> pointers to place where else I could put this question.

The Apache Portable Runtime includes atomic read/set/add/inc/dec 
implementations for many platforms. But for example the sparc 
implementation has been riped out because of unclear copyrights.

-- 
Thomas Mieslinger


From martinmie at PartyGaming.com  Thu Jun 18 14:18:04 2009
From: martinmie at PartyGaming.com (Martin Mielke)
Date: Thu, 18 Jun 2009 14:18:04 +0200
Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
Message-ID: <0B1B9163138571439EAF804646F3F6AA0767973E@GIBSVWIN004X.partygaming.local>

Hi all,

I need to find a way to either convert or import syslog-ng.conf files
into rsyslog or learn the syntax to classify/store the logfiles
depending on the source.

For example, taken from a syslog-ng file:
--
source s_network { udp(); tcp(); };

filter f_systems {
host("1\.2\.3\.\d+");
};

filter f_switches {
host("5\.6\.7\.8|5\.6\.7\.9");
};

destination d_systems {file("/systems.log"); };
destination d_switches {file("/switches.log"); };

log { source(s_network); filter(f_systems); destination(d_systems); };
log { source(s_network); filter(f_switches); destination(d_switches); };
--

I have read the online documentation but I'm unable to find the right
answer...

Thanks!

Martin


This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. 

Any views or opinions are solely those of the sender.  This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk.




From rgerhards at hq.adiscon.com  Thu Jun 18 15:11:27 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 18 Jun 2009 15:11:27 +0200
Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
References: <0B1B9163138571439EAF804646F3F6AA0767973E@GIBSVWIN004X.partygaming.local>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9A4@GRFEXC.intern.adiscon.com>

FYI, there was also a forum post:

http://kb.monitorware.com/syslog-filters-and-destinations-under-rsyslog-t9858
.html

I'd appreciate if any other responses also go to the forum (so that others,
when searching, find the complete info).

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Martin Mielke
> Sent: Thursday, June 18, 2009 2:18 PM
> To: rsyslog-users
> Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
> 
> Hi all,
> 
> I need to find a way to either convert or import syslog-ng.conf files
> into rsyslog or learn the syntax to classify/store the logfiles
> depending on the source.
> 
> For example, taken from a syslog-ng file:
> --
> source s_network { udp(); tcp(); };
> 
> filter f_systems {
> host("1\.2\.3\.\d+");
> };
> 
> filter f_switches {
> host("5\.6\.7\.8|5\.6\.7\.9");
> };
> 
> destination d_systems {file("/systems.log"); };
> destination d_switches {file("/switches.log"); };
> 
> log { source(s_network); filter(f_systems); destination(d_systems); };
> log { source(s_network); filter(f_switches); destination(d_switches); };
> --
> 
> I have read the online documentation but I'm unable to find the right
> answer...
> 
> Thanks!
> 
> Martin
> 
> 
> This email and any attachments are confidential, and may be legally
privileged
> and protected by copyright. If you are not the intended recipient
> dissemination or copying of this email is prohibited. If you have received
> this in error, please notify the sender by replying by email and then
delete
> the email completely from your system.
> 
> Any views or opinions are solely those of the sender.  This communication
is
> not intended to form a binding contract unless expressly indicated to the
> contrary and properly authorised. Any actions taken on the basis of this
email
> are at the recipient's own risk.
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From alex at samad.com.au  Fri Jun 19 10:05:01 2009
From: alex at samad.com.au (Alex Samad)
Date: Fri, 19 Jun 2009 18:05:01 +1000
Subject: [rsyslog] Help with configurin rsyslog
Message-ID: <20090619080501.GB11392@samad.com.au>

Hi

I have a syslog line that looks like this

Jun 18 10:35:53 Thu Jun 18 10:42:37 2009 dlink-dir-855 System Log:
Stored configuration to non-volatile memory

notice the double dates stamps and it seems like I should be able to
rectify this with rsyslog, but I am not sure how.

I have had a quick look at the doco and the wiki, but there aren't any
example's of replacing/reformatting from specific hosts

note the above line is from a dlink dir-855 router/access point

I also have these
Jun 18 10:55:00 127.0.0.1 time="2009-06-18 10:55:00" msg="LOGIN User
logged in on TELNET (192.168.1.100)"

from a speedtouch 546v6 then I need to munge.

Any pointers or working examples would be helpful

Thanks


From rgerhards at hq.adiscon.com  Fri Jun 19 18:14:20 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Jun 2009 18:14:20 +0200
Subject: [rsyslog] memory alloc problems - any advise?
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>

Hi all,

during testing, I found out that some strange things happen when a queue is
set to large size. If I set e.g. the main message queue to 500,000 entries
and then shuffle messages very quickly at rsyslogd, it consumes considerable
memory. This is not surprising, as it is buffering not yet-processed
messages.

What is surprising is that the memory is not given back to the OS when the
data has been processed. Well... I know that some data blocks may still be
held and thus prevent giving memory back. But that should be solved when a
few new messages come in. Still, nothing changes in that scenario. What
puzzles me even more is that when I start a new burst of messages (I send
around one million each), it looks like almost only new memory is obtained
from the OS for them. At least here I had expected that the allocator would
re-use what was already claimed.

The interesting things is that when the queue size is small (50,000 message
for example), this behavior does not occur. Then, memory allocation and
deallocation works as expected.

I have also run the valgrind memory debugger on the app, and it does not
report any leaked memory.

If someone has an explanation of what causes this behavior, I would really
love to hear about it...

Thanks,
Rainer


From aoz.syn at gmail.com  Fri Jun 19 18:26:46 2009
From: aoz.syn at gmail.com (RB)
Date: Fri, 19 Jun 2009 10:26:46 -0600
Subject: [rsyslog] memory alloc problems - any advise?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
Message-ID: <4255c2570906190926v2399fcc7y7feddb010009faf5@mail.gmail.com>

On Fri, Jun 19, 2009 at 10:14, Rainer Gerhards<rgerhards at hq.adiscon.com> wrote:
> What is surprising is that the memory is not given back to the OS when the
> data has been processed. Well... I know that some data blocks may still be
> held and thus prevent giving memory back. But that should be solved when a
> few new messages come in. Still, nothing changes in that scenario. What
> puzzles me even more is that when I start a new burst of messages (I send
> around one million each), it looks like almost only new memory is obtained
> from the OS for them. At least here I had expected that the allocator would
> re-use what was already claimed.

Can you clarify what you mean by "not given back to the OS"?  Do you
mean that the RSS of rsyslog remains the same, or that the system
'free' memory does not go back up, or something else entirely?  I'm
wondering if you're seeing caching effects.


From rgerhards at hq.adiscon.com  Fri Jun 19 18:36:05 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Jun 2009 18:36:05 +0200
Subject: [rsyslog] memory alloc problems - any advise?
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
	<4255c2570906190926v2399fcc7y7feddb010009faf5@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9E0@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of RB
> Sent: Friday, June 19, 2009 6:27 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> 
> On Fri, Jun 19, 2009 at 10:14, Rainer Gerhards<rgerhards at hq.adiscon.com>
> wrote:
> > What is surprising is that the memory is not given back to the OS when
the
> > data has been processed. Well... I know that some data blocks may still
be
> > held and thus prevent giving memory back. But that should be solved when
a
> > few new messages come in. Still, nothing changes in that scenario. What
> > puzzles me even more is that when I start a new burst of messages (I send
> > around one million each), it looks like almost only new memory is
obtained
> > from the OS for them. At least here I had expected that the allocator
would
> > re-use what was already claimed.
> 
> Can you clarify what you mean by "not given back to the OS"?  Do you
> mean that the RSS of rsyslog remains the same, or that the system
> 'free' memory does not go back up, or something else entirely?  I'm
> wondering if you're seeing caching effects.

Rss remains the same, looks like sbrk() is not called. I poked somemore
around, and it looks like I need to do a malloc_trim() call in glibc to solve
this issue. Will post some stats soon...

Rainer 


From rgerhards at hq.adiscon.com  Fri Jun 19 18:57:06 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Jun 2009 18:57:06 +0200
Subject: [rsyslog] memory alloc problems - any advise?
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com><4255c2570906190926v2399fcc7y7feddb010009faf5@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E0@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9E1@GRFEXC.intern.adiscon.com>

The promised stats. I used GNU's malloc_stats() and malloc_trim().

I queried stats every 5 seconds (lines with # are my comments):

#Right after startup:
Arena 0:
system bytes     =     245760
in use bytes     =     244624
Total (incl. mmap):
system bytes     =    4247552
in use bytes     =    4246416
max mmap regions =          1
max mmap bytes   =    4001792

#First burst:
Arena 0:
system bytes     =  108318720
in use bytes     =  107797520
Arena 1:
system bytes     =   67661824
in use bytes     =   18954784
Total (incl. mmap):
system bytes     =  179982336
in use bytes     =  130754096
max mmap regions =          1
max mmap bytes   =    4001792

#processing...
Arena 0:
system bytes     =  143970304
in use bytes     =   93414224
Arena 1:
system bytes     =   91770880
in use bytes     =       2336
Arena 2:
system bytes     =  106659840
in use bytes     =       2336
Total (incl. mmap):
system bytes     =  346402816
in use bytes     =   97420688
max mmap regions =          1
max mmap bytes   =    4001792

#idle
Arena 0:
system bytes     =  143970304
in use bytes     =     255248
Arena 1:
system bytes     =   91770880
in use bytes     =       2288
Arena 2:
system bytes     =  106659840
in use bytes     =       2288
Total (incl. mmap):
system bytes     =  346402816
in use bytes     =    4261616
max mmap regions =          1
max mmap bytes   =    4001792

#second burst - so it looks like the memory is reused (good!)
Arena 0:
system bytes     =  143970304
in use bytes     =  102348464
Arena 1:
system bytes     =   91770880
in use bytes     =   14172352
Arena 2:
system bytes     =  106659840
in use bytes     =       3840
Total (incl. mmap):
system bytes     =  346402816
in use bytes     =  120526448
max mmap regions =          1
max mmap bytes   =    4001792

# first and second burst were equal, same messages, same amount of them.
# Still not all allocs are re-used (they must have been identical, too). 
# As it looks > 50% were not reused...
Arena 0:
system bytes     =  206884864
in use bytes     =  206110592
Arena 1:
system bytes     =   91770880
in use bytes     =       2336
Arena 2:
system bytes     =  162009088
in use bytes     =       2400
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =  210117120
max mmap regions =          1
max mmap bytes   =    4001792

#idle again
Arena 0:
system bytes     =  206884864
in use bytes     =     256000
Arena 1:
system bytes     =   91770880
in use bytes     =       3456
Arena 2:
system bytes     =  162009088
in use bytes     =       2352
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =    4263600
max mmap regions =          1
max mmap bytes   =    4001792

# still idle 
Arena 0:
system bytes     =  206884864
in use bytes     =     256000
Arena 1:
system bytes     =   91770880
in use bytes     =       3456
Arena 2:
system bytes     =  162009088
in use bytes     =       2352
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =    4263600
max mmap regions =          1
max mmap bytes   =    4001792

# still idle
Arena 0:
system bytes     =  206884864
in use bytes     =     256000
Arena 1:
system bytes     =   91770880
in use bytes     =       3456
Arena 2:
system bytes     =  162009088
in use bytes     =       2352
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =    4263600
max mmap regions =          1
max mmap bytes   =    4001792

# I did a malloc_trim() every 10th stat counter and here it happens.
# The number still look the same, but in the OS system monitor, I see freed
# memory (and it actually is freed if I look at how the system then
performs).
# So it looks like only the virtual address space was kept?
malloc trim (1)!
Arena 0:
system bytes     =  206884864
in use bytes     =     256000
Arena 1:
system bytes     =   91770880
in use bytes     =       3456
Arena 2:
system bytes     =  162009088
in use bytes     =       2352
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =    4263600
max mmap regions =          1
max mmap bytes   =    4001792

# no activity, so it remains at this level...
Arena 0:
system bytes     =  206884864
in use bytes     =     256000
Arena 1:
system bytes     =   91770880
in use bytes     =       3456
Arena 2:
system bytes     =  162009088
in use bytes     =       2352
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =    4263600
max mmap regions =          1
max mmap bytes   =    4001792

With that information, I'll probably better turn to a GNU mailing list. Still
I wonder if someone here has an opinion. If so, please let me know. For the
time being, it looks like I need to do some period malloc_trim() calls...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Friday, June 19, 2009 6:36 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of RB
> > Sent: Friday, June 19, 2009 6:27 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] memory alloc problems - any advise?
> >
> > On Fri, Jun 19, 2009 at 10:14, Rainer Gerhards<rgerhards at hq.adiscon.com>
> > wrote:
> > > What is surprising is that the memory is not given back to the OS when
> the
> > > data has been processed. Well... I know that some data blocks may still
> be
> > > held and thus prevent giving memory back. But that should be solved
when
> a
> > > few new messages come in. Still, nothing changes in that scenario. What
> > > puzzles me even more is that when I start a new burst of messages (I
send
> > > around one million each), it looks like almost only new memory is
> obtained
> > > from the OS for them. At least here I had expected that the allocator
> would
> > > re-use what was already claimed.
> >
> > Can you clarify what you mean by "not given back to the OS"?  Do you
> > mean that the RSS of rsyslog remains the same, or that the system
> > 'free' memory does not go back up, or something else entirely?  I'm
> > wondering if you're seeing caching effects.
> 
> Rss remains the same, looks like sbrk() is not called. I poked somemore
> around, and it looks like I need to do a malloc_trim() call in glibc to
solve
> this issue. Will post some stats soon...
> 
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From david at lang.hm  Fri Jun 19 22:14:16 2009
From: david at lang.hm (david at lang.hm)
Date: Fri, 19 Jun 2009 13:14:16 -0700 (PDT)
Subject: [rsyslog] Help with configurin rsyslog
In-Reply-To: <20090619080501.GB11392@samad.com.au>
References: <20090619080501.GB11392@samad.com.au>
Message-ID: <alpine.DEB.1.10.0906191305150.20154@asgard.lang.hm>

On Fri, 19 Jun 2009, Alex Samad wrote:

> I have a syslog line that looks like this
>
> Jun 18 10:35:53 Thu Jun 18 10:42:37 2009 dlink-dir-855 System Log:
> Stored configuration to non-volatile memory
>
> notice the double dates stamps and it seems like I should be able to
> rectify this with rsyslog, but I am not sure how.
>
> I have had a quick look at the doco and the wiki, but there aren't any
> example's of replacing/reformatting from specific hosts
>
> note the above line is from a dlink dir-855 router/access point
>
> I also have these
> Jun 18 10:55:00 127.0.0.1 time="2009-06-18 10:55:00" msg="LOGIN User
> logged in on TELNET (192.168.1.100)"
>
> from a speedtouch 546v6 then I need to munge.
>
> Any pointers or working examples would be helpful

I think that the key approach to do this is to create additional formats 
and then do conditional logging

something like the following pseudocode

if hostip = '1.1.1.1' then /var/log/messages;CustomFormat1
if hostip != '1.1.1.1' then /var/log/messages;StandardFormat

the first issue is to figure out how to filter the different messages 
seperate from each other

depending on how badly the message is misformed on the wire this may be 
hard to do. if you do not go through any relay servers you can filter 
based on the source IP of the connection to rsyslog

after you get the different logs filtered from each other you can figure 
out how to reformat each type of message.

hopefully the 'odd' messages only comes from a small number of sources so 
that you can do conditionals for the specific sources reasonably. (rsyslog 
doesn't currently have a mechanism to effectivly do table lookups)

I hope this helps.

David Lang


From david at lang.hm  Fri Jun 19 22:29:30 2009
From: david at lang.hm (david at lang.hm)
Date: Fri, 19 Jun 2009 13:29:30 -0700 (PDT)
Subject: [rsyslog] memory alloc problems - any advise?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906191324100.20154@asgard.lang.hm>

On Fri, 19 Jun 2009, Rainer Gerhards wrote:

> Hi all,
>
> during testing, I found out that some strange things happen when a queue is
> set to large size. If I set e.g. the main message queue to 500,000 entries
> and then shuffle messages very quickly at rsyslogd, it consumes considerable
> memory. This is not surprising, as it is buffering not yet-processed
> messages.
>
> What is surprising is that the memory is not given back to the OS when the
> data has been processed. Well... I know that some data blocks may still be
> held and thus prevent giving memory back. But that should be solved when a
> few new messages come in. Still, nothing changes in that scenario. What
> puzzles me even more is that when I start a new burst of messages (I send
> around one million each), it looks like almost only new memory is obtained
> from the OS for them. At least here I had expected that the allocator would
> re-use what was already claimed.
>
> The interesting things is that when the queue size is small (50,000 message
> for example), this behavior does not occur. Then, memory allocation and
> deallocation works as expected.

hmm, this doesn't match what I was seeing with the 3.x series. I had a 
machine with 32G of ram and was setting the max queue size to 1M entries.

I don't remember if it was freeing the memory as the queue emptied, but I 
know that I could hit it repeatedly with large peaks and it would not use 
additional memory.

> I have also run the valgrind memory debugger on the app, and it does not
> report any leaked memory.

remember that it will only consider it a leak if you no longer have _any_ 
refrence to the memory. if you have some pointer to it stashed away 
somewhere that you won't use it, it won't consider it a leak.

early next week I should have some new boxes in that I can do high-speed 
tests on again (64G ram with 10G network)

David Lang

> If someone has an explanation of what causes this behavior, I would really
> love to hear about it...
>
> Thanks,
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Fri Jun 19 22:34:48 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Jun 2009 22:34:48 +0200
Subject: [rsyslog] memory alloc problems - any advise?
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906191324100.20154@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9E3@GRFEXC.intern.adiscon.com>


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, June 19, 2009 10:30 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> 
> On Fri, 19 Jun 2009, Rainer Gerhards wrote:
> 
> > Hi all,
> >
> > during testing, I found out that some strange things happen 
> when a queue is
> > set to large size. If I set e.g. the main message queue to 
> 500,000 entries
> > and then shuffle messages very quickly at rsyslogd, it 
> consumes considerable
> > memory. This is not surprising, as it is buffering not yet-processed
> > messages.
> >
> > What is surprising is that the memory is not given back to 
> the OS when the
> > data has been processed. Well... I know that some data 
> blocks may still be
> > held and thus prevent giving memory back. But that should 
> be solved when a
> > few new messages come in. Still, nothing changes in that 
> scenario. What
> > puzzles me even more is that when I start a new burst of 
> messages (I send
> > around one million each), it looks like almost only new 
> memory is obtained
> > from the OS for them. At least here I had expected that the 
> allocator would
> > re-use what was already claimed.
> >
> > The interesting things is that when the queue size is small 
> (50,000 message
> > for example), this behavior does not occur. Then, memory 
> allocation and
> > deallocation works as expected.
> 
> hmm, this doesn't match what I was seeing with the 3.x 
> series. I had a 
> machine with 32G of ram and was setting the max queue size to 
> 1M entries.

I checked with v3, and it had the same behavior (a little less strong, but
still).

> 
> I don't remember if it was freeing the memory as the queue 
> emptied, but I 
> know that I could hit it repeatedly with large peaks and it 
> would not use 
> additional memory.
> 
> > I have also run the valgrind memory debugger on the app, 
> and it does not
> > report any leaked memory.
> 
> remember that it will only consider it a leak if you no 
> longer have _any_ 
> refrence to the memory. if you have some pointer to it stashed away 
> somewhere that you won't use it, it won't consider it a leak.

Valgrind can print out those allocations. In short, there were nothing
unexpected (the usual 81 when rsyslogd terminates, and they pose no problem).
As you can see in my later posts, the issue seems actually to be rooted in
glibc.

> 
> early next week I should have some new boxes in that I can do 
> high-speed 
> tests on again (64G ram with 10G network)

That's good news. I did a lot of optimizations this week, and think I can
release a new version with them mid next week. I hope you will see even
further improvement. In the v5 engine, I think, I have discovered a memory
leak, but I didn't investigate today. Maybe just the same issue I've later
found wiht the malloc calls...

Rainer

> 
> David Lang
> 
> > If someone has an explanation of what causes this behavior, 
> I would really
> > love to hear about it...
> >
> > Thanks,
> > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 


From david at lang.hm  Fri Jun 19 23:36:31 2009
From: david at lang.hm (david at lang.hm)
Date: Fri, 19 Jun 2009 14:36:31 -0700 (PDT)
Subject: [rsyslog] memory alloc problems - any advise?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9E3@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906191324100.20154@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E3@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906191352480.20154@asgard.lang.hm>

On Fri, 19 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Fri, 19 Jun 2009, Rainer Gerhards wrote:
>>
>>> Hi all,
>>>
>>> during testing, I found out that some strange things happen
>> when a queue is
>>> set to large size. If I set e.g. the main message queue to
>> 500,000 entries
>>> and then shuffle messages very quickly at rsyslogd, it
>> consumes considerable
>>> memory. This is not surprising, as it is buffering not yet-processed
>>> messages.
>>>
>>> What is surprising is that the memory is not given back to
>> the OS when the
>>> data has been processed. Well... I know that some data
>> blocks may still be
>>> held and thus prevent giving memory back. But that should
>> be solved when a
>>> few new messages come in. Still, nothing changes in that
>> scenario. What
>>> puzzles me even more is that when I start a new burst of
>> messages (I send
>>> around one million each), it looks like almost only new
>> memory is obtained
>>> from the OS for them. At least here I had expected that the
>> allocator would
>>> re-use what was already claimed.
>>>
>>> The interesting things is that when the queue size is small
>> (50,000 message
>>> for example), this behavior does not occur. Then, memory
>> allocation and
>>> deallocation works as expected.
>>
>> hmm, this doesn't match what I was seeing with the 3.x
>> series. I had a
>> machine with 32G of ram and was setting the max queue size to
>> 1M entries.
>
> I checked with v3, and it had the same behavior (a little less strong, but
> still).

interesting.

when I was doing the tests that showed rsyslog able to recieve at gig-e 
line speed I would run tests repeatedly without any problems (with each 
test sending millions of logs, things sitting idle for hours, then another 
burst of millions of logs, repeat)

if I set the queue size too large for my ram the system would start paging 
and die, so I'm sure that it wasn't failing to release memory.

now, one thing could be that my tests were of messages that were all the 
same size, it could be that you are seeing some effect from fragmentation 
with different message size.

David Lang

>>
>> I don't remember if it was freeing the memory as the queue
>> emptied, but I
>> know that I could hit it repeatedly with large peaks and it
>> would not use
>> additional memory.
>>
>>> I have also run the valgrind memory debugger on the app,
>> and it does not
>>> report any leaked memory.
>>
>> remember that it will only consider it a leak if you no
>> longer have _any_
>> refrence to the memory. if you have some pointer to it stashed away
>> somewhere that you won't use it, it won't consider it a leak.
>
> Valgrind can print out those allocations. In short, there were nothing
> unexpected (the usual 81 when rsyslogd terminates, and they pose no problem).
> As you can see in my later posts, the issue seems actually to be rooted in
> glibc.
>
>>
>> early next week I should have some new boxes in that I can do
>> high-speed
>> tests on again (64G ram with 10G network)
>
> That's good news. I did a lot of optimizations this week, and think I can
> release a new version with them mid next week. I hope you will see even
> further improvement. In the v5 engine, I think, I have discovered a memory
> leak, but I didn't investigate today. Maybe just the same issue I've later
> found wiht the malloc calls...
>
> Rainer
>
>>
>> David Lang
>>
>>> If someone has an explanation of what causes this behavior,
>> I would really
>>> love to hear about it...
>>>
>>> Thanks,
>>> Rainer
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From jiri.hlinka at gmail.com  Sat Jun 20 21:47:45 2009
From: jiri.hlinka at gmail.com (=?ISO-8859-2?Q?Ji=F8=ED_Hlinka?=)
Date: Sat, 20 Jun 2009 21:47:45 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
Message-ID: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com>

Hello,
is there any info about what rsyslog version will be available in RHEL
6? I suppose it will be 3.x, but im not sure, maybe 4.x version in the
time "RHEL6 package freeze" will be in stable status?

Thank you,
Jiri H.


From rgerhards at hq.adiscon.com  Sun Jun 21 11:58:57 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 21 Jun 2009 11:58:57 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>

That a good question and I, too, would like to know the answer. In any case,
I hope it will not be v2. Giving that RHEL is a conservative distribution
(and it has to be for the target base), I would assume that if they go for a
newer version, it is v3.

Rainer 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
> Sent: Saturday, June 20, 2009 9:48 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] RSyslog version in RHEL 6
> 
> Hello,
> is there any info about what rsyslog version will be available in RHEL
> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in the
> time "RHEL6 package freeze" will be in stable status?
> 
> Thank you,
> Jiri H.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 


From david at lang.hm  Sun Jun 21 12:15:12 2009
From: david at lang.hm (david at lang.hm)
Date: Sun, 21 Jun 2009 03:15:12 -0700 (PDT)
Subject: [rsyslog] RSyslog version in RHEL 6
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906210309480.23743@asgard.lang.hm>

On Sun, 21 Jun 2009, Rainer Gerhards wrote:

> That a good question and I, too, would like to know the answer. In any case,
> I hope it will not be v2. Giving that RHEL is a conservative distribution
> (and it has to be for the target base), I would assume that if they go for a
> newer version, it is v3.

not knowing when the freeze date is it's hard to say.

remember that the huge performance benifits of v4 (and v5 when it comes 
out) would also be very useful for the target base.

not to mention all the subtle fixes you have done whiel pursuing 
performance.

David Lang

> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
>> Sent: Saturday, June 20, 2009 9:48 PM
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] RSyslog version in RHEL 6
>>
>> Hello,
>> is there any info about what rsyslog version will be available in RHEL
>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in the
>> time "RHEL6 package freeze" will be in stable status?
>>
>> Thank you,
>> Jiri H.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Sun Jun 21 14:54:49 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 21 Jun 2009 14:54:49 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906210309480.23743@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9E6@GRFEXC.intern.adiscon.com>


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Sunday, June 21, 2009 12:15 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] RSyslog version in RHEL 6
> 
> On Sun, 21 Jun 2009, Rainer Gerhards wrote:
> 
> > That a good question and I, too, would like to know the 
> answer. In any case,
> > I hope it will not be v2. Giving that RHEL is a 
> conservative distribution
> > (and it has to be for the target base), I would assume that 
> if they go for a
> > newer version, it is v3.
> 
> not knowing when the freeze date is it's hard to say.
> 
> remember that the huge performance benifits of v4 (and v5 
> when it comes 
> out) would also be very useful for the target base.
> 
> not to mention all the subtle fixes you have done whiel pursuing 
> performance.

Yeah, that's right. But anyway ... I am not even asked, so I am a bystander
as most of the rest of this list ;)

> 
> David Lang
> 
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
> >> Sent: Saturday, June 20, 2009 9:48 PM
> >> To: rsyslog at lists.adiscon.com
> >> Subject: [rsyslog] RSyslog version in RHEL 6
> >>
> >> Hello,
> >> is there any info about what rsyslog version will be 
> available in RHEL
> >> 6? I suppose it will be 3.x, but im not sure, maybe 4.x 
> version in the
> >> time "RHEL6 package freeze" will be in stable status?
> >>
> >> Thank you,
> >> Jiri H.
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 


From david at lang.hm  Sun Jun 21 23:29:11 2009
From: david at lang.hm (david at lang.hm)
Date: Sun, 21 Jun 2009 14:29:11 -0700 (PDT)
Subject: [rsyslog] RSyslog version in RHEL 6
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9E6@GRFEXC.intern.adiscon.com>
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906210309480.23743@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E6@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906211427280.23743@asgard.lang.hm>

On Sun, 21 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Sun, 21 Jun 2009, Rainer Gerhards wrote:
>>
>>> That a good question and I, too, would like to know the
>> answer. In any case,
>>> I hope it will not be v2. Giving that RHEL is a
>> conservative distribution
>>> (and it has to be for the target base), I would assume that
>> if they go for a
>>> newer version, it is v3.
>>
>> not knowing when the freeze date is it's hard to say.
>>
>> remember that the huge performance benifits of v4 (and v5
>> when it comes
>> out) would also be very useful for the target base.
>>
>> not to mention all the subtle fixes you have done whiel pursuing
>> performance.
>
> Yeah, that's right. But anyway ... I am not even asked, so I am a bystander
> as most of the rest of this list ;)

Iagree that you are not asked, but since you hae a lot of influance on the 
schedule of when 4.x moves to stable you are a little more than a 
bystander ;-)

does anyone know the RHEL schedule? I would hate to see them lock in 3.x a 
few days before 4.x becomes stable

David Lang

>>
>> David Lang
>>
>>> Rainer
>>>
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com
>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
>>>> Sent: Saturday, June 20, 2009 9:48 PM
>>>> To: rsyslog at lists.adiscon.com
>>>> Subject: [rsyslog] RSyslog version in RHEL 6
>>>>
>>>> Hello,
>>>> is there any info about what rsyslog version will be
>> available in RHEL
>>>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x
>> version in the
>>>> time "RHEL6 package freeze" will be in stable status?
>>>>
>>>> Thank you,
>>>> Jiri H.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rio at rio.st  Mon Jun 22 03:44:56 2009
From: rio at rio.st (=?ISO-2022-JP?B?GyRCTkcbKEIgGyRCRiNFRBsoQg==?=)
Date: Mon, 22 Jun 2009 10:44:56 +0900
Subject: [rsyslog] RSyslog version in RHEL 6
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
Message-ID: <44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st>

Hi all,

I guess RHEL6 will be based on Fedora 11 & 12.
Fedora 11 was shipped with rsyslog-3.21.11-1.fc11.
It's sure that RHEL6 includes v3 at least.

Feature Freeze of Fedora 12 is planned on 2009-07-28.
http://fedoraproject.org/wiki/Schedule
If v4 is stable and requested to be included in F12, v4 will be  
shipped in F12.

Best Rio.

On 2009/06/21, at 18:58, Rainer Gerhards wrote:

> That a good question and I, too, would like to know the answer. In  
> any case,
> I hope it will not be v2. Giving that RHEL is a conservative  
> distribution
> (and it has to be for the target base), I would assume that if they  
> go for a
> newer version, it is v3.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
>> Sent: Saturday, June 20, 2009 9:48 PM
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] RSyslog version in RHEL 6
>>
>> Hello,
>> is there any info about what rsyslog version will be available in  
>> RHEL
>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in  
>> the
>> time "RHEL6 package freeze" will be in stable status?
>>
>> Thank you,
>> Jiri H.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com



From rgerhards at hq.adiscon.com  Mon Jun 22 11:16:02 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 22 Jun 2009 11:16:02 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906210309480.23743@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA706F9E6@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906211427280.23743@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9EB@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Sunday, June 21, 2009 11:29 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] RSyslog version in RHEL 6
> 
> On Sun, 21 Jun 2009, Rainer Gerhards wrote:

> >
> > Yeah, that's right. But anyway ... I am not even asked, so I am a
bystander
> > as most of the rest of this list ;)
> 
> Iagree that you are not asked, but since you hae a lot of influance on the
> schedule of when 4.x moves to stable you are a little more than a
> bystander ;-)


Well, the good thing is that the current v4-beta was actually scheduled to be
declared stable "these days". Will see that I make this "tomorrow". However,
the version that has most of the optimizations needs to go through the beta
stage (and needs so for good reasons), so this is another ~ three month from
now. But it probably would be sufficient to have v4 included, hopefully this
means newer versions of that same major version will also be included...

Rainer


From rgerhards at hq.adiscon.com  Mon Jun 22 13:10:32 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 22 Jun 2009 13:10:32 +0200
Subject: [rsyslog] memory alloc problems - any advise?
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906191324100.20154@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA706F9E3@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906191352480.20154@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9EE@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, June 19, 2009 11:37 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> 
> On Fri, 19 Jun 2009, Rainer Gerhards wrote:
> 
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >>
> >> On Fri, 19 Jun 2009, Rainer Gerhards wrote:
> >>
> > I checked with v3, and it had the same behavior (a little less strong,
but
> > still).
> 
> interesting.
> 
> when I was doing the tests that showed rsyslog able to recieve at gig-e
> line speed I would run tests repeatedly without any problems (with each
> test sending millions of logs, things sitting idle for hours, then another
> burst of millions of logs, repeat)
> 
> if I set the queue size too large for my ram the system would start paging
> and die, so I'm sure that it wasn't failing to release memory.
> 
> now, one thing could be that my tests were of messages that were all the
> same size, it could be that you are seeing some effect from fragmentation
> with different message size.

My test driver produces almost identical messages, with mostly the same size.
I have not yet implemented any randomization into it. So in theory, it should
be fairly easy for the malloc() subsystem to reuse freed memory.

Anyhow, based on the malloc stats I got, this really looks like an issue with
the allocator itself. I'll for now implement an alloc_trim() every 100,000
messages or so, that should keep care of this anomaly. Interestingly, v4 now
does fewer allocs than v3. It almost looks like the alloc subsystem also has
a periodic "garbage collection" counter, which now seems to be triggered less
often.

For v5, I have different memory alloc on my mind. My long-term goal is to
have almost no alloc/free pre message (in the magnitude of 1/1024 per
message). But this is rather intrusive and will not have any immediate hard
benefit. Thus I am not targeting this is something that needs to be done now.
For v4, however, I intend to do some more optimizations, some of which
further reduce the number of alloc/free calls needed per message.

I would definitely be a perfect match if you could test out these things on a
high-performance machine.

Rainer



From pvrabec at redhat.com  Mon Jun 22 15:53:10 2009
From: pvrabec at redhat.com (Peter Vrabec)
Date: Mon, 22 Jun 2009 15:53:10 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
In-Reply-To: <44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st>
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
	<44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st>
Message-ID: <200906221553.10791.pvrabec@redhat.com>

Hi folks,

rsyslog maintainer is on vacation at the moment so I'm answering this Q. :)

At first we need v4 stable. If we have stable v4 we will push it into rawhide 
and F11. This must be done ASAP, I see deadline in mid-July.

Peter.


On Monday 22 June 2009 03:44:56 am ? ?? wrote:
> Hi all,
>
> I guess RHEL6 will be based on Fedora 11 & 12.
> Fedora 11 was shipped with rsyslog-3.21.11-1.fc11.
> It's sure that RHEL6 includes v3 at least.
>
> Feature Freeze of Fedora 12 is planned on 2009-07-28.
> http://fedoraproject.org/wiki/Schedule
> If v4 is stable and requested to be included in F12, v4 will be
> shipped in F12.
>
> Best Rio.
>
> On 2009/06/21, at 18:58, Rainer Gerhards wrote:
> > That a good question and I, too, would like to know the answer. In
> > any case,
> > I hope it will not be v2. Giving that RHEL is a conservative
> > distribution
> > (and it has to be for the target base), I would assume that if they
> > go for a
> > newer version, it is v3.
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
> >> Sent: Saturday, June 20, 2009 9:48 PM
> >> To: rsyslog at lists.adiscon.com
> >> Subject: [rsyslog] RSyslog version in RHEL 6
> >>
> >> Hello,
> >> is there any info about what rsyslog version will be available in
> >> RHEL
> >> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in
> >> the
> >> time "RHEL6 package freeze" will be in stable status?
> >>
> >> Thank you,
> >> Jiri H.
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com



From rgerhards at hq.adiscon.com  Mon Jun 22 14:24:13 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 22 Jun 2009 14:24:13 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com><44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st>
	<200906221553.10791.pvrabec@redhat.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com>

Hi Peter,

that sounds quite interesting. I'll see that I get the v4-stable out
tomorrow.

Thanks for following up...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Peter Vrabec
> Sent: Monday, June 22, 2009 3:53 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] RSyslog version in RHEL 6
> 
> Hi folks,
> 
> rsyslog maintainer is on vacation at the moment so I'm answering this Q. :)
> 
> At first we need v4 stable. If we have stable v4 we will push it into
rawhide
> and F11. This must be done ASAP, I see deadline in mid-July.
> 
> Peter.
> 
> 
> On Monday 22 June 2009 03:44:56 am ? ?? wrote:
> > Hi all,
> >
> > I guess RHEL6 will be based on Fedora 11 & 12.
> > Fedora 11 was shipped with rsyslog-3.21.11-1.fc11.
> > It's sure that RHEL6 includes v3 at least.
> >
> > Feature Freeze of Fedora 12 is planned on 2009-07-28.
> > http://fedoraproject.org/wiki/Schedule
> > If v4 is stable and requested to be included in F12, v4 will be
> > shipped in F12.
> >
> > Best Rio.
> >
> > On 2009/06/21, at 18:58, Rainer Gerhards wrote:
> > > That a good question and I, too, would like to know the answer. In
> > > any case,
> > > I hope it will not be v2. Giving that RHEL is a conservative
> > > distribution
> > > (and it has to be for the target base), I would assume that if they
> > > go for a
> > > newer version, it is v3.
> > >
> > > Rainer
> > >
> > >> -----Original Message-----
> > >> From: rsyslog-bounces at lists.adiscon.com
> > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
> > >> Sent: Saturday, June 20, 2009 9:48 PM
> > >> To: rsyslog at lists.adiscon.com
> > >> Subject: [rsyslog] RSyslog version in RHEL 6
> > >>
> > >> Hello,
> > >> is there any info about what rsyslog version will be available in
> > >> RHEL
> > >> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in
> > >> the
> > >> time "RHEL6 package freeze" will be in stable status?
> > >>
> > >> Thank you,
> > >> Jiri H.
> > >> _______________________________________________
> > >> rsyslog mailing list
> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> http://www.rsyslog.com
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Mon Jun 22 19:04:35 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 22 Jun 2009 19:04:35 +0200
Subject: [rsyslog] memory alloc problems - any advise?
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906191324100.20154@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA706F9E3@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906191352480.20154@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706F9EE@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9F6@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Monday, June 22, 2009 1:11 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> My test driver produces almost identical messages, with mostly the same
size.
> I have not yet implemented any randomization into it. So in theory, it
should
> be fairly easy for the malloc() subsystem to reuse freed memory.
> 
> Anyhow, based on the malloc stats I got, this really looks like an issue
with
> the allocator itself. I'll for now implement an alloc_trim() every 100,000
> messages or so, that should keep care of this anomaly. Interestingly, v4
now
> does fewer allocs than v3. It almost looks like the alloc subsystem also
has
> a periodic "garbage collection" counter, which now seems to be triggered
less
> often.

Just FYI, I have implemented this algorithm now and it works pretty well.
Also, if there is an impact on performance, it is barely noticeable. 

I have also fixed a memory leak in v5 today, and run the same tests with v5
that I ran against v4. It is really impressive how much better it performs
(factor two, at least). On my low-end lab virtual machine, I am currently
processing around 150,00 tcp msgs/s (around 70,000 with v4) and this is also
close to sustained rate (that probably is around 10% below for v5).

Just thought I let you know.

Rainer


From tbergfeld at hq.adiscon.com  Tue Jun 23 11:25:52 2009
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Tue, 23 Jun 2009 11:25:52 +0200
Subject: [rsyslog] rsyslog 4.2.0 (v4-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA01@GRFEXC.intern.adiscon.com>

Hi all,
We have released the first version of rsyslog v4-stable today. It is the next
major step after v3 most importantly offering large performance improvements.
This version went through a long beta-phase and we did not have any specifc
bug reports within the last couple of weeks, so we honestly think this
release is ready for production use.

Download:

http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-161.phtml

Changelog:

http://www.rsyslog.com/Article376.phtml

As always, feedback is appreciated.

Best regards,
Tom Bergfeld

--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From sachulinux at gmail.com  Tue Jun 23 13:59:20 2009
From: sachulinux at gmail.com (Sachin Sawant)
Date: Tue, 23 Jun 2009 17:29:20 +0530
Subject: [rsyslog] template for file input module in Rsyslog
Message-ID: <edc9be030906230459q4090383sc8427501d8a413a7@mail.gmail.com>

Hi All,



I would like to monitor PostgreSQL log files using file input module in
Rsyslog. PostgreSQL create log files based on date, so is it possible to
monitor file without using absolute path?



Regards,

* *

*Sachin Sawant

*


From rgerhards at hq.adiscon.com  Tue Jun 23 14:47:34 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 23 Jun 2009 14:47:34 +0200
Subject: [rsyslog] template for file input module in Rsyslog
References: <edc9be030906230459q4090383sc8427501d8a413a7@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA0D@GRFEXC.intern.adiscon.com>

I think this is not currently supported..

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Sachin Sawant
> Sent: Tuesday, June 23, 2009 1:59 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] template for file input module in Rsyslog
> 
> Hi All,
> 
> 
> 
> I would like to monitor PostgreSQL log files using file input module in
> Rsyslog. PostgreSQL create log files based on date, so is it possible to
> monitor file without using absolute path?
> 
> 
> 
> Regards,
> 
> * *
> 
> *Sachin Sawant
> 
> *
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From martinmie at PartyGaming.com  Tue Jun 23 17:07:35 2009
From: martinmie at PartyGaming.com (Martin Mielke)
Date: Tue, 23 Jun 2009 17:07:35 +0200
Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9A4@GRFEXC.intern.adiscon.com>
References: <0B1B9163138571439EAF804646F3F6AA0767973E@GIBSVWIN004X.partygaming.local>
	<9B6E2A8877C38245BFB15CC491A11DA706F9A4@GRFEXC.intern.adiscon.com>
Message-ID: <0B1B9163138571439EAF804646F3F6AA07878F4E@GIBSVWIN004X.partygaming.local>

Sorry for the noise.

It seems that the post is pending on some reply ;-)


Cheers,


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: 18 June 2009 15:11
> To: rsyslog-users
> Subject: Re: [rsyslog] syslog-ng filters and destinations under
rsyslog
> 
> FYI, there was also a forum post:
> 
>
http://kb.monitorware.com/syslog-filters-and-destinations-under-rsyslog-
> t9858
> .html
> 
> I'd appreciate if any other responses also go to the forum (so that
> others,
> when searching, find the complete info).
> 
> Rainer
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Martin Mielke
> > Sent: Thursday, June 18, 2009 2:18 PM
> > To: rsyslog-users
> > Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
> >
> > Hi all,
> >
> > I need to find a way to either convert or import syslog-ng.conf
files
> > into rsyslog or learn the syntax to classify/store the logfiles
> > depending on the source.
> >
> > For example, taken from a syslog-ng file:
> > --
> > source s_network { udp(); tcp(); };
> >
> > filter f_systems {
> > host("1\.2\.3\.\d+");
> > };
> >
> > filter f_switches {
> > host("5\.6\.7\.8|5\.6\.7\.9");
> > };
> >
> > destination d_systems {file("/systems.log"); };
> > destination d_switches {file("/switches.log"); };
> >
> > log { source(s_network); filter(f_systems); destination(d_systems);
};
> > log { source(s_network); filter(f_switches);
destination(d_switches); };
> > --
> >
> > I have read the online documentation but I'm unable to find the
right
> > answer...
> >
> > Thanks!
> >
> > Martin
> >
> >
> > This email and any attachments are confidential, and may be legally
> privileged
> > and protected by copyright. If you are not the intended recipient
> > dissemination or copying of this email is prohibited. If you have
> received
> > this in error, please notify the sender by replying by email and
then
> delete
> > the email completely from your system.
> >
> > Any views or opinions are solely those of the sender.  This
> communication
> is
> > not intended to form a binding contract unless expressly indicated
to
> the
> > contrary and properly authorised. Any actions taken on the basis of
this
> email
> > are at the recipient's own risk.
> >
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> yslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Tue Jun 23 17:23:05 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 23 Jun 2009 17:23:05 +0200
Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
References: <0B1B9163138571439EAF804646F3F6AA0767973E@GIBSVWIN004X.partygaming.local><9B6E2A8877C38245BFB15CC491A11DA706F9A4@GRFEXC.intern.adiscon.com>
	<0B1B9163138571439EAF804646F3F6AA07878F4E@GIBSVWIN004X.partygaming.local>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA11@GRFEXC.intern.adiscon.com>

Thanks. Overlooked, you now have a reply ;)

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Martin Mielke
> Sent: Tuesday, June 23, 2009 5:08 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] syslog-ng filters and destinations under rsyslog
> 
> Sorry for the noise.
> 
> It seems that the post is pending on some reply ;-)
> 
> 
> Cheers,
> 
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> > Sent: 18 June 2009 15:11
> > To: rsyslog-users
> > Subject: Re: [rsyslog] syslog-ng filters and destinations under
> rsyslog
> >
> > FYI, there was also a forum post:
> >
> >
> http://kb.monitorware.com/syslog-filters-and-destinations-under-rsyslog-
> > t9858
> > .html
> >
> > I'd appreciate if any other responses also go to the forum (so that
> > others,
> > when searching, find the complete info).
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > bounces at lists.adiscon.com] On Behalf Of Martin Mielke
> > > Sent: Thursday, June 18, 2009 2:18 PM
> > > To: rsyslog-users
> > > Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
> > >
> > > Hi all,
> > >
> > > I need to find a way to either convert or import syslog-ng.conf
> files
> > > into rsyslog or learn the syntax to classify/store the logfiles
> > > depending on the source.
> > >
> > > For example, taken from a syslog-ng file:
> > > --
> > > source s_network { udp(); tcp(); };
> > >
> > > filter f_systems {
> > > host("1\.2\.3\.\d+");
> > > };
> > >
> > > filter f_switches {
> > > host("5\.6\.7\.8|5\.6\.7\.9");
> > > };
> > >
> > > destination d_systems {file("/systems.log"); };
> > > destination d_switches {file("/switches.log"); };
> > >
> > > log { source(s_network); filter(f_systems); destination(d_systems);
> };
> > > log { source(s_network); filter(f_switches);
> destination(d_switches); };
> > > --
> > >
> > > I have read the online documentation but I'm unable to find the
> right
> > > answer...
> > >
> > > Thanks!
> > >
> > > Martin
> > >
> > >
> > > This email and any attachments are confidential, and may be legally
> > privileged
> > > and protected by copyright. If you are not the intended recipient
> > > dissemination or copying of this email is prohibited. If you have
> > received
> > > this in error, please notify the sender by replying by email and
> then
> > delete
> > > the email completely from your system.
> > >
> > > Any views or opinions are solely those of the sender.  This
> > communication
> > is
> > > not intended to form a binding contract unless expressly indicated
> to
> > the
> > > contrary and properly authorised. Any actions taken on the basis of
> this
> > email
> > > are at the recipient's own risk.
> > >
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> > yslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Tue Jun 23 17:57:11 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 23 Jun 2009 17:57:11 +0200
Subject: [rsyslog] template for file input module in Rsyslog
References: <edc9be030906230459q4090383sc8427501d8a413a7@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706FA0D@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA17@GRFEXC.intern.adiscon.com>

I just checked, it is NOT supported.

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Tuesday, June 23, 2009 2:48 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] template for file input module in Rsyslog
> 
> I think this is not currently supported..
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Sachin Sawant
> > Sent: Tuesday, June 23, 2009 1:59 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: [rsyslog] template for file input module in Rsyslog
> >
> > Hi All,
> >
> >
> >
> > I would like to monitor PostgreSQL log files using file input module in
> > Rsyslog. PostgreSQL create log files based on date, so is it possible to
> > monitor file without using absolute path?
> >
> >
> >
> > Regards,
> >
> > * *
> >
> > *Sachin Sawant
> >
> > *
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From tbergfeld at hq.adiscon.com  Wed Jun 24 12:15:05 2009
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Wed, 24 Jun 2009 12:15:05 +0200
Subject: [rsyslog] rsyslog 4.3.2 (beta) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA31@GRFEXC.intern.adiscon.com>

Hi all,
The new rsyslog beta 4.3.2 has been released. This release consists of
several enhancements like the added generic network stream server for syslog,
capability to rebind the send socket of the UDP output section, a improved
testbench with a config switch and some bugfixes. This release is recommended
for all users of the beta branch.

Download:

http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-162.phtml

Changelog:

http://www.rsyslog.com/Article378.phtml

As always, feedback is appreciated.

Best regards,
Tom Bergfeld

Send this story to someone

--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From rgerhards at hq.adiscon.com  Wed Jun 24 14:31:04 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 24 Jun 2009 14:31:04 +0200
Subject: [rsyslog] segfault in v5-devel git branch
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>

Hi all,

I just wanted to let you know that the v5-devel git branch currently does not
work correctly when running with multiple worker threads. It will segfault
rather quickly. I am currently working on that and hope to resolve it soon.
Still, I thought it is better to tell you before someone begins to test with
that version.

Rainer


From rgerhards at hq.adiscon.com  Thu Jun 25 16:30:27 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 25 Jun 2009 16:30:27 +0200
Subject: [rsyslog] segfault in v5-devel git branch
References: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA5A@GRFEXC.intern.adiscon.com>

Hi all,

I just wanted to let you know that I hopefully have fixed this issue - at
least in my lab as well as under the threading debugging no problem shows up.
But of course, things like these require confirmation from the field.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Wednesday, June 24, 2009 2:31 PM
> To: rsyslog-users
> Subject: [rsyslog] segfault in v5-devel git branch
> 
> Hi all,
> 
> I just wanted to let you know that the v5-devel git branch currently does
not
> work correctly when running with multiple worker threads. It will segfault
> rather quickly. I am currently working on that and hope to resolve it soon.
> Still, I thought it is better to tell you before someone begins to test
with
> that version.
> 
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From kenneho.ndu at gmail.com  Fri Jun 26 14:30:09 2009
From: kenneho.ndu at gmail.com (Kenneth Holter)
Date: Fri, 26 Jun 2009 14:30:09 +0200
Subject: [rsyslog] rsyslogd:too many tcp sessions - dropping incoming request
Message-ID: <c25f25140906260530q7d102391k1154b1e5dca7a385@mail.gmail.com>

Hello list.


I'm running rsyslog 2.0.6, and have suddenly started to get this error
message:


rsyslogd:too many tcp sessions - dropping incoming request

I edited the /etc/sysconfig/rsyslog file and changed "-t200" to "-t200,500".
Is there any (performance) reasons or something that says that max number of
sessions should be 200 (default). If not, I'll stick with 500..


Regards,
Kenneth Holter


From rgerhards at hq.adiscon.com  Fri Jun 26 14:39:45 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 26 Jun 2009 14:39:45 +0200
Subject: [rsyslog] rsyslogd:too many tcp sessions - dropping incoming
	request
References: <c25f25140906260530q7d102391k1154b1e5dca7a385@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA68@GRFEXC.intern.adiscon.com>

No, I don't think so (in recent builds it is not). If I remember correctly,
the ",500" sets the accept queue but not the max # of sessions.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kenneth Holter
> Sent: Friday, June 26, 2009 2:30 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] rsyslogd:too many tcp sessions - dropping incoming
request
> 
> Hello list.
> 
> 
> I'm running rsyslog 2.0.6, and have suddenly started to get this error
> message:
> 
> 
> rsyslogd:too many tcp sessions - dropping incoming request
> 
> I edited the /etc/sysconfig/rsyslog file and changed "-t200" to
"-t200,500".
> Is there any (performance) reasons or something that says that max number
of
> sessions should be 200 (default). If not, I'll stick with 500..
> 
> 
> Regards,
> Kenneth Holter
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From mbiebl at gmail.com  Fri Jun 26 14:44:29 2009
From: mbiebl at gmail.com (Michael Biebl)
Date: Fri, 26 Jun 2009 14:44:29 +0200
Subject: [rsyslog] segfault in v5-devel git branch
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>
Message-ID: <aa76cdac0906260544l24ddecexb7c3b35436b05875@mail.gmail.com>

2009/6/24 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Hi all,
>
> I just wanted to let you know that the v5-devel git branch currently does not
> work correctly when running with multiple worker threads. It will segfault
> rather quickly. I am currently working on that and hope to resolve it soon.
> Still, I thought it is better to tell you before someone begins to test with
> that version.

I'm not even able to compile the v5-devel branch :-/

  CC    stringbuf.o
In file included from zlibw.h:28,
                 from stream.h:72,
                 from obj.h:50,
                 from rsyslog.h:441,
                 from stringbuf.c:39:
/usr/include/zlib.h:1366: error: expected ?=?, ?,?, ?;?, ?asm? or
?__attribute__? before ?gzseek64?
/usr/include/zlib.h:1367: error: expected ?=?, ?,?, ?;?, ?asm? or
?__attribute__? before ?gztell64?
/usr/include/zlib.h:1368: error: expected declaration specifiers or
?...? before ?off64_t?
/usr/include/zlib.h:1369: error: expected declaration specifiers or
?...? before ?off64_t?
make[2]: *** [librsyslog_la-stringbuf.lo] Fehler 1
make[1]: *** [all-recursive] Fehler 1
make: *** [all] Fehler 2
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Fri Jun 26 15:19:28 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 26 Jun 2009 15:19:28 +0200
Subject: [rsyslog] segfault in v5-devel git branch
References: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>
	<aa76cdac0906260544l24ddecexb7c3b35436b05875@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA6A@GRFEXC.intern.adiscon.com>

Can you please let me know what the offending line in your zlib.h include
file is? I guess there is a name conflict in one of the types used.

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Friday, June 26, 2009 2:44 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] segfault in v5-devel git branch
> 
> 2009/6/24 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Hi all,
> >
> > I just wanted to let you know that the v5-devel git branch currently does
> not
> > work correctly when running with multiple worker threads. It will
segfault
> > rather quickly. I am currently working on that and hope to resolve it
soon.
> > Still, I thought it is better to tell you before someone begins to test
with
> > that version.
> 
> I'm not even able to compile the v5-devel branch :-/
> 
>   CC    stringbuf.o
> In file included from zlibw.h:28,
>                  from stream.h:72,
>                  from obj.h:50,
>                  from rsyslog.h:441,
>                  from stringbuf.c:39:
> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or
> '__attribute__' before 'gzseek64'
> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or
> '__attribute__' before 'gztell64'
> /usr/include/zlib.h:1368: error: expected declaration specifiers or
> '...' before 'off64_t'
> /usr/include/zlib.h:1369: error: expected declaration specifiers or
> '...' before 'off64_t'
> make[2]: *** [librsyslog_la-stringbuf.lo] Fehler 1
> make[1]: *** [all-recursive] Fehler 1
> make: *** [all] Fehler 2
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From mbiebl at gmail.com  Fri Jun 26 15:36:18 2009
From: mbiebl at gmail.com (Michael Biebl)
Date: Fri, 26 Jun 2009 15:36:18 +0200
Subject: [rsyslog] segfault in v5-devel git branch
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA6A@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>
	<aa76cdac0906260544l24ddecexb7c3b35436b05875@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706FA6A@GRFEXC.intern.adiscon.com>
Message-ID: <aa76cdac0906260636ia9a3cc2gba9a4bfc6fecdec4@mail.gmail.com>

#ifdef _LARGEFILE64_SOURCE
   ZEXTERN gzFile ZEXPORT gzopen64 OF((const char *, const char *));
   ZEXTERN off64_t ZEXPORT gzseek64 OF((gzFile, off64_t, int));
   ZEXTERN off64_t ZEXPORT gztell64 OF((gzFile));
   ZEXTERN uLong ZEXPORT adler32_combine64 OF((uLong, uLong, off64_t));
   ZEXTERN uLong ZEXPORT crc32_combine64 OF((uLong, uLong, off64_t));
#endif


2009/6/26 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Can you please let me know what the offending line in your zlib.h include
> file is? I guess there is a name conflict in one of the types used.
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
>> Sent: Friday, June 26, 2009 2:44 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] segfault in v5-devel git branch
>>
>> 2009/6/24 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> > Hi all,
>> >
>> > I just wanted to let you know that the v5-devel git branch currently does
>> not
>> > work correctly when running with multiple worker threads. It will
> segfault
>> > rather quickly. I am currently working on that and hope to resolve it
> soon.
>> > Still, I thought it is better to tell you before someone begins to test
> with
>> > that version.
>>
>> I'm not even able to compile the v5-devel branch :-/
>>
>> ? CC ? ?stringbuf.o
>> In file included from zlibw.h:28,
>> ? ? ? ? ? ? ? ? ?from stream.h:72,
>> ? ? ? ? ? ? ? ? ?from obj.h:50,
>> ? ? ? ? ? ? ? ? ?from rsyslog.h:441,
>> ? ? ? ? ? ? ? ? ?from stringbuf.c:39:
>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or
>> '__attribute__' before 'gzseek64'
>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or
>> '__attribute__' before 'gztell64'
>> /usr/include/zlib.h:1368: error: expected declaration specifiers or
>> '...' before 'off64_t'
>> /usr/include/zlib.h:1369: error: expected declaration specifiers or
>> '...' before 'off64_t'
>> make[2]: *** [librsyslog_la-stringbuf.lo] Fehler 1
>> make[1]: *** [all-recursive] Fehler 1
>> make: *** [all] Fehler 2
>> --
>> Why is it that all of the instruments seeking intelligent life in the
>> universe are pointed away from Earth?
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>



-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From iamsayan at gmail.com  Mon Jun 29 18:25:18 2009
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Mon, 29 Jun 2009 12:25:18 -0400
Subject: [rsyslog] How to test rsyslog performance
Message-ID: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com>

Hi,
I saw the announcement of the 4.2.0 stable version of rsyslog. I was
wondering if there is a way I can test the performance of rsyslog with my
own rsyslog.conf.
The way I am trying to use rsyslog is on a local system, which has following
rules

logging different facilities to different files with different formats.
forwarding certain facilities to an external server.
receive messages from one(and only one) other machine over udp.

Are there any tool/scripts available which I can use to perform the tests
myself?
Basically what I am thinking of is running a script which writes is bunch of
messages to syslog, but then how do I make sure all those messages were
actually logged and not lost? It may be very painful to look through the
entire log file 1 message at a time :)
Is there a stat/counter based api which shows me how many messages were
logged?

Regards,
Sayan


From david at lang.hm  Mon Jun 29 18:46:53 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 29 Jun 2009 09:46:53 -0700 (PDT)
Subject: [rsyslog] How to test rsyslog performance
In-Reply-To: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com>
References: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906290943480.9159@asgard>

On Mon, 29 Jun 2009, Sayan Chowdhury wrote:

> Hi,
> I saw the announcement of the 4.2.0 stable version of rsyslog. I was
> wondering if there is a way I can test the performance of rsyslog with my
> own rsyslog.conf.
> The way I am trying to use rsyslog is on a local system, which has following
> rules
>
> logging different facilities to different files with different formats.
> forwarding certain facilities to an external server.
> receive messages from one(and only one) other machine over udp.
>
> Are there any tool/scripts available which I can use to perform the tests
> myself?
> Basically what I am thinking of is running a script which writes is bunch of
> messages to syslog, but then how do I make sure all those messages were
> actually logged and not lost? It may be very painful to look through the
> entire log file 1 message at a time :)
> Is there a stat/counter based api which shows me how many messages were
> logged?

what I do is to send a known quantity of messages and include a identifier 
string in the message ('rsyslog-test-message' for example), and then grep 
the resulting files to make sure that the right number of messages arrive.

to send messages to rsyslog I have used tcpreplay to replay UDP messages. 
I first setup a system to send the messages via UDP, start tcpdump, send 
the messages to the local box (which then sends them via UDP to a remote 
server), then I stop tcpdump and can use the resulting capture file with 
tcpreplay to replay the messages at different speeds.

David Lang


From iamsayan at gmail.com  Mon Jun 29 19:55:15 2009
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Mon, 29 Jun 2009 13:55:15 -0400
Subject: [rsyslog] How to test rsyslog performance
In-Reply-To: <alpine.DEB.1.10.0906290943480.9159@asgard>
References: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com>
	<alpine.DEB.1.10.0906290943480.9159@asgard>
Message-ID: <87a1ae540906291055n7625b369naf02a1576e8ee187@mail.gmail.com>

Hello David,
Thanks a bunch ..
I will try this, I was also thinking in similar lines, in the sense that I
will log 1 line of message, set the file size to be huge, and then count the
number of lines on the file.
I just read the numbers on Rainer's blog, and I think I will need the
tcprelay approach as well, there is no way I can generate so much load with
a shell script.
Thanks for the tip.
Regadrs,
Sayan


On Mon, Jun 29, 2009 at 12:46 PM, <david at lang.hm> wrote:

> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
>
> > Hi,
> > I saw the announcement of the 4.2.0 stable version of rsyslog. I was
> > wondering if there is a way I can test the performance of rsyslog with my
> > own rsyslog.conf.
> > The way I am trying to use rsyslog is on a local system, which has
> following
> > rules
> >
> > logging different facilities to different files with different formats.
> > forwarding certain facilities to an external server.
> > receive messages from one(and only one) other machine over udp.
> >
> > Are there any tool/scripts available which I can use to perform the tests
> > myself?
> > Basically what I am thinking of is running a script which writes is bunch
> of
> > messages to syslog, but then how do I make sure all those messages were
> > actually logged and not lost? It may be very painful to look through the
> > entire log file 1 message at a time :)
> > Is there a stat/counter based api which shows me how many messages were
> > logged?
>
> what I do is to send a known quantity of messages and include a identifier
> string in the message ('rsyslog-test-message' for example), and then grep
> the resulting files to make sure that the right number of messages arrive.
>
> to send messages to rsyslog I have used tcpreplay to replay UDP messages.
> I first setup a system to send the messages via UDP, start tcpdump, send
> the messages to the local box (which then sends them via UDP to a remote
> server), then I stop tcpdump and can use the resulting capture file with
> tcpreplay to replay the messages at different speeds.
>
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From david at lang.hm  Mon Jun 29 20:03:33 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 29 Jun 2009 11:03:33 -0700 (PDT)
Subject: [rsyslog] How to test rsyslog performance
In-Reply-To: <87a1ae540906291055n7625b369naf02a1576e8ee187@mail.gmail.com>
References: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com>
	<alpine.DEB.1.10.0906290943480.9159@asgard>
	<87a1ae540906291055n7625b369naf02a1576e8ee187@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906291100010.9159@asgard>

On Mon, 29 Jun 2009, Sayan Chowdhury wrote:

> Hello David,
> Thanks a bunch ..
> I will try this, I was also thinking in similar lines, in the sense that I
> will log 1 line of message, set the file size to be huge, and then count the
> number of lines on the file.
> I just read the numbers on Rainer's blog, and I think I will need the
> tcprelay approach as well, there is no way I can generate so much load with
> a shell script.
> Thanks for the tip.

with an earlier version of the 4.x series I was seeing numbers like the 
following

with 256 byte messages

recieving via UDP and storing in the queue, >300K/sec (wire speed, gig-E)

recieving via UDP and writing to disk, ~78K/sec

recieving via UDP, writing to disk and sending to another machine, ~30k/sec

since then there has been a lot of work to optimize the queue functions 
that appeared to be the bottleneck in my testing, and so I am very 
interested in seeing the new results (I expect to be testing shortly, but 
will be out much of this week, one of my 'hobbies' is doing professional 
fireworks, and so I will be busy preparing for the July 4th shows)

David Lang


> Regadrs,
> Sayan
>
>
> On Mon, Jun 29, 2009 at 12:46 PM, <david at lang.hm> wrote:
>
>> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
>>
>>> Hi,
>>> I saw the announcement of the 4.2.0 stable version of rsyslog. I was
>>> wondering if there is a way I can test the performance of rsyslog with my
>>> own rsyslog.conf.
>>> The way I am trying to use rsyslog is on a local system, which has
>> following
>>> rules
>>>
>>> logging different facilities to different files with different formats.
>>> forwarding certain facilities to an external server.
>>> receive messages from one(and only one) other machine over udp.
>>>
>>> Are there any tool/scripts available which I can use to perform the tests
>>> myself?
>>> Basically what I am thinking of is running a script which writes is bunch
>> of
>>> messages to syslog, but then how do I make sure all those messages were
>>> actually logged and not lost? It may be very painful to look through the
>>> entire log file 1 message at a time :)
>>> Is there a stat/counter based api which shows me how many messages were
>>> logged?
>>
>> what I do is to send a known quantity of messages and include a identifier
>> string in the message ('rsyslog-test-message' for example), and then grep
>> the resulting files to make sure that the right number of messages arrive.
>>
>> to send messages to rsyslog I have used tcpreplay to replay UDP messages.
>> I first setup a system to send the messages via UDP, start tcpdump, send
>> the messages to the local box (which then sends them via UDP to a remote
>> server), then I stop tcpdump and can use the resulting capture file with
>> tcpreplay to replay the messages at different speeds.
>>
>> David Lang
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Mon Jun 29 21:33:54 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 29 Jun 2009 21:33:54 +0200
Subject: [rsyslog] How to test rsyslog performance
References: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com><alpine.DEB.1.10.0906290943480.9159@asgard><87a1ae540906291055n7625b369naf02a1576e8ee187@mail.gmail.com>
	<alpine.DEB.1.10.0906291100010.9159@asgard>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA8A@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Monday, June 29, 2009 8:04 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] How to test rsyslog performance
> 
> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
> 
> > Hello David,
> > Thanks a bunch ..
> > I will try this, I was also thinking in similar lines, in 
> the sense that I
> > will log 1 line of message, set the file size to be huge, 
> and then count the
> > number of lines on the file.
> > I just read the numbers on Rainer's blog, and I think I 
> will need the
> > tcprelay approach as well, there is no way I can generate 
> so much load with
> > a shell script.
> > Thanks for the tip.
> 
> with an earlier version of the 4.x series I was seeing 
> numbers like the 
> following
> 
> with 256 byte messages
> 
> recieving via UDP and storing in the queue, >300K/sec (wire 
> speed, gig-E)
> 
> recieving via UDP and writing to disk, ~78K/sec
> 
> recieving via UDP, writing to disk and sending to another 
> machine, ~30k/sec
> 
> since then there has been a lot of work to optimize the queue 
> functions 
> that appeared to be the bottleneck in my testing,

In the mean time, I have identified at least one more bottleneck, and this
was alloc/free. These come with implicit synchroniation primitives (to
protect the heap) and it turned out that they cause quite some performance
loss as soon as we have enough threads running concurrently. I have removed
much of these alloc calls during the past two weeks, and the results - at
least in my non-high-end lab - are quite pleasant. This work is still not
100% finished, but I think I have reched the mark where further reduction of
alloc does not case any notable performance increase. I'll still continue to
finish up some things (so that the code base is consistent), but I don't
expect much performance increase from that. 

All in all, the "new v4" engine should perform around twice as fast as the
"old v4" engine (depending a bit on configuration). The v5 engine offers more
advanced optimizations and will receive even more of them. At this time, the
v5 engine in my lab (with TCP!) is at least twice as fast as the "new v4"
engine, probably a bit less with UDP.

There is still ample of tuning potential. Two things that immediately come to
my mind is that the inputs still use select() [originally used for
portability reasons] and the inputs themselves are single-threaded. For v5 I
can also envision a new malloc subsytem, that will do 1/1024th malloc per
message (right now, we do around 5 mallocs per message, tomorrow evening
probably only 3).

If someone is looking into performance, one should definitely look into the
v5 engine. This version is experimental, but if nobody tries it out, it can't
improve (everything I could do in my lab I have done, but I know from past
experience that the real world and high-end hardware is a different story).
But even the test with the threading debugger went quite well. I'll release a
new v5 shortly, and that one is definitely worth a try (if you need it
urgent, just let me know).

> and so I am very 
> interested in seeing the new results (I expect to be testing 
> shortly, but 
> will be out much of this week, one of my 'hobbies' is doing 
> professional 
> fireworks, and so I will be busy preparing for the July 4th shows)

Cool - wish I were there ;)

Rainer 

> David Lang
> 
> 
> > Regadrs,
> > Sayan
> >
> >
> > On Mon, Jun 29, 2009 at 12:46 PM, <david at lang.hm> wrote:
> >
> >> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
> >>
> >>> Hi,
> >>> I saw the announcement of the 4.2.0 stable version of 
> rsyslog. I was
> >>> wondering if there is a way I can test the performance of 
> rsyslog with my
> >>> own rsyslog.conf.
> >>> The way I am trying to use rsyslog is on a local system, which has
> >> following
> >>> rules
> >>>
> >>> logging different facilities to different files with 
> different formats.
> >>> forwarding certain facilities to an external server.
> >>> receive messages from one(and only one) other machine over udp.
> >>>
> >>> Are there any tool/scripts available which I can use to 
> perform the tests
> >>> myself?
> >>> Basically what I am thinking of is running a script which 
> writes is bunch
> >> of
> >>> messages to syslog, but then how do I make sure all those 
> messages were
> >>> actually logged and not lost? It may be very painful to 
> look through the
> >>> entire log file 1 message at a time :)
> >>> Is there a stat/counter based api which shows me how many 
> messages were
> >>> logged?
> >>
> >> what I do is to send a known quantity of messages and 
> include a identifier
> >> string in the message ('rsyslog-test-message' for 
> example), and then grep
> >> the resulting files to make sure that the right number of 
> messages arrive.
> >>
> >> to send messages to rsyslog I have used tcpreplay to 
> replay UDP messages.
> >> I first setup a system to send the messages via UDP, start 
> tcpdump, send
> >> the messages to the local box (which then sends them via 
> UDP to a remote
> >> server), then I stop tcpdump and can use the resulting 
> capture file with
> >> tcpreplay to replay the messages at different speeds.
> >>
> >> David Lang
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 


From david at lang.hm  Mon Jun 29 22:18:50 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 29 Jun 2009 13:18:50 -0700 (PDT)
Subject: [rsyslog] How to test rsyslog performance
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA8A@GRFEXC.intern.adiscon.com>
References: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com><alpine.DEB.1.10.0906290943480.9159@asgard><87a1ae540906291055n7625b369naf02a1576e8ee187@mail.gmail.com>
	<alpine.DEB.1.10.0906291100010.9159@asgard>
	<9B6E2A8877C38245BFB15CC491A11DA706FA8A@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906291317300.9159@asgard>

On Mon, 29 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
>>
>>> Hello David,
>>> Thanks a bunch ..
>>> I will try this, I was also thinking in similar lines, in
>> the sense that I
>>> will log 1 line of message, set the file size to be huge,
>> and then count the
>>> number of lines on the file.
>>> I just read the numbers on Rainer's blog, and I think I
>> will need the
>>> tcprelay approach as well, there is no way I can generate
>> so much load with
>>> a shell script.
>>> Thanks for the tip.
>>
>> with an earlier version of the 4.x series I was seeing
>> numbers like the
>> following
>>
>> with 256 byte messages
>>
>> recieving via UDP and storing in the queue, >300K/sec (wire
>> speed, gig-E)
>>
>> recieving via UDP and writing to disk, ~78K/sec
>>
>> recieving via UDP, writing to disk and sending to another
>> machine, ~30k/sec
>>
>> since then there has been a lot of work to optimize the queue
>> functions
>> that appeared to be the bottleneck in my testing,
>
> In the mean time, I have identified at least one more bottleneck, and this
> was alloc/free. These come with implicit synchroniation primitives (to
> protect the heap) and it turned out that they cause quite some performance
> loss as soon as we have enough threads running concurrently. I have removed
> much of these alloc calls during the past two weeks, and the results - at
> least in my non-high-end lab - are quite pleasant. This work is still not
> 100% finished, but I think I have reched the mark where further reduction of
> alloc does not case any notable performance increase. I'll still continue to
> finish up some things (so that the code base is consistent), but I don't
> expect much performance increase from that.
>
> All in all, the "new v4" engine should perform around twice as fast as the
> "old v4" engine (depending a bit on configuration). The v5 engine offers more
> advanced optimizations and will receive even more of them. At this time, the
> v5 engine in my lab (with TCP!) is at least twice as fast as the "new v4"
> engine, probably a bit less with UDP.
>
> There is still ample of tuning potential. Two things that immediately come to
> my mind is that the inputs still use select() [originally used for
> portability reasons] and the inputs themselves are single-threaded. For v5 I
> can also envision a new malloc subsytem, that will do 1/1024th malloc per
> message (right now, we do around 5 mallocs per message, tomorrow evening
> probably only 3).
>
> If someone is looking into performance, one should definitely look into the
> v5 engine. This version is experimental, but if nobody tries it out, it can't
> improve (everything I could do in my lab I have done, but I know from past
> experience that the real world and high-end hardware is a different story).
> But even the test with the threading debugger went quite well. I'll release a
> new v5 shortly, and that one is definitely worth a try (if you need it
> urgent, just let me know).

I tried to compile the v5 branch over the weekend, but got a compile 
error, so I just moved to the 4.2.0 release. on the ultra-reliable testbed 
it's getting ~8500/sec.

David Lang

>> and so I am very
>> interested in seeing the new results (I expect to be testing
>> shortly, but
>> will be out much of this week, one of my 'hobbies' is doing
>> professional
>> fireworks, and so I will be busy preparing for the July 4th shows)
>
> Cool - wish I were there ;)
>
> Rainer
>
>> David Lang
>>
>>
>>> Regadrs,
>>> Sayan
>>>
>>>
>>> On Mon, Jun 29, 2009 at 12:46 PM, <david at lang.hm> wrote:
>>>
>>>> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
>>>>
>>>>> Hi,
>>>>> I saw the announcement of the 4.2.0 stable version of
>> rsyslog. I was
>>>>> wondering if there is a way I can test the performance of
>> rsyslog with my
>>>>> own rsyslog.conf.
>>>>> The way I am trying to use rsyslog is on a local system, which has
>>>> following
>>>>> rules
>>>>>
>>>>> logging different facilities to different files with
>> different formats.
>>>>> forwarding certain facilities to an external server.
>>>>> receive messages from one(and only one) other machine over udp.
>>>>>
>>>>> Are there any tool/scripts available which I can use to
>> perform the tests
>>>>> myself?
>>>>> Basically what I am thinking of is running a script which
>> writes is bunch
>>>> of
>>>>> messages to syslog, but then how do I make sure all those
>> messages were
>>>>> actually logged and not lost? It may be very painful to
>> look through the
>>>>> entire log file 1 message at a time :)
>>>>> Is there a stat/counter based api which shows me how many
>> messages were
>>>>> logged?
>>>>
>>>> what I do is to send a known quantity of messages and
>> include a identifier
>>>> string in the message ('rsyslog-test-message' for
>> example), and then grep
>>>> the resulting files to make sure that the right number of
>> messages arrive.
>>>>
>>>> to send messages to rsyslog I have used tcpreplay to
>> replay UDP messages.
>>>> I first setup a system to send the messages via UDP, start
>> tcpdump, send
>>>> the messages to the local box (which then sends them via
>> UDP to a remote
>>>> server), then I stop tcpdump and can use the resulting
>> capture file with
>>>> tcpreplay to replay the messages at different speeds.
>>>>
>>>> David Lang
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From sfzhi at yahoo.com  Tue Jun 30 11:11:14 2009
From: sfzhi at yahoo.com (Sergei Zhirikov)
Date: Tue, 30 Jun 2009 11:11:14 +0200
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
Message-ID: <h2ckri$16j$1@ger.gmane.org>

Rainer Gerhards wrote:
> Hi all,
> 
> The most prominent use case is to size-limit files. $OutChannel provides the
> ability to execute a command when a certain file size is reached. The command
> may then rotate the file. After command execution, omfile checks the file
> size again. If it still is above the configured limit, this file will be
> disabled (no longer be written to until restart).
> 
> If you use this for size-limiting, please let me know how you do it. Please
> also let me know if you would like to see things handled differently.
> 
> Looking forward to your feedback.
> 
> Thanks,
> Rainer
> 

Hi Rainer,

I do use $OutChannel for size-limiting and it would be a disappointment 
to me if that feature disappeared. As a matter of fact, it was the main 
reason why I chose to use rsyslog in the first place.

I find the "classic" way of using cron+logrotate+SIGHUP quite ugly and 
in some cases unacceptable. I have a small home gateway with read-only 
main file system, so all the log files are written into a tmpfs mount. 
It happened a couple of times that some misconfigured or malfunctioning 
software produced a lot of logging and filled in most of the available 
RAM before the next scheduled run of logrotate. In this situation the 
size limiting feature of rsyslog is exactly what I need in order to be 
able to avoid such this kind of problems.

Kind regards,
Sergei.



From david at lang.hm  Tue Jun 30 12:35:58 2009
From: david at lang.hm (david at lang.hm)
Date: Tue, 30 Jun 2009 03:35:58 -0700 (PDT)
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <h2ckri$16j$1@ger.gmane.org>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
	<h2ckri$16j$1@ger.gmane.org>
Message-ID: <alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>

On Tue, 30 Jun 2009, Sergei Zhirikov wrote:

> Rainer Gerhards wrote:
>> Hi all,
>>
>> The most prominent use case is to size-limit files. $OutChannel provides the
>> ability to execute a command when a certain file size is reached. The command
>> may then rotate the file. After command execution, omfile checks the file
>> size again. If it still is above the configured limit, this file will be
>> disabled (no longer be written to until restart).
>>
>> If you use this for size-limiting, please let me know how you do it. Please
>> also let me know if you would like to see things handled differently.
>>
>> Looking forward to your feedback.
>>
>> Thanks,
>> Rainer
>>
>
> Hi Rainer,
>
> I do use $OutChannel for size-limiting and it would be a disappointment
> to me if that feature disappeared. As a matter of fact, it was the main
> reason why I chose to use rsyslog in the first place.
>
> I find the "classic" way of using cron+logrotate+SIGHUP quite ugly and
> in some cases unacceptable. I have a small home gateway with read-only
> main file system, so all the log files are written into a tmpfs mount.
> It happened a couple of times that some misconfigured or malfunctioning
> software produced a lot of logging and filled in most of the available
> RAM before the next scheduled run of logrotate. In this situation the
> size limiting feature of rsyslog is exactly what I need in order to be
> able to avoid such this kind of problems.

this is a very legitimate use. I suspect the biggest issue is how to give 
you this capability without slowing down the people who don't use it.

I wonder if rainerscript could be extended to where it could be used to do 
this?

something like:

for each message logged, add the length of the message to a counter, and 
have an if statement that calls the external program to roll the log and 
reset the counter.

David Lang


From rgerhards at hq.adiscon.com  Tue Jun 30 12:43:14 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 30 Jun 2009 12:43:14 +0200
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com><h2ckri$16j$1@ger.gmane.org>
	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, June 30, 2009 12:36 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] omfile redesign / $OutChannel will go away!
> 
> On Tue, 30 Jun 2009, Sergei Zhirikov wrote:
> 
> > Rainer Gerhards wrote:
> >> Hi all,
> >>
> >> The most prominent use case is to size-limit files. $OutChannel provides
> the
> >> ability to execute a command when a certain file size is reached. The
> command
> >> may then rotate the file. After command execution, omfile checks the
file
> >> size again. If it still is above the configured limit, this file will be
> >> disabled (no longer be written to until restart).
> >>
> >> If you use this for size-limiting, please let me know how you do it.
Please
> >> also let me know if you would like to see things handled differently.
> >>
> >> Looking forward to your feedback.
> >>
> >> Thanks,
> >> Rainer
> >>
> >
> > Hi Rainer,
> >
> > I do use $OutChannel for size-limiting and it would be a disappointment
> > to me if that feature disappeared. As a matter of fact, it was the main
> > reason why I chose to use rsyslog in the first place.
> >
> > I find the "classic" way of using cron+logrotate+SIGHUP quite ugly and
> > in some cases unacceptable. I have a small home gateway with read-only
> > main file system, so all the log files are written into a tmpfs mount.
> > It happened a couple of times that some misconfigured or malfunctioning
> > software produced a lot of logging and filled in most of the available
> > RAM before the next scheduled run of logrotate. In this situation the
> > size limiting feature of rsyslog is exactly what I need in order to be
> > able to avoid such this kind of problems.
> 
> this is a very legitimate use. I suspect the biggest issue is how to give
> you this capability without slowing down the people who don't use it.
> 
> I wonder if rainerscript could be extended to where it could be used to do
> this?
> 
> something like:
> 
> for each message logged, add the length of the message to a counter, and
> have an if statement that calls the external program to roll the log and
> reset the counter.

I probably should have phrased the initial question less agressive, but than
I'd probably not received any feedback. From the feedback I received so far,
there seem to be two use cases:

a) circular logs
b) stop logging when file hits max size (was that really what this post
described?)

In any case, these two seem to be the only things that are done with
outchannel. So when $outchannel goes away (and it will at some time), the
"regular" file writer will have options to support these use cases - probably
more elegant than before. For example, for case a), circular logs, I do not
need to involve an external process, so I can remove that feature without
harm, as long as I replace it with something suitable (and think that omfile
now uses the stream writer, which *already does* circular "logging" for disk
queues).

So functionality in actual need will not go away, as long as people tell what
they need.

If I look at the b) case, I suspect that this is not even the intent of the
original poster. I think he would be much better served with a circular log
that records as many log records as fit into a finite, operator-specified log
file size. Or am I wrong here? (If nobody tells me I am wrong, I'll probably
remove the b) use case in the long term!).

Rainer



From david at lang.hm  Tue Jun 30 13:03:54 2009
From: david at lang.hm (david at lang.hm)
Date: Tue, 30 Jun 2009 04:03:54 -0700 (PDT)
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com><h2ckri$16j$1@ger.gmane.org>
	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906300354370.11336@asgard.lang.hm>

On Tue, 30 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Tue, 30 Jun 2009, Sergei Zhirikov wrote:
>>
>>> Rainer Gerhards wrote:
>>>> Hi all,
>>>>
>>>> The most prominent use case is to size-limit files. $OutChannel provides
>> the
>>>> ability to execute a command when a certain file size is reached. The
>> command
>>>> may then rotate the file. After command execution, omfile checks the
> file
>>>> size again. If it still is above the configured limit, this file will be
>>>> disabled (no longer be written to until restart).
>>>>
>>>> If you use this for size-limiting, please let me know how you do it.
> Please
>>>> also let me know if you would like to see things handled differently.
>>>>
>>>> Looking forward to your feedback.
>>>>
>>>> Thanks,
>>>> Rainer
>>>>
>>>
>>> Hi Rainer,
>>>
>>> I do use $OutChannel for size-limiting and it would be a disappointment
>>> to me if that feature disappeared. As a matter of fact, it was the main
>>> reason why I chose to use rsyslog in the first place.
>>>
>>> I find the "classic" way of using cron+logrotate+SIGHUP quite ugly and
>>> in some cases unacceptable. I have a small home gateway with read-only
>>> main file system, so all the log files are written into a tmpfs mount.
>>> It happened a couple of times that some misconfigured or malfunctioning
>>> software produced a lot of logging and filled in most of the available
>>> RAM before the next scheduled run of logrotate. In this situation the
>>> size limiting feature of rsyslog is exactly what I need in order to be
>>> able to avoid such this kind of problems.
>>
>> this is a very legitimate use. I suspect the biggest issue is how to give
>> you this capability without slowing down the people who don't use it.
>>
>> I wonder if rainerscript could be extended to where it could be used to do
>> this?
>>
>> something like:
>>
>> for each message logged, add the length of the message to a counter, and
>> have an if statement that calls the external program to roll the log and
>> reset the counter.
>
> I probably should have phrased the initial question less agressive, but than
> I'd probably not received any feedback. From the feedback I received so far,
> there seem to be two use cases:
>
> a) circular logs
> b) stop logging when file hits max size (was that really what this post
> described?)

no, this is a third case
c) limit the size of the logfile that is being added to, and when it hits 
this limit roll it off to permanent storage (at which point the file 
becomes read-only)

this is a need at both the low-end and the high-end

for this poster, he needs it at the low end as his 'working' files are in 
ram (his /var/log/messages), but then he rolls it off to network storage.

at this high end you can have large volumes of logs arriving, and want to 
have ver fast (but expensive) storage to handle the high file modification 
rate, but want to make sure that you don't run out of space there by 
rolling the logs off to slower, but larger disks so that they don't 
fillthe fast drive and cause you to loose logs.

> In any case, these two seem to be the only things that are done with
> outchannel. So when $outchannel goes away (and it will at some time), the
> "regular" file writer will have options to support these use cases - probably
> more elegant than before. For example, for case a), circular logs, I do not
> need to involve an external process, so I can remove that feature without
> harm, as long as I replace it with something suitable (and think that omfile
> now uses the stream writer, which *already does* circular "logging" for disk
> queues).

that sounds reasonable

> So functionality in actual need will not go away, as long as people tell what
> they need.
>
> If I look at the b) case, I suspect that this is not even the intent of the
> original poster. I think he would be much better served with a circular log
> that records as many log records as fit into a finite, operator-specified log
> file size. Or am I wrong here? (If nobody tells me I am wrong, I'll probably
> remove the b) use case in the long term!).

no, it's not that total space for logs is limited to a fixed amount, it's 
that you have two spaces for logs, a small fast space and a large slow 
space, and you need to not run out of the fast space by moving logs to the 
slow space 'soon enough'


from a security point of view there are two arguments about option b

on the one hand, if a bad guy can run you out of space he can then do new 
things that aren't logged.

on the other hand, if you have a circular buffer, the bad buy can generate 
enough logs to cause you to overwrite the evidence of him breaking in to 
your system

there are places for both.

David Lang


From sfzhi at yahoo.com  Tue Jun 30 15:24:53 2009
From: sfzhi at yahoo.com (Sergei Zhirikov)
Date: Tue, 30 Jun 2009 15:24:53 +0200
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com><h2ckri$16j$1@ger.gmane.org>	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
Message-ID: <h2d3n6$cp5$1@ger.gmane.org>

> From the feedback I received so far, there seem to be two use cases:
> 
> a) circular logs
> b) stop logging when file hits max size (was that really what this post
> described?)
> 
> If I look at the b) case, I suspect that this is not even the intent of the
> original poster. I think he would be much better served with a circular log
> that records as many log records as fit into a finite, operator-specified log
> file size. Or am I wrong here? (If nobody tells me I am wrong, I'll probably
> remove the b) use case in the long term!).
> 
> Rainer
> 

In the case of log flood neither of the two options is perfect. The 
difference is only in what you lose, the oldest or the newest log 
entries. What I really like about the current rsyslog functionality is 
the possibility to call an external process to deal with the oversized 
logs. Stopping logging I consider only a safeguard in case that external 
process fails to do its job for whatever reason.

Having an external process makes the whole setup a lot more flexible, 
because it gives almost unlimited opportunities, for example:

   * gzip the older logs rather than delete them. In case of log flood 
gzipping is usually very efficient, because the same message is repeated 
in the log multiple times, so there is no need to delete anything.
   * filter the older logs based on conditions that involve analyzing 
more than one log entry at the same time. This can be useful, for 
example, do detect break-in attempts if desired.
   * measure the average speed of logging and send an email or generate 
another kind of alert when the logs are growing unusually fast.
   * make a decision whether to gzip the older logs, to move them to 
another location (e.g. a network storage, if available), or to delete 
them based on conditions outside of control of rsyslog (e.g. available 
system resources, availability of alternative storage, etc.)

The list can go on infinitely. I believe the possibility to call an 
external process to handle log rotation gives the flexibility that can 
not be provided within rsyslog itself, because there are so many 
different things one might want to do. And I would very much like to see 
this functionality stay in one or another form. Hope I'm not the only 
one who thinks that way.

Kind regards,
Sergei.



From epiphani at gmail.com  Tue Jun 30 15:38:11 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Tue, 30 Jun 2009 09:38:11 -0400
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <h2d3n6$cp5$1@ger.gmane.org>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
	<h2ckri$16j$1@ger.gmane.org>
	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
	<h2d3n6$cp5$1@ger.gmane.org>
Message-ID: <e7ca40f70906300638n2b7dd90eo49429daec8caddc2@mail.gmail.com>

On Tue, Jun 30, 2009 at 9:24 AM, Sergei Zhirikov<sfzhi at yahoo.com> wrote:
>
> ? * gzip the older logs rather than delete them. In case of log flood
> gzipping is usually very efficient, because the same message is repeated
> in the log multiple times, so there is no need to delete anything.

One thing worth noting is that the new omfile branch contains changes
that will allow gzipping of files automatically out of rsyslogd, which
removes the requirement to do it after writing to disk.  :)

-Aaron


From aoz.syn at gmail.com  Tue Jun 30 16:44:55 2009
From: aoz.syn at gmail.com (RB)
Date: Tue, 30 Jun 2009 08:44:55 -0600
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
	<h2ckri$16j$1@ger.gmane.org>
	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
Message-ID: <4255c2570906300744h5db55712m135c67aa2d15bcea@mail.gmail.com>

On Tue, Jun 30, 2009 at 04:43, Rainer Gerhards<rgerhards at hq.adiscon.com> wrote:
> a) circular logs

This.  I keep getting distracted with other projects, but have had on
my TODO list to add an 'omclog' output module that both handles the
*BSD clog format and extends it for some particular shortcomings.
Specifically, I'm thinking of embedded concerns here - compression,
ringing on number of records, and a few miscallaneous implementation
oversights it (clog) has.

> b) stop logging when file hits max size (was that really what this post
> described?)

As noted, most people want this use-case to cut off logs when they hit
a particular size so they may be archived/processed/etc.  I honestly
couldn't care less whether rsyslog has internal compression
capability, as long as it closes the handle and releases it to me to
deal with.  Maybe a hook to call a program when the size trigger is
hit, but even with [di]notify that becomes less necessary.


From sfzhi at yahoo.com  Tue Jun 30 18:12:28 2009
From: sfzhi at yahoo.com (Sergei Zhirikov)
Date: Tue, 30 Jun 2009 18:12:28 +0200
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <4255c2570906300744h5db55712m135c67aa2d15bcea@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>	<h2ckri$16j$1@ger.gmane.org>	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>	<9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
	<4255c2570906300744h5db55712m135c67aa2d15bcea@mail.gmail.com>
Message-ID: <h2ddhf$8q5$1@ger.gmane.org>

RB wrote:
> I honestly couldn't care less whether rsyslog has internal compression
> capability, as long as it closes the handle and releases it to me to
> deal with.  Maybe a hook to call a program when the size trigger is
> hit, but even with [di]notify that becomes less necessary.

I completely agree.

By the way, I use the inotify method to handle logs written by Samba 
server, which does have a log size limit, but does not call an external 
program when that limit is exceeded. So far I did not have any problems 
with that setup.

--
Sergei.



From david at lang.hm  Tue Jun 30 23:46:38 2009
From: david at lang.hm (david at lang.hm)
Date: Tue, 30 Jun 2009 14:46:38 -0700 (PDT)
Subject: [rsyslog] v5-devel won't compile
Message-ID: <alpine.DEB.1.10.0906301445160.9159@asgard>

ubuntu 9.04

zlib version
/* zlib.h -- interface of the 'zlib' general purpose compression library
   version 1.2.3.3, October 2nd, 2006

ii  zlib1g-dev                          1:1.2.3.3.dfsg-12 
compression library - development


error

In file included from zlibw.h:28,
                  from stream.h:72,
                  from obj.h:50,
                  from rsyslog.h:482,
                  from stringbuf.c:39:
/usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or 
'__attribute__' before 'gzseek64'
/usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or 
'__attribute__' before 'gztell64'
/usr/include/zlib.h:1368: error: expected declaration specifiers or '...' 
before 'off64_t'
/usr/include/zlib.h:1369: error: expected declaration specifiers or '...' 
before 'off64_t'
make[2]: *** [librsyslog_la-stringbuf.lo] Error 1
make[2]: Leaving directory `/usr/src/rsyslog/runtime'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/usr/src/rsyslog'
make: *** [all] Error 2


David Lang


From iamsayan at gmail.com  Tue Jun  2 23:42:11 2009
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Tue, 2 Jun 2009 17:42:11 -0400
Subject: [rsyslog] Property replacer manipulation question
Message-ID: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com>

Hello All,

This is my first post to this group, apologize if the question seems naive.

I am trying to change strings within the syslog message before forwarding it
to an external log server. For example, if my regular expression matches an
mac or an IP address present within the message, I want to replace it with a
string like this "<hidden ip>"
Another requirement is before forwarding the log to the external server, I
have to change the facility to a different number (say from local0 to
local7)

The approach I was taking is defining a template with the regular expression
in it, checking for the pattern and then replacing the mac/ip address.

However, I think this kind of pattern replacement is not supported with the
property replacer.
Is that correct?

I am using rsyslog version 3.21 to try this out and here is an example of
what I am trying to do

A sample log message

May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE|All elements Connected

I need to change it to

May 25 08:41:13. [Hidden MAC][12703]: 4|172|CORE|All elements Connected


Before it gets forwarded to the external machine.

Regards,
Sayan


From david at lang.hm  Tue Jun  2 23:50:40 2009
From: david at lang.hm (david at lang.hm)
Date: Tue, 2 Jun 2009 14:50:40 -0700 (PDT)
Subject: [rsyslog] Property replacer manipulation question
In-Reply-To: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com>
References: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906021445440.7953@asgard>

On Tue, 2 Jun 2009, Sayan Chowdhury wrote:

> Hello All,
>
> This is my first post to this group, apologize if the question seems naive.
>
> I am trying to change strings within the syslog message before forwarding it
> to an external log server. For example, if my regular expression matches an
> mac or an IP address present within the message, I want to replace it with a
> string like this "<hidden ip>"
> Another requirement is before forwarding the log to the external server, I
> have to change the facility to a different number (say from local0 to
> local7)
>
> The approach I was taking is defining a template with the regular expression
> in it, checking for the pattern and then replacing the mac/ip address.
>
> However, I think this kind of pattern replacement is not supported with the
> property replacer.
> Is that correct?
>
> I am using rsyslog version 3.21 to try this out and here is an example of
> what I am trying to do
>
> A sample log message
>
> May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE|All elements Connected
>
> I need to change it to
>
> May 25 08:41:13. [Hidden MAC][12703]: 4|172|CORE|All elements Connected
>
>
> Before it gets forwarded to the external machine.

in this case I believe that the MAC would show up in the programname (and 
syslog tag) fields.

so instead of the default format which looks something like
%timestamp% %hostname% %syslogtag%%msg%

you would do something like
%timestamp% %hostname% [Hidden MAC]%msg%

and setup the conditions so that if it matches your condition it sends it 
out with the second format, but if it doesn't it uses the first.

this isn't the search-and-replace functionality you were thinking of, but 
I think it will do the basic job (although the example I gave doesn't 
include the PID portion of the syslog tag, doing a substring on the syslog 
tag would let you add that back in if needed)

David Lang


From iamsayan at gmail.com  Wed Jun  3 01:56:37 2009
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Tue, 2 Jun 2009 19:56:37 -0400
Subject: [rsyslog] Property replacer manipulation question
In-Reply-To: <alpine.DEB.1.10.0906021445440.7953@asgard>
References: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com>
	<alpine.DEB.1.10.0906021445440.7953@asgard>
Message-ID: <87a1ae540906021656s34696amfc3f2e327a51b4be@mail.gmail.com>

Hello David,
Thanks for replying.

I think I should have made my template clearer as well ...
Actually these mac/ip addresses appear in the message portion. with
different kind of logging message that the system generates
so my rule should work for for a message like this as well


May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE| IP 10.0.0.87 assigned to
device DEVICE_0
should become something like


May 25 08:41:13. [HIDDEN MAC][12703]: 4|172|CORE| IP [HIDDEN IP] assigned to
device DEVICE_0


also, the other requirement I have is to change the facility Id in the
outgoing syslog message.

That's when I thought about a search and replace functionality with the
property replacer ...
Not sure whether it is possible though, so far I have been able to write
property replacers which can drop line feed, or replace / with an _ but not
something like this.

Regards,
Sayan


On Tue, Jun 2, 2009 at 5:50 PM, <david at lang.hm> wrote:

> On Tue, 2 Jun 2009, Sayan Chowdhury wrote:
>
> > Hello All,
> >
> > This is my first post to this group, apologize if the question seems
> naive.
> >
> > I am trying to change strings within the syslog message before forwarding
> it
> > to an external log server. For example, if my regular expression matches
> an
> > mac or an IP address present within the message, I want to replace it
> with a
> > string like this "<hidden ip>"
> > Another requirement is before forwarding the log to the external server,
> I
> > have to change the facility to a different number (say from local0 to
> > local7)
> >
> > The approach I was taking is defining a template with the regular
> expression
> > in it, checking for the pattern and then replacing the mac/ip address.
> >
> > However, I think this kind of pattern replacement is not supported with
> the
> > property replacer.
> > Is that correct?
> >
> > I am using rsyslog version 3.21 to try this out and here is an example of
> > what I am trying to do
> >
> > A sample log message
> >
> > May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE|All elements Connected
> >
> > I need to change it to
> >
> > May 25 08:41:13. [Hidden MAC][12703]: 4|172|CORE|All elements Connected
> >
> >
> > Before it gets forwarded to the external machine.
>
> in this case I believe that the MAC would show up in the programname (and
> syslog tag) fields.
>
> so instead of the default format which looks something like
> %timestamp% %hostname% %syslogtag%%msg%
>
> you would do something like
> %timestamp% %hostname% [Hidden MAC]%msg%
>
> and setup the conditions so that if it matches your condition it sends it
> out with the second format, but if it doesn't it uses the first.
>
> this isn't the search-and-replace functionality you were thinking of, but
> I think it will do the basic job (although the example I gave doesn't
> include the PID portion of the syslog tag, doing a substring on the syslog
> tag would let you add that back in if needed)
>
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From david at lang.hm  Wed Jun  3 04:04:16 2009
From: david at lang.hm (david at lang.hm)
Date: Tue, 2 Jun 2009 19:04:16 -0700 (PDT)
Subject: [rsyslog] Property replacer manipulation question
In-Reply-To: <87a1ae540906021656s34696amfc3f2e327a51b4be@mail.gmail.com>
References: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com>
	<alpine.DEB.1.10.0906021445440.7953@asgard>
	<87a1ae540906021656s34696amfc3f2e327a51b4be@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906021901220.7953@asgard>

On Tue, 2 Jun 2009, Sayan Chowdhury wrote:

> Hello David,
> Thanks for replying.
>
> I think I should have made my template clearer as well ...
> Actually these mac/ip addresses appear in the message portion. with
> different kind of logging message that the system generates
> so my rule should work for for a message like this as well
>
>
> May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE| IP 10.0.0.87 assigned to
> device DEVICE_0
> should become something like
>
>
> May 25 08:41:13. [HIDDEN MAC][12703]: 4|172|CORE| IP [HIDDEN IP] assigned to
> device DEVICE_0
>
>
> also, the other requirement I have is to change the facility Id in the
> outgoing syslog message.

there are few enough facility Ids that you can put them into the template

> That's when I thought about a search and replace functionality with the
> property replacer ...
> Not sure whether it is possible though, so far I have been able to write
> property replacers which can drop line feed, or replace / with an _ but not
> something like this.

I haven't seen any ability to do arbatrary search-and-replace in rsyslog.

David Lang


> Regards,
> Sayan
>
>
> On Tue, Jun 2, 2009 at 5:50 PM, <david at lang.hm> wrote:
>
>> On Tue, 2 Jun 2009, Sayan Chowdhury wrote:
>>
>>> Hello All,
>>>
>>> This is my first post to this group, apologize if the question seems
>> naive.
>>>
>>> I am trying to change strings within the syslog message before forwarding
>> it
>>> to an external log server. For example, if my regular expression matches
>> an
>>> mac or an IP address present within the message, I want to replace it
>> with a
>>> string like this "<hidden ip>"
>>> Another requirement is before forwarding the log to the external server,
>> I
>>> have to change the facility to a different number (say from local0 to
>>> local7)
>>>
>>> The approach I was taking is defining a template with the regular
>> expression
>>> in it, checking for the pattern and then replacing the mac/ip address.
>>>
>>> However, I think this kind of pattern replacement is not supported with
>> the
>>> property replacer.
>>> Is that correct?
>>>
>>> I am using rsyslog version 3.21 to try this out and here is an example of
>>> what I am trying to do
>>>
>>> A sample log message
>>>
>>> May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE|All elements Connected
>>>
>>> I need to change it to
>>>
>>> May 25 08:41:13. [Hidden MAC][12703]: 4|172|CORE|All elements Connected
>>>
>>>
>>> Before it gets forwarded to the external machine.
>>
>> in this case I believe that the MAC would show up in the programname (and
>> syslog tag) fields.
>>
>> so instead of the default format which looks something like
>> %timestamp% %hostname% %syslogtag%%msg%
>>
>> you would do something like
>> %timestamp% %hostname% [Hidden MAC]%msg%
>>
>> and setup the conditions so that if it matches your condition it sends it
>> out with the second format, but if it doesn't it uses the first.
>>
>> this isn't the search-and-replace functionality you were thinking of, but
>> I think it will do the basic job (although the example I gave doesn't
>> include the PID portion of the syslog tag, doing a substring on the syslog
>> tag would let you add that back in if needed)
>>
>> David Lang
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Wed Jun  3 07:36:49 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Jun 2009 07:36:49 +0200
Subject: [rsyslog] Property replacer manipulation question
References: <87a1ae540906021442id365506s3fa89641c0223f39@mail.gmail.com><alpine.DEB.1.10.0906021445440.7953@asgard><87a1ae540906021656s34696amfc3f2e327a51b4be@mail.gmail.com>
	<alpine.DEB.1.10.0906021901220.7953@asgard>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F853@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Wednesday, June 03, 2009 4:04 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Property replacer manipulation question
> 
> On Tue, 2 Jun 2009, Sayan Chowdhury wrote:
> 
> > Hello David,
> > Thanks for replying.
> >
> > I think I should have made my template clearer as well ...
> > Actually these mac/ip addresses appear in the message portion. with
> > different kind of logging message that the system generates
> > so my rule should work for for a message like this as well
> >
> >
> > May 25 08:41:13. 00145e178cc8[12703]: 4|172|CORE| IP 10.0.0.87 assigned
to
> > device DEVICE_0
> > should become something like
> >
> >
> > May 25 08:41:13. [HIDDEN MAC][12703]: 4|172|CORE| IP [HIDDEN IP] assigned
to
> > device DEVICE_0
> >
> >
> > also, the other requirement I have is to change the facility Id in the
> > outgoing syslog message.
> 
> there are few enough facility Ids that you can put them into the template
> 
> > That's when I thought about a search and replace functionality with the
> > property replacer ...
> > Not sure whether it is possible though, so far I have been able to write
> > property replacers which can drop line feed, or replace / with an _ but
not
> > something like this.
> 
> I haven't seen any ability to do arbatrary search-and-replace in rsyslog.

This is not available, but sounds like a good idea. I suggest to add a
feature enhancement ticket to the bug tracker. Be warned, though, that I
currently simply have no time to implement it. But it may happen as a
by-activity when I change things in the property replacer, so it would be
good not to forget about it.

Rainer


From rgerhards at hq.adiscon.com  Wed Jun  3 18:33:19 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 3 Jun 2009 18:33:19 +0200
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>

Hi all,

I have begun to redesign the pretty old omfile output plugin. Large parts of
this work are covered by a sponsorship.

I will probably post a number of questions during the redesign process.
Please speak up if you have any opinion on them. Once I have finished the new
design, it is not likely to change during the next three to five years. For
the same reason, please speak up now if you are desperately missing something
in omfile.

A concrete question I have is on the $OutChannel directive[1]. It is
scheduled for replacement for quite a while, and now may be the time that it
finally goes away. If you use $OutChannel, please let me know why.

The most prominent use case is to size-limit files. $OutChannel provides the
ability to execute a command when a certain file size is reached. The command
may then rotate the file. After command execution, omfile checks the file
size again. If it still is above the configured limit, this file will be
disabled (no longer be written to until restart).

If you use this for size-limiting, please let me know how you do it. Please
also let me know if you would like to see things handled differently.

Looking forward to your feedback.

Thanks,
Rainer


[1] http://www.rsyslog.com/doc-rsyslog_conf_output.html


From iamsayan at gmail.com  Wed Jun  3 19:28:24 2009
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Wed, 3 Jun 2009 13:28:24 -0400
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
Message-ID: <87a1ae540906031028s3584fbdat428ca35a17f638ef@mail.gmail.com>

Hello Rainer,

I do use this facility. This is how I do it

$outchannel log_rotation_16,/var/private/log/logsip,
10000000,/opt/platform/bin/log_rotate.pl

if $syslogfacility == '16' and $syslogseverity <= '6' then
$log_rotation_16;precise

The log_rotate.pl perl script that I have rotates the files
and maintains a set of 10 files. When the current file reaches the 10 mb
limit, the older file is zipped and copied over to a different directory.
I am on release 3.21 now

I would definitely like to have the feature be present in the future
releases as well. I am happy as long as this feature of file size based
rotation is there and I can migrate my rules/scripts to whatever new
mechanism that get implemented.

Regards,
Sayan






On Wed, Jun 3, 2009 at 12:33 PM, Rainer Gerhards
<rgerhards at hq.adiscon.com>wrote:

> Hi all,
>
> I have begun to redesign the pretty old omfile output plugin. Large parts
> of
> this work are covered by a sponsorship.
>
> I will probably post a number of questions during the redesign process.
> Please speak up if you have any opinion on them. Once I have finished the
> new
> design, it is not likely to change during the next three to five years. For
> the same reason, please speak up now if you are desperately missing
> something
> in omfile.
>
> A concrete question I have is on the $OutChannel directive[1]. It is
> scheduled for replacement for quite a while, and now may be the time that
> it
> finally goes away. If you use $OutChannel, please let me know why.
>
> The most prominent use case is to size-limit files. $OutChannel provides
> the
> ability to execute a command when a certain file size is reached. The
> command
> may then rotate the file. After command execution, omfile checks the file
> size again. If it still is above the configured limit, this file will be
> disabled (no longer be written to until restart).
>
> If you use this for size-limiting, please let me know how you do it. Please
> also let me know if you would like to see things handled differently.
>
> Looking forward to your feedback.
>
> Thanks,
> Rainer
>
>
> [1] http://www.rsyslog.com/doc-rsyslog_conf_output.html
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Thu Jun  4 11:01:46 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 4 Jun 2009 11:01:46 +0200
Subject: [rsyslog] request for feedback: dynafile cache size
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F879@GRFEXC.intern.adiscon.com>

Hi all,

as you probably know, dynafiles keep a cache of the n most recently used
files open. This is done in order to speed up processing. I wonder how many
files you usually expect to be open. The root of my question is that I think
about the algorithm that finds cache entries. So far, a simple linked list is
used, but I wonder if something more efficient (like a binary tree) would be
useful. On the other hand, if the cache size is really small, there is not
much difference between O(n) and O(log n), so it may not justify added code
complexity.

Feedback appreciated.

Rainer


From david at lang.hm  Thu Jun  4 13:50:42 2009
From: david at lang.hm (david at lang.hm)
Date: Thu, 4 Jun 2009 04:50:42 -0700 (PDT)
Subject: [rsyslog] request for feedback: dynafile cache size
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F879@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F879@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906040448240.11064@asgard>

On Thu, 4 Jun 2009, Rainer Gerhards wrote:

> Hi all,
>
> as you probably know, dynafiles keep a cache of the n most recently used
> files open. This is done in order to speed up processing. I wonder how many
> files you usually expect to be open. The root of my question is that I think
> about the algorithm that finds cache entries. So far, a simple linked list is
> used, but I wonder if something more efficient (like a binary tree) would be
> useful. On the other hand, if the cache size is really small, there is not
> much difference between O(n) and O(log n), so it may not justify added code
> complexity.
>
> Feedback appreciated.

I have a use case where it could be several thousand files

I would expect the normal worst case to be dozens to a few hundred files.

David Lang


From rgerhards at hq.adiscon.com  Thu Jun  4 13:52:18 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 4 Jun 2009 13:52:18 +0200
Subject: [rsyslog] request for feedback: dynafile cache size
References: <9B6E2A8877C38245BFB15CC491A11DA706F879@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906040448240.11064@asgard>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F888@GRFEXC.intern.adiscon.com>

Thanks for the feedback. That definitely asks for a binary tree search. So I
am convinced now (I'll not do it immediately, but probably towards the end of
my redesign work).

Raienr

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Thursday, June 04, 2009 1:51 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] request for feedback: dynafile cache size
> 
> On Thu, 4 Jun 2009, Rainer Gerhards wrote:
> 
> > Hi all,
> >
> > as you probably know, dynafiles keep a cache of the n most recently used
> > files open. This is done in order to speed up processing. I wonder how
many
> > files you usually expect to be open. The root of my question is that I
think
> > about the algorithm that finds cache entries. So far, a simple linked
list
> is
> > used, but I wonder if something more efficient (like a binary tree) would
be
> > useful. On the other hand, if the cache size is really small, there is
not
> > much difference between O(n) and O(log n), so it may not justify added
code
> > complexity.
> >
> > Feedback appreciated.
> 
> I have a use case where it could be several thousand files
> 
> I would expect the normal worst case to be dozens to a few hundred files.
> 
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From nathan at gossamer-threads.com  Sat Jun  6 00:09:16 2009
From: nathan at gossamer-threads.com (Nathan March)
Date: Fri, 05 Jun 2009 15:09:16 -0700
Subject: [rsyslog] Possible bug? 100% cpu on a server thread and
 the	client stops sending data
In-Reply-To: <4A1A57A0.1060903@net-m.de>
References: <4A15999D.3000909@gossamer-threads.com>
	<4A1A37DE.8050004@net-m.de>	<9B6E2A8877C38245BFB15CC491A11DA702B0FB@GRFEXC.intern.adiscon.com>
	<4A1A57A0.1060903@net-m.de>
Message-ID: <4A29978C.9080505@gossamer-threads.com>

Any hope of this bug getting some attention soon? =)

Thanks,
Nathan


Patrick Shen wrote:
> Hi Rainer,
>
> Thanks for the quick reply.
>
> My environment is like below:
>
> CLIENT ---
>          |
>          | (TCP)
>          --------> SERVER ----
>                              |
>                              | (RELP Relay)
>                              --------------> SERVER Standby
>
> I've tested, even if the standby server is under high load, one of our
> clients won't accept any new network connections.
>
> Just for your information.
>
> Best regards,
> Patrick
>
> Rainer Gerhards wrote:
>   
>> Hi all,
>>
>> thanks for the detailed info. Will look at it asap, but I am currently tied
>> up with some other work...
>>
>> Please let me know anything else that may be relevant.
>>
>> Thanks,
>> Rainer
>>
>>     
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of Patrick Shen
>>> Sent: Monday, May 25, 2009 8:17 AM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server thread and the
>>> client stops sending data
>>>
>>> Hi Nathan,
>>>
>>> In my company, we have the similar symptom. We have 50+ servers which
>>> send logs to the central logging servers. Both clients and server use
>>> rsyslog v3.20.5 right now. In the past when we use v3.20.0, the symptom
>>> is more common when server is under high load average.
>>>
>>> It's also weird for us, not all of clients were lost of connections. We
>>> have 2-3 clients which is running the same application will looks like
>>> hanging up when server is high-load. New ssh connection is very slow at
>>> that time. But the connected ssh session works normal (We have some
>>> prepared screen session on other clients which connect to them via ssh).
>>>
>>> When the clients are lost of connections. So far our solution is to
>>> restart rsyslog both on affected clients and server. Then the clients
>>> get back and look normal.
>>>
>>> We've suffered it for a while, but I didn't take chance to analyze it
>>> like you.
>>>
>>> Best regards,
>>> Patrick
>>>
>>> Nathan March wrote:
>>>       
>>>> Having a weird issue here..... Testing out a new rsyslog deployment and
>>>> I've got around 30 servers logging to one machine. On one of the clients
>>>> we had an issue where ssh was mysteriously very slow to login and
>>>> tracked it down to rsyslog causing issues. Simultaneously, one of the
>>>> threads on the logging server spiked up to 100%. New log data from the
>>>> client would stop showing up on the server at this poitn.
>>>>
>>>> On the client after attempting to gracefully stop rsyslog:
>>>>
>>>> vanguard etc # ps aux | grep -i rsyslog
>>>> root      3764  0.0  0.1  5092 2564 ?        S    15:06   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f /etc/rsyslog.conf
>>>> root      3766  0.0  0.1  5092 2564 ?        S    15:06   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f /etc/rsyslog.conf
>>>> root      6203  0.0  0.0  1512  524 pts/3    S    15:09   0:00 grep -i
>>>> rsyslog
>>>> vanguard etc # strace -p 3764
>>>> Process 3764 attached - interrupt to quit
>>>> send(11, "\25\3\2\1\0uE\254\304\n\311\377\204}p\237O\211\322\211"...,
>>>> 261, 0 <unfinished ...>
>>>> Process 3764 detached
>>>>
>>>> FD 11 being the socket to the logging server, it never does anything
>>>> asides from that send.
>>>>
>>>> On the server:
>>>>
>>>> ldap ~ # ps aux | grep rsyslog
>>>> root      7370  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7379  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7380  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7381  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7382  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7383  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7384  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> root      7385 28.3  0.0  83936  3380 ?        R    14:54   5:07
>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>         
>> /etc/rsyslog/rsyslog.conf
>>     
>>>> Stracing the 7385 pid just shows this repeating over and over (FD 4
>>>> being the network socket again):
>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 EAGAIN (Resource
>>>> temporarily unavailable)
>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 EAGAIN (Resource
>>>> temporarily unavailable)
>>>>
>>>> It happened 3 times within a relatively short period of time, sometimes
>>>> within minutes of me restarting rsyslog on both machines. I enabled
>>>> debugging and it took around 8 hours to pop up again.
>>>>
>>>> With debugging, I pulled this from the rsyslog server:
>>>>
>>>> 6640.408108000:imtcp.c: New connect on NSD 0x59ddc0.
>>>> 6640.408129000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: pszRcvBuf (nil),
>>>> lenRcvBuf 0
>>>> 6640.408144000:imtcp.c: hasRcvInBuffer on nsd 0x59dcd0: pszRcvBuf
>>>> 0x5c11c0, lenRcvBuf -1
>>>> 6640.408158000:imtcp.c: hasRcvInBuffer on nsd 0x59dc10: pszRcvBuf
>>>> 0x5cb510, lenRcvBuf -1
>>>> 6640.408171000:imtcp.c: hasRcvInBuffer on nsd 0x5c3d70: pszRcvBuf
>>>> 0x5d45c0, lenRcvBuf -1
>>>> 6640.408185000:imtcp.c: hasRcvInBuffer on nsd 0x5cf460: pszRcvBuf
>>>> 0x5dd010, lenRcvBuf -1
>>>> 6640.408199000:imtcp.c: hasRcvInBuffer on nsd 0x5d8600: pszRcvBuf
>>>> 0x5e58b0, lenRcvBuf -1
>>>> 6640.408213000:imtcp.c: hasRcvInBuffer on nsd 0x59d690: pszRcvBuf
>>>> 0x5ee150, lenRcvBuf -1
>>>> 6640.408227000:imtcp.c: hasRcvInBuffer on nsd 0x59d2f0: pszRcvBuf
>>>> 0x5f7320, lenRcvBuf -1
>>>> 6640.408241000:imtcp.c: hasRcvInBuffer on nsd 0x5e8840: pszRcvBuf
>>>> 0x5fe0a0, lenRcvBuf -1
>>>> 6640.408255000:imtcp.c: hasRcvInBuffer on nsd 0x6027b0: pszRcvBuf
>>>> 0x608350, lenRcvBuf -1
>>>> 6640.408269000:imtcp.c: hasRcvInBuffer on nsd 0x5c3660: pszRcvBuf
>>>> 0x612110, lenRcvBuf -1
>>>> 6640.408283000:imtcp.c: hasRcvInBuffer on nsd 0x602690: pszRcvBuf
>>>> 0x615040, lenRcvBuf -1
>>>> 6640.408296000:imtcp.c: hasRcvInBuffer on nsd 0x5c3b60: pszRcvBuf
>>>> 0x6218f0, lenRcvBuf -1
>>>> 6640.408310000:imtcp.c: hasRcvInBuffer on nsd 0x60c150: pszRcvBuf
>>>> 0x62a190, lenRcvBuf -1
>>>> 6640.408324000:imtcp.c: hasRcvInBuffer on nsd 0x625420: pszRcvBuf
>>>> 0x634fe0, lenRcvBuf -1
>>>> 6640.408338000:imtcp.c: hasRcvInBuffer on nsd 0x6117b0: pszRcvBuf
>>>> 0x63b000, lenRcvBuf -1
>>>> 6640.408352000:imtcp.c: hasRcvInBuffer on nsd 0x603990: pszRcvBuf
>>>> 0x643c70, lenRcvBuf -1
>>>> 6640.408365000:imtcp.c: hasRcvInBuffer on nsd 0x633c00: pszRcvBuf
>>>> 0x64dbb0, lenRcvBuf -1
>>>> 6640.408379000:imtcp.c: hasRcvInBuffer on nsd 0x62cfa0: pszRcvBuf
>>>> 0x650960, lenRcvBuf -1
>>>> 6640.408393000:imtcp.c: hasRcvInBuffer on nsd 0x633d10: pszRcvBuf
>>>> 0x65dbc0, lenRcvBuf -1
>>>> 6640.408407000:imtcp.c: hasRcvInBuffer on nsd 0x5f2860: pszRcvBuf
>>>> 0x666020, lenRcvBuf -1
>>>> 6640.408421000:imtcp.c: hasRcvInBuffer on nsd 0x64d250: pszRcvBuf
>>>> 0x66e480, lenRcvBuf -1
>>>> 6640.408435000:imtcp.c: hasRcvInBuffer on nsd 0x659b30: pszRcvBuf
>>>> 0x676d20, lenRcvBuf 78
>>>> 6640.408448000:imtcp.c: hasRcvInBuffer on nsd 0x669fb0: pszRcvBuf
>>>> 0x67fdd0, lenRcvBuf -1
>>>> 6640.408462000:imtcp.c: hasRcvInBuffer on nsd 0x59a630: pszRcvBuf
>>>> 0x687f10, lenRcvBuf -1
>>>> 6640.408476000:imtcp.c: hasRcvInBuffer on nsd 0x6614c0: pszRcvBuf
>>>> 0x6907b0, lenRcvBuf -1
>>>> 6640.408490000:imtcp.c: hasRcvInBuffer on nsd 0x60c2d0: pszRcvBuf
>>>> 0x699050, lenRcvBuf -1
>>>> 6640.408504000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: pszRcvBuf (nil),
>>>> lenRcvBuf 0
>>>>
>>>> This repeats enough to generate around 29 million lines in the debug log
>>>> file.
>>>>
>>>> In the client log, there's plenty of these which seems somewhat normal:
>>>>
>>>> 6509.743164000:imuxsock.c: --------imuxsock calling select, active file
>>>> descriptors (max 11): 11
>>>> 6509.743231000:imuxsock.c: Message from UNIX socket: #11
>>>> 6509.743262000:imuxsock.c: logmsg: flags 4, from 'vanguard', msg May 20
>>>> 23:15:09 bin/qmail-local[23198]: using .qmail .qmail-cfsc-forum
>>>> 6509.743281000:imuxsock.c: Message has legacy syslog format.
>>>> 6509.743302000:imuxsock.c: main queue: entry added, size now 579 entries
>>>> 6509.743325000:imuxsock.c: wtpAdviseMaxWorkers signals busy
>>>> 6509.743343000:imuxsock.c: main queue: EnqueueMsg advised worker start
>>>>
>>>> However comparing it to an earlier point in the log  there's none of the
>>>> associated tcp queue entries like this:
>>>>
>>>> 3587.234178000:main queue:Reg/w0: TCP sent 107 bytes, requested 107
>>>> 3587.234204000:main queue:Reg/w0: main queue: entering rate limiter
>>>> 3587.234229000:main queue:Reg/w0: main queue: entry deleted, state 0,
>>>> size now 0 entries
>>>> 3587.234255000:main queue:Reg/w0: Called action, logging to builtin-file
>>>> 3587.234285000:main queue:Reg/w0:  (/var/log/maillog)
>>>> 3587.234314000:main queue:Reg/w0: Called action, logging to builtin-fwd
>>>> 3587.234340000:main queue:Reg/w0:  ldap.nmsrv.com
>>>> 3587.234359000:main queue:Reg/w0:  ldap.nmsrv.com:10514/tcp
>>>> 3587.234425000:main queue:Reg/w0: TCP sent 78 bytes, requested 78
>>>> 3587.234446000:main queue:Reg/w0: main queue: entering rate limiter
>>>> 3587.234471000:main queue:Reg/w0: main queue:Reg/w0: worker IDLE,
>>>> waiting for work.
>>>>
>>>> Anyone able to shed some light on this situation? I've got the full
>>>> debug logs if it's useful to anyone to take a closer look.
>>>>
>>>> Thanks!
>>>>
>>>> - Nathan
>>>>
>>>>
>>>>
>>>>         
>> ----------------------------------------------------------------------------
>>     
>>> --------
>>>       
>>>> Server config file
>>>>
>>>>         
>> ----------------------------------------------------------------------------
>>     
>>> --------
>>>       
>>>> $ModLoad immark.so # provides --MARK-- message capability
>>>> $ModLoad imuxsock.so # provides support for local system logging (e.g.
>>>> via logger command)
>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
>>>>
>>>> $umask 0137
>>>> $DirCreateMode 0640
>>>> $FileCreateMode 0640
>>>> $FileOwner root
>>>> $FileGroup admin
>>>>
>>>> $template DynFile,"/var/log/rsyslog/%HOSTNAME%/messages"
>>>> $template DynFileCron,"/var/log/rsyslog/%HOSTNAME%/cron"
>>>> $template DynFileSecure,"/var/log/rsyslog/%HOSTNAME%/secure"
>>>> $template DynFileMail,"/var/log/rsyslog/%HOSTNAME%/maillog"
>>>> $template DynFileSpool,"/var/log/rsyslog/%HOSTNAME%/spooler"
>>>> $template DynFileBoot,"/var/log/rsyslog/%HOSTNAME%/boot"
>>>> $template DynFileSyslog,"/var/log/rsyslog/%HOSTNAME%/syslog"
>>>>
>>>> :msg, contains, "no keys found for"            ~
>>>> :msg, contains, "session opened for user"        ~
>>>>
>>>> # Log all kernel messages to the console.
>>>> # Logging much else clutters up the screen.
>>>> kern.*                                                 /dev/console
>>>>
>>>> # Log anything (except mail) of level info or higher.
>>>> # Don't log private authentication messages!
>>>> *.info;mail.none;cron.none                     ?DynFile
>>>>
>>>> *.warn;\
>>>>         authpriv.none;cron.none;mail.none;news.none     ?DynFileSyslog
>>>>
>>>> # Log all the mail messages in one place.
>>>> mail.*                                                  ?DynFileMail
>>>>
>>>> # Log cron stuff
>>>> cron.*                                                  ?DynFileCron
>>>>
>>>> # Everybody gets emergency messages
>>>> *.emerg                                                 *
>>>>
>>>> # Save news errors of level crit and higher in a special file
>>>> uucp,news.crit                                          ?DynFileSpool
>>>>
>>>> # Save boot messages also to boot.log
>>>> local7.*                                                ?DynFileBoot
>>>>
>>>> # Relp config for PCI
>>>> $ModLoad imrelp
>>>> $InputRELPServerRun 2515
>>>>
>>>> # make gtls driver the default
>>>> $DefaultNetstreamDriver gtls
>>>>
>>>> # certificate files
>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
>>>> $DefaultNetstreamDriverCertFile /etc/ssl/servercrt.pem
>>>> $DefaultNetstreamDriverKeyFile /etc/ssl/serverkey.pem
>>>>
>>>> $ModLoad imtcp.so
>>>>
>>>> $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
>>>> $InputTCPServerStreamDriverAuthMode anon # client is NOT authenticated
>>>> $InputTCPServerRun 10514 # start up listener at port 10514
>>>> $InputTCPMaxSessions 500
>>>>
>>>>
>>>>         
>> ----------------------------------------------------------------------------
>>     
>>> --------
>>>       
>>>> Client config file
>>>>
>>>>         
>> ----------------------------------------------------------------------------
>>     
>>> --------
>>>       
>>>> $ModLoad imuxsock.so # provides support for local system logging (e.g.
>>>> via logger command)
>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
>>>>
>>>> $umask 0137
>>>> $DirCreateMode 0640
>>>> $FileCreateMode 0640
>>>> $FileOwner root
>>>> $FileGroup admin
>>>>
>>>>
>>>> # Log anything 'info' or higher, but lower than 'warn'.
>>>> # Exclude authpriv, cron, mail, and news.  These are logged elsewhere.
>>>> *.info;*.!warn;\
>>>>         authpriv.none;cron.none;mail.none;news.none     /var/log/messages
>>>>
>>>> # Log anything 'warn' or higher.
>>>> # Exclude authpriv, cron, mail, and news.  These are logged elsewhere.
>>>> *.warn;\
>>>>         authpriv.none;cron.none;mail.none;news.none     /var/log/syslog
>>>>
>>>> # Debugging information is logged here.
>>>> *.=debug                                                /var/log/debug
>>>>
>>>> # Private authentication message logging:
>>>> authpriv.*                                              /var/log/secure
>>>>
>>>> # Cron related logs:
>>>> cron.*                                                  /var/log/cron
>>>>
>>>> # Mail related logs:
>>>> mail.*                                                  /var/log/maillog
>>>>
>>>> # Emergency level messages go to all users:
>>>> *.emerg                                                 *
>>>>
>>>>
>>>> ## Uncomment these lines to use RELP instead for PCI compliance (stunnel
>>>> required)
>>>> #$ModLoad omrelp
>>>> #*.*;mail.none :omrelp:localhost:2515;RSYSLOG_ForwardFormat
>>>>
>>>> # certificate files - just CA for a client
>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
>>>>
>>>> # set up the action
>>>> $DefaultNetstreamDriver gtls # use gtls netstream driver
>>>> $ActionSendStreamDriverMode 1 # require TLS for the connection
>>>> $ActionSendStreamDriverAuthMode anon # server is NOT authenticated
>>>> *.* @@(o)ldap.nmsrv.com:10514 # send (all) messages
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.comoften
>>>>         
>>> --
>>> Patrick Shen
>>> Operations Engineer
>>>
>>> net mobile AG Shanghai office
>>>
>>> T: +86  21 61350900 - 222
>>> F: +86  21 61350906
>>> M: +86  13122245730
>>>
>>> FE55 6A7F 3192 F359 C1DF  12F6 9078 58B4 57AF 0BE3
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>       
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>     
>
>
>   


From rgerhards at hq.adiscon.com  Sun Jun  7 20:46:54 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 7 Jun 2009 20:46:54 +0200
Subject: [rsyslog] Possible bug? 100% cpu on a server thread and
	the	client stops sending data
References: <4A15999D.3000909@gossamer-threads.com><4A1A37DE.8050004@net-m.de>	<9B6E2A8877C38245BFB15CC491A11DA702B0FB@GRFEXC.intern.adiscon.com><4A1A57A0.1060903@net-m.de>
	<4A29978C.9080505@gossamer-threads.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8BA@GRFEXC.intern.adiscon.com>

Please give v4 a try, the bug is potentially fixed there. If it actually is,
this gives me a clue of where to look at.

Rainer 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Nathan March
> Sent: Saturday, June 06, 2009 12:09 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server 
> thread and the client stops sending data
> 
> Any hope of this bug getting some attention soon? =)
> 
> Thanks,
> Nathan
> 
> 
> Patrick Shen wrote:
> > Hi Rainer,
> >
> > Thanks for the quick reply.
> >
> > My environment is like below:
> >
> > CLIENT ---
> >          |
> >          | (TCP)
> >          --------> SERVER ----
> >                              |
> >                              | (RELP Relay)
> >                              --------------> SERVER Standby
> >
> > I've tested, even if the standby server is under high load, 
> one of our
> > clients won't accept any new network connections.
> >
> > Just for your information.
> >
> > Best regards,
> > Patrick
> >
> > Rainer Gerhards wrote:
> >   
> >> Hi all,
> >>
> >> thanks for the detailed info. Will look at it asap, but I 
> am currently tied
> >> up with some other work...
> >>
> >> Please let me know anything else that may be relevant.
> >>
> >> Thanks,
> >> Rainer
> >>
> >>     
> >>> -----Original Message-----
> >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>> bounces at lists.adiscon.com] On Behalf Of Patrick Shen
> >>> Sent: Monday, May 25, 2009 8:17 AM
> >>> To: rsyslog-users
> >>> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server 
> thread and the
> >>> client stops sending data
> >>>
> >>> Hi Nathan,
> >>>
> >>> In my company, we have the similar symptom. We have 50+ 
> servers which
> >>> send logs to the central logging servers. Both clients 
> and server use
> >>> rsyslog v3.20.5 right now. In the past when we use 
> v3.20.0, the symptom
> >>> is more common when server is under high load average.
> >>>
> >>> It's also weird for us, not all of clients were lost of 
> connections. We
> >>> have 2-3 clients which is running the same application 
> will looks like
> >>> hanging up when server is high-load. New ssh connection 
> is very slow at
> >>> that time. But the connected ssh session works normal (We 
> have some
> >>> prepared screen session on other clients which connect to 
> them via ssh).
> >>>
> >>> When the clients are lost of connections. So far our 
> solution is to
> >>> restart rsyslog both on affected clients and server. Then 
> the clients
> >>> get back and look normal.
> >>>
> >>> We've suffered it for a while, but I didn't take chance 
> to analyze it
> >>> like you.
> >>>
> >>> Best regards,
> >>> Patrick
> >>>
> >>> Nathan March wrote:
> >>>       
> >>>> Having a weird issue here..... Testing out a new rsyslog 
> deployment and
> >>>> I've got around 30 servers logging to one machine. On 
> one of the clients
> >>>> we had an issue where ssh was mysteriously very slow to login and
> >>>> tracked it down to rsyslog causing issues. 
> Simultaneously, one of the
> >>>> threads on the logging server spiked up to 100%. New log 
> data from the
> >>>> client would stop showing up on the server at this poitn.
> >>>>
> >>>> On the client after attempting to gracefully stop rsyslog:
> >>>>
> >>>> vanguard etc # ps aux | grep -i rsyslog
> >>>> root      3764  0.0  0.1  5092 2564 ?        S    15:06   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f 
> /etc/rsyslog.conf
> >>>> root      3766  0.0  0.1  5092 2564 ?        S    15:06   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f 
> /etc/rsyslog.conf
> >>>> root      6203  0.0  0.0  1512  524 pts/3    S    15:09  
>  0:00 grep -i
> >>>> rsyslog
> >>>> vanguard etc # strace -p 3764
> >>>> Process 3764 attached - interrupt to quit
> >>>> send(11, 
> "\25\3\2\1\0uE\254\304\n\311\377\204}p\237O\211\322\211"...,
> >>>> 261, 0 <unfinished ...>
> >>>> Process 3764 detached
> >>>>
> >>>> FD 11 being the socket to the logging server, it never 
> does anything
> >>>> asides from that send.
> >>>>
> >>>> On the server:
> >>>>
> >>>> ldap ~ # ps aux | grep rsyslog
> >>>> root      7370  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7379  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7380  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7381  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7382  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7383  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7384  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> root      7385 28.3  0.0  83936  3380 ?        R    14:54   5:07
> >>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>         
> >> /etc/rsyslog/rsyslog.conf
> >>     
> >>>> Stracing the 7385 pid just shows this repeating over and 
> over (FD 4
> >>>> being the network socket again):
> >>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 
> EAGAIN (Resource
> >>>> temporarily unavailable)
> >>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 
> EAGAIN (Resource
> >>>> temporarily unavailable)
> >>>>
> >>>> It happened 3 times within a relatively short period of 
> time, sometimes
> >>>> within minutes of me restarting rsyslog on both 
> machines. I enabled
> >>>> debugging and it took around 8 hours to pop up again.
> >>>>
> >>>> With debugging, I pulled this from the rsyslog server:
> >>>>
> >>>> 6640.408108000:imtcp.c: New connect on NSD 0x59ddc0.
> >>>> 6640.408129000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: 
> pszRcvBuf (nil),
> >>>> lenRcvBuf 0
> >>>> 6640.408144000:imtcp.c: hasRcvInBuffer on nsd 0x59dcd0: pszRcvBuf
> >>>> 0x5c11c0, lenRcvBuf -1
> >>>> 6640.408158000:imtcp.c: hasRcvInBuffer on nsd 0x59dc10: pszRcvBuf
> >>>> 0x5cb510, lenRcvBuf -1
> >>>> 6640.408171000:imtcp.c: hasRcvInBuffer on nsd 0x5c3d70: pszRcvBuf
> >>>> 0x5d45c0, lenRcvBuf -1
> >>>> 6640.408185000:imtcp.c: hasRcvInBuffer on nsd 0x5cf460: pszRcvBuf
> >>>> 0x5dd010, lenRcvBuf -1
> >>>> 6640.408199000:imtcp.c: hasRcvInBuffer on nsd 0x5d8600: pszRcvBuf
> >>>> 0x5e58b0, lenRcvBuf -1
> >>>> 6640.408213000:imtcp.c: hasRcvInBuffer on nsd 0x59d690: pszRcvBuf
> >>>> 0x5ee150, lenRcvBuf -1
> >>>> 6640.408227000:imtcp.c: hasRcvInBuffer on nsd 0x59d2f0: pszRcvBuf
> >>>> 0x5f7320, lenRcvBuf -1
> >>>> 6640.408241000:imtcp.c: hasRcvInBuffer on nsd 0x5e8840: pszRcvBuf
> >>>> 0x5fe0a0, lenRcvBuf -1
> >>>> 6640.408255000:imtcp.c: hasRcvInBuffer on nsd 0x6027b0: pszRcvBuf
> >>>> 0x608350, lenRcvBuf -1
> >>>> 6640.408269000:imtcp.c: hasRcvInBuffer on nsd 0x5c3660: pszRcvBuf
> >>>> 0x612110, lenRcvBuf -1
> >>>> 6640.408283000:imtcp.c: hasRcvInBuffer on nsd 0x602690: pszRcvBuf
> >>>> 0x615040, lenRcvBuf -1
> >>>> 6640.408296000:imtcp.c: hasRcvInBuffer on nsd 0x5c3b60: pszRcvBuf
> >>>> 0x6218f0, lenRcvBuf -1
> >>>> 6640.408310000:imtcp.c: hasRcvInBuffer on nsd 0x60c150: pszRcvBuf
> >>>> 0x62a190, lenRcvBuf -1
> >>>> 6640.408324000:imtcp.c: hasRcvInBuffer on nsd 0x625420: pszRcvBuf
> >>>> 0x634fe0, lenRcvBuf -1
> >>>> 6640.408338000:imtcp.c: hasRcvInBuffer on nsd 0x6117b0: pszRcvBuf
> >>>> 0x63b000, lenRcvBuf -1
> >>>> 6640.408352000:imtcp.c: hasRcvInBuffer on nsd 0x603990: pszRcvBuf
> >>>> 0x643c70, lenRcvBuf -1
> >>>> 6640.408365000:imtcp.c: hasRcvInBuffer on nsd 0x633c00: pszRcvBuf
> >>>> 0x64dbb0, lenRcvBuf -1
> >>>> 6640.408379000:imtcp.c: hasRcvInBuffer on nsd 0x62cfa0: pszRcvBuf
> >>>> 0x650960, lenRcvBuf -1
> >>>> 6640.408393000:imtcp.c: hasRcvInBuffer on nsd 0x633d10: pszRcvBuf
> >>>> 0x65dbc0, lenRcvBuf -1
> >>>> 6640.408407000:imtcp.c: hasRcvInBuffer on nsd 0x5f2860: pszRcvBuf
> >>>> 0x666020, lenRcvBuf -1
> >>>> 6640.408421000:imtcp.c: hasRcvInBuffer on nsd 0x64d250: pszRcvBuf
> >>>> 0x66e480, lenRcvBuf -1
> >>>> 6640.408435000:imtcp.c: hasRcvInBuffer on nsd 0x659b30: pszRcvBuf
> >>>> 0x676d20, lenRcvBuf 78
> >>>> 6640.408448000:imtcp.c: hasRcvInBuffer on nsd 0x669fb0: pszRcvBuf
> >>>> 0x67fdd0, lenRcvBuf -1
> >>>> 6640.408462000:imtcp.c: hasRcvInBuffer on nsd 0x59a630: pszRcvBuf
> >>>> 0x687f10, lenRcvBuf -1
> >>>> 6640.408476000:imtcp.c: hasRcvInBuffer on nsd 0x6614c0: pszRcvBuf
> >>>> 0x6907b0, lenRcvBuf -1
> >>>> 6640.408490000:imtcp.c: hasRcvInBuffer on nsd 0x60c2d0: pszRcvBuf
> >>>> 0x699050, lenRcvBuf -1
> >>>> 6640.408504000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: 
> pszRcvBuf (nil),
> >>>> lenRcvBuf 0
> >>>>
> >>>> This repeats enough to generate around 29 million lines 
> in the debug log
> >>>> file.
> >>>>
> >>>> In the client log, there's plenty of these which seems 
> somewhat normal:
> >>>>
> >>>> 6509.743164000:imuxsock.c: --------imuxsock calling 
> select, active file
> >>>> descriptors (max 11): 11
> >>>> 6509.743231000:imuxsock.c: Message from UNIX socket: #11
> >>>> 6509.743262000:imuxsock.c: logmsg: flags 4, from 
> 'vanguard', msg May 20
> >>>> 23:15:09 bin/qmail-local[23198]: using .qmail .qmail-cfsc-forum
> >>>> 6509.743281000:imuxsock.c: Message has legacy syslog format.
> >>>> 6509.743302000:imuxsock.c: main queue: entry added, size 
> now 579 entries
> >>>> 6509.743325000:imuxsock.c: wtpAdviseMaxWorkers signals busy
> >>>> 6509.743343000:imuxsock.c: main queue: EnqueueMsg 
> advised worker start
> >>>>
> >>>> However comparing it to an earlier point in the log  
> there's none of the
> >>>> associated tcp queue entries like this:
> >>>>
> >>>> 3587.234178000:main queue:Reg/w0: TCP sent 107 bytes, 
> requested 107
> >>>> 3587.234204000:main queue:Reg/w0: main queue: entering 
> rate limiter
> >>>> 3587.234229000:main queue:Reg/w0: main queue: entry 
> deleted, state 0,
> >>>> size now 0 entries
> >>>> 3587.234255000:main queue:Reg/w0: Called action, logging 
> to builtin-file
> >>>> 3587.234285000:main queue:Reg/w0:  (/var/log/maillog)
> >>>> 3587.234314000:main queue:Reg/w0: Called action, logging 
> to builtin-fwd
> >>>> 3587.234340000:main queue:Reg/w0:  ldap.nmsrv.com
> >>>> 3587.234359000:main queue:Reg/w0:  ldap.nmsrv.com:10514/tcp
> >>>> 3587.234425000:main queue:Reg/w0: TCP sent 78 bytes, requested 78
> >>>> 3587.234446000:main queue:Reg/w0: main queue: entering 
> rate limiter
> >>>> 3587.234471000:main queue:Reg/w0: main queue:Reg/w0: worker IDLE,
> >>>> waiting for work.
> >>>>
> >>>> Anyone able to shed some light on this situation? I've 
> got the full
> >>>> debug logs if it's useful to anyone to take a closer look.
> >>>>
> >>>> Thanks!
> >>>>
> >>>> - Nathan
> >>>>
> >>>>
> >>>>
> >>>>         
> >> 
> --------------------------------------------------------------
> --------------
> >>     
> >>> --------
> >>>       
> >>>> Server config file
> >>>>
> >>>>         
> >> 
> --------------------------------------------------------------
> --------------
> >>     
> >>> --------
> >>>       
> >>>> $ModLoad immark.so # provides --MARK-- message capability
> >>>> $ModLoad imuxsock.so # provides support for local system 
> logging (e.g.
> >>>> via logger command)
> >>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> >>>>
> >>>> $umask 0137
> >>>> $DirCreateMode 0640
> >>>> $FileCreateMode 0640
> >>>> $FileOwner root
> >>>> $FileGroup admin
> >>>>
> >>>> $template DynFile,"/var/log/rsyslog/%HOSTNAME%/messages"
> >>>> $template DynFileCron,"/var/log/rsyslog/%HOSTNAME%/cron"
> >>>> $template DynFileSecure,"/var/log/rsyslog/%HOSTNAME%/secure"
> >>>> $template DynFileMail,"/var/log/rsyslog/%HOSTNAME%/maillog"
> >>>> $template DynFileSpool,"/var/log/rsyslog/%HOSTNAME%/spooler"
> >>>> $template DynFileBoot,"/var/log/rsyslog/%HOSTNAME%/boot"
> >>>> $template DynFileSyslog,"/var/log/rsyslog/%HOSTNAME%/syslog"
> >>>>
> >>>> :msg, contains, "no keys found for"            ~
> >>>> :msg, contains, "session opened for user"        ~
> >>>>
> >>>> # Log all kernel messages to the console.
> >>>> # Logging much else clutters up the screen.
> >>>> kern.*                                                 
> /dev/console
> >>>>
> >>>> # Log anything (except mail) of level info or higher.
> >>>> # Don't log private authentication messages!
> >>>> *.info;mail.none;cron.none                     ?DynFile
> >>>>
> >>>> *.warn;\
> >>>>         authpriv.none;cron.none;mail.none;news.none     
> ?DynFileSyslog
> >>>>
> >>>> # Log all the mail messages in one place.
> >>>> mail.*                                                  
> ?DynFileMail
> >>>>
> >>>> # Log cron stuff
> >>>> cron.*                                                  
> ?DynFileCron
> >>>>
> >>>> # Everybody gets emergency messages
> >>>> *.emerg                                                 *
> >>>>
> >>>> # Save news errors of level crit and higher in a special file
> >>>> uucp,news.crit                                          
> ?DynFileSpool
> >>>>
> >>>> # Save boot messages also to boot.log
> >>>> local7.*                                                
> ?DynFileBoot
> >>>>
> >>>> # Relp config for PCI
> >>>> $ModLoad imrelp
> >>>> $InputRELPServerRun 2515
> >>>>
> >>>> # make gtls driver the default
> >>>> $DefaultNetstreamDriver gtls
> >>>>
> >>>> # certificate files
> >>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
> >>>> $DefaultNetstreamDriverCertFile /etc/ssl/servercrt.pem
> >>>> $DefaultNetstreamDriverKeyFile /etc/ssl/serverkey.pem
> >>>>
> >>>> $ModLoad imtcp.so
> >>>>
> >>>> $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
> >>>> $InputTCPServerStreamDriverAuthMode anon # client is NOT 
> authenticated
> >>>> $InputTCPServerRun 10514 # start up listener at port 10514
> >>>> $InputTCPMaxSessions 500
> >>>>
> >>>>
> >>>>         
> >> 
> --------------------------------------------------------------
> --------------
> >>     
> >>> --------
> >>>       
> >>>> Client config file
> >>>>
> >>>>         
> >> 
> --------------------------------------------------------------
> --------------
> >>     
> >>> --------
> >>>       
> >>>> $ModLoad imuxsock.so # provides support for local system 
> logging (e.g.
> >>>> via logger command)
> >>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> >>>>
> >>>> $umask 0137
> >>>> $DirCreateMode 0640
> >>>> $FileCreateMode 0640
> >>>> $FileOwner root
> >>>> $FileGroup admin
> >>>>
> >>>>
> >>>> # Log anything 'info' or higher, but lower than 'warn'.
> >>>> # Exclude authpriv, cron, mail, and news.  These are 
> logged elsewhere.
> >>>> *.info;*.!warn;\
> >>>>         authpriv.none;cron.none;mail.none;news.none     
> /var/log/messages
> >>>>
> >>>> # Log anything 'warn' or higher.
> >>>> # Exclude authpriv, cron, mail, and news.  These are 
> logged elsewhere.
> >>>> *.warn;\
> >>>>         authpriv.none;cron.none;mail.none;news.none     
> /var/log/syslog
> >>>>
> >>>> # Debugging information is logged here.
> >>>> *.=debug                                                
> /var/log/debug
> >>>>
> >>>> # Private authentication message logging:
> >>>> authpriv.*                                              
> /var/log/secure
> >>>>
> >>>> # Cron related logs:
> >>>> cron.*                                                  
> /var/log/cron
> >>>>
> >>>> # Mail related logs:
> >>>> mail.*                                                  
> /var/log/maillog
> >>>>
> >>>> # Emergency level messages go to all users:
> >>>> *.emerg                                                 *
> >>>>
> >>>>
> >>>> ## Uncomment these lines to use RELP instead for PCI 
> compliance (stunnel
> >>>> required)
> >>>> #$ModLoad omrelp
> >>>> #*.*;mail.none :omrelp:localhost:2515;RSYSLOG_ForwardFormat
> >>>>
> >>>> # certificate files - just CA for a client
> >>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
> >>>>
> >>>> # set up the action
> >>>> $DefaultNetstreamDriver gtls # use gtls netstream driver
> >>>> $ActionSendStreamDriverMode 1 # require TLS for the connection
> >>>> $ActionSendStreamDriverAuthMode anon # server is NOT 
> authenticated
> >>>> *.* @@(o)ldap.nmsrv.com:10514 # send (all) messages
> >>>>
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.comoften
> >>>>         
> >>> --
> >>> Patrick Shen
> >>> Operations Engineer
> >>>
> >>> net mobile AG Shanghai office
> >>>
> >>> T: +86  21 61350900 - 222
> >>> F: +86  21 61350906
> >>> M: +86  13122245730
> >>>
> >>> FE55 6A7F 3192 F359 C1DF  12F6 9078 58B4 57AF 0BE3
> >>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>       
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>     
> >
> >
> >   
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 


From rgerhards at hq.adiscon.com  Mon Jun  8 17:33:00 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 8 Jun 2009 17:33:00 +0200
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>

Hi all,

I am working on the ultra-reliable file writing capability. I have revisited
the doc and am considering fsync() vs. fdatasync(). From the man pages, it
looks like fdatasync() is sufficient if I just need to ensure that any data
written can also be read. The advantage of fdatasync() is that it is somewhat
faster as not all meta data is necessarily written.

I would appreciate if someone could comment on this issue.

Thanks,
Rainer


From david at lang.hm  Mon Jun  8 18:09:56 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 8 Jun 2009 09:09:56 -0700 (PDT)
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906080908030.32468@asgard>

On Mon, 8 Jun 2009, Rainer Gerhards wrote:

> Hi all,
>
> I am working on the ultra-reliable file writing capability. I have revisited
> the doc and am considering fsync() vs. fdatasync(). From the man pages, it
> looks like fdatasync() is sufficient if I just need to ensure that any data
> written can also be read. The advantage of fdatasync() is that it is somewhat
> faster as not all meta data is necessarily written.
>
> I would appreciate if someone could comment on this issue.

my understanding is that if you are growing the file you need to do a 
fsync of the directory, but if you are re-writing part of an already 
allocated file fdatasync is the better thing to use.

David Lang


From rgerhards at hq.adiscon.com  Mon Jun  8 18:18:12 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 8 Jun 2009 18:18:12 +0200
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906080908030.32468@asgard>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8C6@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Monday, June 08, 2009 6:10 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
> 
> On Mon, 8 Jun 2009, Rainer Gerhards wrote:
> 
> > Hi all,
> >
> > I am working on the ultra-reliable file writing capability. I have
revisited
> > the doc and am considering fsync() vs. fdatasync(). From the man pages,
it
> > looks like fdatasync() is sufficient if I just need to ensure that any
data
> > written can also be read. The advantage of fdatasync() is that it is
> somewhat
> > faster as not all meta data is necessarily written.
> >
> > I would appreciate if someone could comment on this issue.
> 
> my understanding is that if you are growing the file you need to do a
> fsync of the directory, but if you are re-writing part of an already
> allocated file fdatasync is the better thing to use.

The man page[1] is somewhat vague. It tells:

"Calling fsync() does not necessarily ensure that the entry in the directory
containing the file has also reached disk. For that an explicit fsync() on a
file descriptor for the directory is also needed."

But also

"fdatasync() is similar to fsync(), but does not flush modified metadata
unless that metadata is needed in order to allow a subsequent data retrieval
to be correctly handled. For example, changes to st_atime or st_mtime
(respectively, time of last access and time of last modification; see
stat(2)) do not not require flushing because they are not necessary for a
subsequent data read to be handled correctly. On the other hand, a change to
the file size (st_size, as made by say ftruncate(2)), would require a
metadata flush.
"

So my current understanding is that fdatasync() is always safe, but there
still exist situations where a sync on the directory is needed. What I can
think of is a newly created file.

I have to admit that I am a bit puzzled of what it takes to do the "explicit
fsync on the directory". Most importantly, I need to have the file handle
open for writing, else the fsync will potentially fail. I don't think this is
such an excellent idea on directories.. I am right now working on this, and
my search has not brought up many good hints. So what I currently plan to do
is an open(dirname, O_RDWR) on the directory and then do an fsync on it
later. 

And I guess even after the fsync() on the directeroy, there potentially is an
issue with newly created files inside newly create directories, but I don't
think I will address this immediately. But I may be useful/necessary to add a
tree walker who actually walks the full path name and calls the fsyncs on all
of them...

Comments are highly appreciated.

Rainer

[1] http://linux.die.net/man/2/fsync


From rgerhards at hq.adiscon.com  Mon Jun  8 18:31:28 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 8 Jun 2009 18:31:28 +0200
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906080908030.32468@asgard>
	<9B6E2A8877C38245BFB15CC491A11DA706F8C6@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8C7@GRFEXC.intern.adiscon.com>

> my search has not brought up many good hints. So what I currently plan to
do
> is an open(dirname, O_RDWR) on the directory and then do an fsync on it
> later.

... which does not work, because I receive EISDIR ;)

No I do an O_RDONLY, which works and then even fsync() on it works as well.
So this is what I will now be doing, but with a somewhat bad feeling...

Rainer


From epiphani at gmail.com  Mon Jun  8 18:12:13 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 8 Jun 2009 12:12:13 -0400
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
In-Reply-To: <alpine.DEB.1.10.0906080908030.32468@asgard>
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906080908030.32468@asgard>
Message-ID: <e7ca40f70906080912i189153a6m1fe4633008048c12@mail.gmail.com>

On Mon, Jun 8, 2009 at 12:09 PM, <david at lang.hm> wrote:
>> I am working on the ultra-reliable file writing capability. I have revisited
>> the doc and am considering fsync() vs. fdatasync(). From the man pages, it
>> looks like fdatasync() is sufficient if I just need to ensure that any data
>> written can also be read. The advantage of fdatasync() is that it is somewhat
>> faster as not all meta data is necessarily written.
>>
>> I would appreciate if someone could comment on this issue.
>
> my understanding is that if you are growing the file you need to do a
> fsync of the directory, but if you are re-writing part of an already
> allocated file fdatasync is the better thing to use.

While following this path, it would also be a good idea to check NFS
compatibility, as I would bet that is a common configuration.  NFS can
have its own unique quirks....

-Aaron


From rgerhards at hq.adiscon.com  Mon Jun  8 18:53:18 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 8 Jun 2009 18:53:18 +0200
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906080908030.32468@asgard>
	<e7ca40f70906080912i189153a6m1fe4633008048c12@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8C8@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> Sent: Monday, June 08, 2009 6:12 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
> 
> On Mon, Jun 8, 2009 at 12:09 PM, <david at lang.hm> wrote:
> >> I am working on the ultra-reliable file writing capability. I have
> revisited
> >> the doc and am considering fsync() vs. fdatasync(). From the man pages,
it
> >> looks like fdatasync() is sufficient if I just need to ensure that any
data
> >> written can also be read. The advantage of fdatasync() is that it is
> somewhat
> >> faster as not all meta data is necessarily written.
> >>
> >> I would appreciate if someone could comment on this issue.
> >
> > my understanding is that if you are growing the file you need to do a
> > fsync of the directory, but if you are re-writing part of an already
> > allocated file fdatasync is the better thing to use.
> 
> While following this path, it would also be a good idea to check NFS
> compatibility, as I would bet that is a common configuration.  NFS can
> have its own unique quirks....

>From what I have seen during my research, I would expect that NFS does not
always correctly fsync(). It seems to depend a lot on the actual
implementation, maybe version of it. What I currently do is ignore an error
during fsync() (fdatasync to be more precise). The reason is that there is
not much I could do if I see the error. Aborting the run would probably be
much more fatal than ignoring the error. The same goes if the fsync() for the
directory fails or if I can not open the directory.

I would assume that if a truly reliable system is requested, local disks must
be used.

Comments welcome,
Rainer


From david at lang.hm  Mon Jun  8 19:07:35 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 8 Jun 2009 10:07:35 -0700 (PDT)
Subject: [rsyslog] ultra-reliable disk files fsync vs. fdatasync
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F8C6@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F8C5@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906080908030.32468@asgard>
	<9B6E2A8877C38245BFB15CC491A11DA706F8C6@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906080956030.9159@asgard>

On Mon, 8 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Mon, 8 Jun 2009, Rainer Gerhards wrote:
>>
>>> Hi all,
>>>
>>> I am working on the ultra-reliable file writing capability. I have
> revisited
>>> the doc and am considering fsync() vs. fdatasync(). From the man pages,
> it
>>> looks like fdatasync() is sufficient if I just need to ensure that any
> data
>>> written can also be read. The advantage of fdatasync() is that it is
>> somewhat
>>> faster as not all meta data is necessarily written.
>>>
>>> I would appreciate if someone could comment on this issue.
>>
>> my understanding is that if you are growing the file you need to do a
>> fsync of the directory, but if you are re-writing part of an already
>> allocated file fdatasync is the better thing to use.
>
> The man page[1] is somewhat vague. It tells:
>
> "Calling fsync() does not necessarily ensure that the entry in the directory
> containing the file has also reached disk. For that an explicit fsync() on a
> file descriptor for the directory is also needed."
>
> But also
>
> "fdatasync() is similar to fsync(), but does not flush modified metadata
> unless that metadata is needed in order to allow a subsequent data retrieval
> to be correctly handled. For example, changes to st_atime or st_mtime
> (respectively, time of last access and time of last modification; see
> stat(2)) do not not require flushing because they are not necessary for a
> subsequent data read to be handled correctly. On the other hand, a change to
> the file size (st_size, as made by say ftruncate(2)), would require a
> metadata flush.
> "
>
> So my current understanding is that fdatasync() is always safe, but there
> still exist situations where a sync on the directory is needed. What I can
> think of is a newly created file.

any time the file size changes (so that the metadata about the disk 
allocation is safe)

> I have to admit that I am a bit puzzled of what it takes to do the "explicit
> fsync on the directory". Most importantly, I need to have the file handle
> open for writing, else the fsync will potentially fail. I don't think this is
> such an excellent idea on directories.. I am right now working on this, and
> my search has not brought up many good hints. So what I currently plan to do
> is an open(dirname, O_RDWR) on the directory and then do an fsync on it
> later.

I believe that that's correct.

> And I guess even after the fsync() on the directeroy, there potentially is an
> issue with newly created files inside newly create directories, but I don't
> think I will address this immediately. But I may be useful/necessary to add a
> tree walker who actually walks the full path name and calls the fsyncs on all
> of them...

in an ideal world yes, but in practice you would only need to do this when 
you create a new file and that new directory entry caused the filesystem 
to allocate more space on disk to hole the directory entry, so it's almost 
never an issue.

another advantage of fdatasync is that it tells the OS exactly what 
portion of the file it needs to work on, with a fsync it needs to search 
all buffers related to that file to figure out what needs to be written 
(not a big hit, but a hit)

David Lang


From nathan at gossamer-threads.com  Tue Jun  9 18:57:49 2009
From: nathan at gossamer-threads.com (Nathan March)
Date: Tue, 09 Jun 2009 09:57:49 -0700
Subject: [rsyslog] Possible bug? 100% cpu on a server thread
 and	the	client stops sending data
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F8BA@GRFEXC.intern.adiscon.com>
References: <4A15999D.3000909@gossamer-threads.com><4A1A37DE.8050004@net-m.de>	<9B6E2A8877C38245BFB15CC491A11DA702B0FB@GRFEXC.intern.adiscon.com><4A1A57A0.1060903@net-m.de>	<4A29978C.9080505@gossamer-threads.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F8BA@GRFEXC.intern.adiscon.com>
Message-ID: <4A2E948D.4060809@gossamer-threads.com>

Confirmed as still happening on 4.1.7

- Nathan

Rainer Gerhards wrote:
> Please give v4 a try, the bug is potentially fixed there. If it actually is,
> this gives me a clue of where to look at.
>
> Rainer 
>
>   
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com 
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Nathan March
>> Sent: Saturday, June 06, 2009 12:09 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server 
>> thread and the client stops sending data
>>
>> Any hope of this bug getting some attention soon? =)
>>
>> Thanks,
>> Nathan
>>
>>
>> Patrick Shen wrote:
>>     
>>> Hi Rainer,
>>>
>>> Thanks for the quick reply.
>>>
>>> My environment is like below:
>>>
>>> CLIENT ---
>>>          |
>>>          | (TCP)
>>>          --------> SERVER ----
>>>                              |
>>>                              | (RELP Relay)
>>>                              --------------> SERVER Standby
>>>
>>> I've tested, even if the standby server is under high load, 
>>>       
>> one of our
>>     
>>> clients won't accept any new network connections.
>>>
>>> Just for your information.
>>>
>>> Best regards,
>>> Patrick
>>>
>>> Rainer Gerhards wrote:
>>>   
>>>       
>>>> Hi all,
>>>>
>>>> thanks for the detailed info. Will look at it asap, but I 
>>>>         
>> am currently tied
>>     
>>>> up with some other work...
>>>>
>>>> Please let me know anything else that may be relevant.
>>>>
>>>> Thanks,
>>>> Rainer
>>>>
>>>>     
>>>>         
>>>>> -----Original Message-----
>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>> bounces at lists.adiscon.com] On Behalf Of Patrick Shen
>>>>> Sent: Monday, May 25, 2009 8:17 AM
>>>>> To: rsyslog-users
>>>>> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server 
>>>>>           
>> thread and the
>>     
>>>>> client stops sending data
>>>>>
>>>>> Hi Nathan,
>>>>>
>>>>> In my company, we have the similar symptom. We have 50+ 
>>>>>           
>> servers which
>>     
>>>>> send logs to the central logging servers. Both clients 
>>>>>           
>> and server use
>>     
>>>>> rsyslog v3.20.5 right now. In the past when we use 
>>>>>           
>> v3.20.0, the symptom
>>     
>>>>> is more common when server is under high load average.
>>>>>
>>>>> It's also weird for us, not all of clients were lost of 
>>>>>           
>> connections. We
>>     
>>>>> have 2-3 clients which is running the same application 
>>>>>           
>> will looks like
>>     
>>>>> hanging up when server is high-load. New ssh connection 
>>>>>           
>> is very slow at
>>     
>>>>> that time. But the connected ssh session works normal (We 
>>>>>           
>> have some
>>     
>>>>> prepared screen session on other clients which connect to 
>>>>>           
>> them via ssh).
>>     
>>>>> When the clients are lost of connections. So far our 
>>>>>           
>> solution is to
>>     
>>>>> restart rsyslog both on affected clients and server. Then 
>>>>>           
>> the clients
>>     
>>>>> get back and look normal.
>>>>>
>>>>> We've suffered it for a while, but I didn't take chance 
>>>>>           
>> to analyze it
>>     
>>>>> like you.
>>>>>
>>>>> Best regards,
>>>>> Patrick
>>>>>
>>>>> Nathan March wrote:
>>>>>       
>>>>>           
>>>>>> Having a weird issue here..... Testing out a new rsyslog 
>>>>>>             
>> deployment and
>>     
>>>>>> I've got around 30 servers logging to one machine. On 
>>>>>>             
>> one of the clients
>>     
>>>>>> we had an issue where ssh was mysteriously very slow to login and
>>>>>> tracked it down to rsyslog causing issues. 
>>>>>>             
>> Simultaneously, one of the
>>     
>>>>>> threads on the logging server spiked up to 100%. New log 
>>>>>>             
>> data from the
>>     
>>>>>> client would stop showing up on the server at this poitn.
>>>>>>
>>>>>> On the client after attempting to gracefully stop rsyslog:
>>>>>>
>>>>>> vanguard etc # ps aux | grep -i rsyslog
>>>>>> root      3764  0.0  0.1  5092 2564 ?        S    15:06   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f 
>>>>>>             
>> /etc/rsyslog.conf
>>     
>>>>>> root      3766  0.0  0.1  5092 2564 ?        S    15:06   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f 
>>>>>>             
>> /etc/rsyslog.conf
>>     
>>>>>> root      6203  0.0  0.0  1512  524 pts/3    S    15:09  
>>>>>>             
>>  0:00 grep -i
>>     
>>>>>> rsyslog
>>>>>> vanguard etc # strace -p 3764
>>>>>> Process 3764 attached - interrupt to quit
>>>>>> send(11, 
>>>>>>             
>> "\25\3\2\1\0uE\254\304\n\311\377\204}p\237O\211\322\211"...,
>>     
>>>>>> 261, 0 <unfinished ...>
>>>>>> Process 3764 detached
>>>>>>
>>>>>> FD 11 being the socket to the logging server, it never 
>>>>>>             
>> does anything
>>     
>>>>>> asides from that send.
>>>>>>
>>>>>> On the server:
>>>>>>
>>>>>> ldap ~ # ps aux | grep rsyslog
>>>>>> root      7370  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7379  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7380  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7381  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7382  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7383  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7384  0.0  0.0  83936  3380 ?        S    14:54   0:00
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> root      7385 28.3  0.0  83936  3380 ?        R    14:54   5:07
>>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
>>>>>>         
>>>>>>             
>>>> /etc/rsyslog/rsyslog.conf
>>>>     
>>>>         
>>>>>> Stracing the 7385 pid just shows this repeating over and 
>>>>>>             
>> over (FD 4
>>     
>>>>>> being the network socket again):
>>>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 
>>>>>>             
>> EAGAIN (Resource
>>     
>>>>>> temporarily unavailable)
>>>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1 
>>>>>>             
>> EAGAIN (Resource
>>     
>>>>>> temporarily unavailable)
>>>>>>
>>>>>> It happened 3 times within a relatively short period of 
>>>>>>             
>> time, sometimes
>>     
>>>>>> within minutes of me restarting rsyslog on both 
>>>>>>             
>> machines. I enabled
>>     
>>>>>> debugging and it took around 8 hours to pop up again.
>>>>>>
>>>>>> With debugging, I pulled this from the rsyslog server:
>>>>>>
>>>>>> 6640.408108000:imtcp.c: New connect on NSD 0x59ddc0.
>>>>>> 6640.408129000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: 
>>>>>>             
>> pszRcvBuf (nil),
>>     
>>>>>> lenRcvBuf 0
>>>>>> 6640.408144000:imtcp.c: hasRcvInBuffer on nsd 0x59dcd0: pszRcvBuf
>>>>>> 0x5c11c0, lenRcvBuf -1
>>>>>> 6640.408158000:imtcp.c: hasRcvInBuffer on nsd 0x59dc10: pszRcvBuf
>>>>>> 0x5cb510, lenRcvBuf -1
>>>>>> 6640.408171000:imtcp.c: hasRcvInBuffer on nsd 0x5c3d70: pszRcvBuf
>>>>>> 0x5d45c0, lenRcvBuf -1
>>>>>> 6640.408185000:imtcp.c: hasRcvInBuffer on nsd 0x5cf460: pszRcvBuf
>>>>>> 0x5dd010, lenRcvBuf -1
>>>>>> 6640.408199000:imtcp.c: hasRcvInBuffer on nsd 0x5d8600: pszRcvBuf
>>>>>> 0x5e58b0, lenRcvBuf -1
>>>>>> 6640.408213000:imtcp.c: hasRcvInBuffer on nsd 0x59d690: pszRcvBuf
>>>>>> 0x5ee150, lenRcvBuf -1
>>>>>> 6640.408227000:imtcp.c: hasRcvInBuffer on nsd 0x59d2f0: pszRcvBuf
>>>>>> 0x5f7320, lenRcvBuf -1
>>>>>> 6640.408241000:imtcp.c: hasRcvInBuffer on nsd 0x5e8840: pszRcvBuf
>>>>>> 0x5fe0a0, lenRcvBuf -1
>>>>>> 6640.408255000:imtcp.c: hasRcvInBuffer on nsd 0x6027b0: pszRcvBuf
>>>>>> 0x608350, lenRcvBuf -1
>>>>>> 6640.408269000:imtcp.c: hasRcvInBuffer on nsd 0x5c3660: pszRcvBuf
>>>>>> 0x612110, lenRcvBuf -1
>>>>>> 6640.408283000:imtcp.c: hasRcvInBuffer on nsd 0x602690: pszRcvBuf
>>>>>> 0x615040, lenRcvBuf -1
>>>>>> 6640.408296000:imtcp.c: hasRcvInBuffer on nsd 0x5c3b60: pszRcvBuf
>>>>>> 0x6218f0, lenRcvBuf -1
>>>>>> 6640.408310000:imtcp.c: hasRcvInBuffer on nsd 0x60c150: pszRcvBuf
>>>>>> 0x62a190, lenRcvBuf -1
>>>>>> 6640.408324000:imtcp.c: hasRcvInBuffer on nsd 0x625420: pszRcvBuf
>>>>>> 0x634fe0, lenRcvBuf -1
>>>>>> 6640.408338000:imtcp.c: hasRcvInBuffer on nsd 0x6117b0: pszRcvBuf
>>>>>> 0x63b000, lenRcvBuf -1
>>>>>> 6640.408352000:imtcp.c: hasRcvInBuffer on nsd 0x603990: pszRcvBuf
>>>>>> 0x643c70, lenRcvBuf -1
>>>>>> 6640.408365000:imtcp.c: hasRcvInBuffer on nsd 0x633c00: pszRcvBuf
>>>>>> 0x64dbb0, lenRcvBuf -1
>>>>>> 6640.408379000:imtcp.c: hasRcvInBuffer on nsd 0x62cfa0: pszRcvBuf
>>>>>> 0x650960, lenRcvBuf -1
>>>>>> 6640.408393000:imtcp.c: hasRcvInBuffer on nsd 0x633d10: pszRcvBuf
>>>>>> 0x65dbc0, lenRcvBuf -1
>>>>>> 6640.408407000:imtcp.c: hasRcvInBuffer on nsd 0x5f2860: pszRcvBuf
>>>>>> 0x666020, lenRcvBuf -1
>>>>>> 6640.408421000:imtcp.c: hasRcvInBuffer on nsd 0x64d250: pszRcvBuf
>>>>>> 0x66e480, lenRcvBuf -1
>>>>>> 6640.408435000:imtcp.c: hasRcvInBuffer on nsd 0x659b30: pszRcvBuf
>>>>>> 0x676d20, lenRcvBuf 78
>>>>>> 6640.408448000:imtcp.c: hasRcvInBuffer on nsd 0x669fb0: pszRcvBuf
>>>>>> 0x67fdd0, lenRcvBuf -1
>>>>>> 6640.408462000:imtcp.c: hasRcvInBuffer on nsd 0x59a630: pszRcvBuf
>>>>>> 0x687f10, lenRcvBuf -1
>>>>>> 6640.408476000:imtcp.c: hasRcvInBuffer on nsd 0x6614c0: pszRcvBuf
>>>>>> 0x6907b0, lenRcvBuf -1
>>>>>> 6640.408490000:imtcp.c: hasRcvInBuffer on nsd 0x60c2d0: pszRcvBuf
>>>>>> 0x699050, lenRcvBuf -1
>>>>>> 6640.408504000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550: 
>>>>>>             
>> pszRcvBuf (nil),
>>     
>>>>>> lenRcvBuf 0
>>>>>>
>>>>>> This repeats enough to generate around 29 million lines 
>>>>>>             
>> in the debug log
>>     
>>>>>> file.
>>>>>>
>>>>>> In the client log, there's plenty of these which seems 
>>>>>>             
>> somewhat normal:
>>     
>>>>>> 6509.743164000:imuxsock.c: --------imuxsock calling 
>>>>>>             
>> select, active file
>>     
>>>>>> descriptors (max 11): 11
>>>>>> 6509.743231000:imuxsock.c: Message from UNIX socket: #11
>>>>>> 6509.743262000:imuxsock.c: logmsg: flags 4, from 
>>>>>>             
>> 'vanguard', msg May 20
>>     
>>>>>> 23:15:09 bin/qmail-local[23198]: using .qmail .qmail-cfsc-forum
>>>>>> 6509.743281000:imuxsock.c: Message has legacy syslog format.
>>>>>> 6509.743302000:imuxsock.c: main queue: entry added, size 
>>>>>>             
>> now 579 entries
>>     
>>>>>> 6509.743325000:imuxsock.c: wtpAdviseMaxWorkers signals busy
>>>>>> 6509.743343000:imuxsock.c: main queue: EnqueueMsg 
>>>>>>             
>> advised worker start
>>     
>>>>>> However comparing it to an earlier point in the log  
>>>>>>             
>> there's none of the
>>     
>>>>>> associated tcp queue entries like this:
>>>>>>
>>>>>> 3587.234178000:main queue:Reg/w0: TCP sent 107 bytes, 
>>>>>>             
>> requested 107
>>     
>>>>>> 3587.234204000:main queue:Reg/w0: main queue: entering 
>>>>>>             
>> rate limiter
>>     
>>>>>> 3587.234229000:main queue:Reg/w0: main queue: entry 
>>>>>>             
>> deleted, state 0,
>>     
>>>>>> size now 0 entries
>>>>>> 3587.234255000:main queue:Reg/w0: Called action, logging 
>>>>>>             
>> to builtin-file
>>     
>>>>>> 3587.234285000:main queue:Reg/w0:  (/var/log/maillog)
>>>>>> 3587.234314000:main queue:Reg/w0: Called action, logging 
>>>>>>             
>> to builtin-fwd
>>     
>>>>>> 3587.234340000:main queue:Reg/w0:  ldap.nmsrv.com
>>>>>> 3587.234359000:main queue:Reg/w0:  ldap.nmsrv.com:10514/tcp
>>>>>> 3587.234425000:main queue:Reg/w0: TCP sent 78 bytes, requested 78
>>>>>> 3587.234446000:main queue:Reg/w0: main queue: entering 
>>>>>>             
>> rate limiter
>>     
>>>>>> 3587.234471000:main queue:Reg/w0: main queue:Reg/w0: worker IDLE,
>>>>>> waiting for work.
>>>>>>
>>>>>> Anyone able to shed some light on this situation? I've 
>>>>>>             
>> got the full
>>     
>>>>>> debug logs if it's useful to anyone to take a closer look.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> - Nathan
>>>>>>
>>>>>>
>>>>>>
>>>>>>         
>>>>>>             
>> --------------------------------------------------------------
>> --------------
>>     
>>>>     
>>>>         
>>>>> --------
>>>>>       
>>>>>           
>>>>>> Server config file
>>>>>>
>>>>>>         
>>>>>>             
>> --------------------------------------------------------------
>> --------------
>>     
>>>>     
>>>>         
>>>>> --------
>>>>>       
>>>>>           
>>>>>> $ModLoad immark.so # provides --MARK-- message capability
>>>>>> $ModLoad imuxsock.so # provides support for local system 
>>>>>>             
>> logging (e.g.
>>     
>>>>>> via logger command)
>>>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
>>>>>>
>>>>>> $umask 0137
>>>>>> $DirCreateMode 0640
>>>>>> $FileCreateMode 0640
>>>>>> $FileOwner root
>>>>>> $FileGroup admin
>>>>>>
>>>>>> $template DynFile,"/var/log/rsyslog/%HOSTNAME%/messages"
>>>>>> $template DynFileCron,"/var/log/rsyslog/%HOSTNAME%/cron"
>>>>>> $template DynFileSecure,"/var/log/rsyslog/%HOSTNAME%/secure"
>>>>>> $template DynFileMail,"/var/log/rsyslog/%HOSTNAME%/maillog"
>>>>>> $template DynFileSpool,"/var/log/rsyslog/%HOSTNAME%/spooler"
>>>>>> $template DynFileBoot,"/var/log/rsyslog/%HOSTNAME%/boot"
>>>>>> $template DynFileSyslog,"/var/log/rsyslog/%HOSTNAME%/syslog"
>>>>>>
>>>>>> :msg, contains, "no keys found for"            ~
>>>>>> :msg, contains, "session opened for user"        ~
>>>>>>
>>>>>> # Log all kernel messages to the console.
>>>>>> # Logging much else clutters up the screen.
>>>>>> kern.*                                                 
>>>>>>             
>> /dev/console
>>     
>>>>>> # Log anything (except mail) of level info or higher.
>>>>>> # Don't log private authentication messages!
>>>>>> *.info;mail.none;cron.none                     ?DynFile
>>>>>>
>>>>>> *.warn;\
>>>>>>         authpriv.none;cron.none;mail.none;news.none     
>>>>>>             
>> ?DynFileSyslog
>>     
>>>>>> # Log all the mail messages in one place.
>>>>>> mail.*                                                  
>>>>>>             
>> ?DynFileMail
>>     
>>>>>> # Log cron stuff
>>>>>> cron.*                                                  
>>>>>>             
>> ?DynFileCron
>>     
>>>>>> # Everybody gets emergency messages
>>>>>> *.emerg                                                 *
>>>>>>
>>>>>> # Save news errors of level crit and higher in a special file
>>>>>> uucp,news.crit                                          
>>>>>>             
>> ?DynFileSpool
>>     
>>>>>> # Save boot messages also to boot.log
>>>>>> local7.*                                                
>>>>>>             
>> ?DynFileBoot
>>     
>>>>>> # Relp config for PCI
>>>>>> $ModLoad imrelp
>>>>>> $InputRELPServerRun 2515
>>>>>>
>>>>>> # make gtls driver the default
>>>>>> $DefaultNetstreamDriver gtls
>>>>>>
>>>>>> # certificate files
>>>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
>>>>>> $DefaultNetstreamDriverCertFile /etc/ssl/servercrt.pem
>>>>>> $DefaultNetstreamDriverKeyFile /etc/ssl/serverkey.pem
>>>>>>
>>>>>> $ModLoad imtcp.so
>>>>>>
>>>>>> $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
>>>>>> $InputTCPServerStreamDriverAuthMode anon # client is NOT 
>>>>>>             
>> authenticated
>>     
>>>>>> $InputTCPServerRun 10514 # start up listener at port 10514
>>>>>> $InputTCPMaxSessions 500
>>>>>>
>>>>>>
>>>>>>         
>>>>>>             
>> --------------------------------------------------------------
>> --------------
>>     
>>>>     
>>>>         
>>>>> --------
>>>>>       
>>>>>           
>>>>>> Client config file
>>>>>>
>>>>>>         
>>>>>>             
>> --------------------------------------------------------------
>> --------------
>>     
>>>>     
>>>>         
>>>>> --------
>>>>>       
>>>>>           
>>>>>> $ModLoad imuxsock.so # provides support for local system 
>>>>>>             
>> logging (e.g.
>>     
>>>>>> via logger command)
>>>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
>>>>>>
>>>>>> $umask 0137
>>>>>> $DirCreateMode 0640
>>>>>> $FileCreateMode 0640
>>>>>> $FileOwner root
>>>>>> $FileGroup admin
>>>>>>
>>>>>>
>>>>>> # Log anything 'info' or higher, but lower than 'warn'.
>>>>>> # Exclude authpriv, cron, mail, and news.  These are 
>>>>>>             
>> logged elsewhere.
>>     
>>>>>> *.info;*.!warn;\
>>>>>>         authpriv.none;cron.none;mail.none;news.none     
>>>>>>             
>> /var/log/messages
>>     
>>>>>> # Log anything 'warn' or higher.
>>>>>> # Exclude authpriv, cron, mail, and news.  These are 
>>>>>>             
>> logged elsewhere.
>>     
>>>>>> *.warn;\
>>>>>>         authpriv.none;cron.none;mail.none;news.none     
>>>>>>             
>> /var/log/syslog
>>     
>>>>>> # Debugging information is logged here.
>>>>>> *.=debug                                                
>>>>>>             
>> /var/log/debug
>>     
>>>>>> # Private authentication message logging:
>>>>>> authpriv.*                                              
>>>>>>             
>> /var/log/secure
>>     
>>>>>> # Cron related logs:
>>>>>> cron.*                                                  
>>>>>>             
>> /var/log/cron
>>     
>>>>>> # Mail related logs:
>>>>>> mail.*                                                  
>>>>>>             
>> /var/log/maillog
>>     
>>>>>> # Emergency level messages go to all users:
>>>>>> *.emerg                                                 *
>>>>>>
>>>>>>
>>>>>> ## Uncomment these lines to use RELP instead for PCI 
>>>>>>             
>> compliance (stunnel
>>     
>>>>>> required)
>>>>>> #$ModLoad omrelp
>>>>>> #*.*;mail.none :omrelp:localhost:2515;RSYSLOG_ForwardFormat
>>>>>>
>>>>>> # certificate files - just CA for a client
>>>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
>>>>>>
>>>>>> # set up the action
>>>>>> $DefaultNetstreamDriver gtls # use gtls netstream driver
>>>>>> $ActionSendStreamDriverMode 1 # require TLS for the connection
>>>>>> $ActionSendStreamDriverAuthMode anon # server is NOT 
>>>>>>             
>> authenticated
>>     
>>>>>> *.* @@(o)ldap.nmsrv.com:10514 # send (all) messages
>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.comoften
>>>>>>         
>>>>>>             
>>>>> --
>>>>> Patrick Shen
>>>>> Operations Engineer
>>>>>
>>>>> net mobile AG Shanghai office
>>>>>
>>>>> T: +86  21 61350900 - 222
>>>>> F: +86  21 61350906
>>>>> M: +86  13122245730
>>>>>
>>>>> FE55 6A7F 3192 F359 C1DF  12F6 9078 58B4 57AF 0BE3
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>>       
>>>>>           
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>     
>>>>         
>>>   
>>>       
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
>>     
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>   


From rgerhards at hq.adiscon.com  Wed Jun 10 18:12:38 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 10 Jun 2009 18:12:38 +0200
Subject: [rsyslog] Possible bug? 100% cpu on a server thread
	and	the	client stops sending data
References: <4A15999D.3000909@gossamer-threads.com><4A1A37DE.8050004@net-m.de>	<9B6E2A8877C38245BFB15CC491A11DA702B0FB@GRFEXC.intern.adiscon.com><4A1A57A0.1060903@net-m.de>	<4A29978C.9080505@gossamer-threads.com><9B6E2A8877C38245BFB15CC491A11DA706F8BA@GRFEXC.intern.adiscon.com>
	<4A2E948D.4060809@gossamer-threads.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F8FD@GRFEXC.intern.adiscon.com>

Args, sorry.. I hadn't thought about the beta. Could you retry with the
current devel version? This is the one that potentially has a fix.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Nathan March
> Sent: Tuesday, June 09, 2009 6:58 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server thread and the
> client stops sending data
> 
> Confirmed as still happening on 4.1.7
> 
> - Nathan
> 
> Rainer Gerhards wrote:
> > Please give v4 a try, the bug is potentially fixed there. If it actually
is,
> > this gives me a clue of where to look at.
> >
> > Rainer
> >
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Nathan March
> >> Sent: Saturday, June 06, 2009 12:09 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server
> >> thread and the client stops sending data
> >>
> >> Any hope of this bug getting some attention soon? =)
> >>
> >> Thanks,
> >> Nathan
> >>
> >>
> >> Patrick Shen wrote:
> >>
> >>> Hi Rainer,
> >>>
> >>> Thanks for the quick reply.
> >>>
> >>> My environment is like below:
> >>>
> >>> CLIENT ---
> >>>          |
> >>>          | (TCP)
> >>>          --------> SERVER ----
> >>>                              |
> >>>                              | (RELP Relay)
> >>>                              --------------> SERVER Standby
> >>>
> >>> I've tested, even if the standby server is under high load,
> >>>
> >> one of our
> >>
> >>> clients won't accept any new network connections.
> >>>
> >>> Just for your information.
> >>>
> >>> Best regards,
> >>> Patrick
> >>>
> >>> Rainer Gerhards wrote:
> >>>
> >>>
> >>>> Hi all,
> >>>>
> >>>> thanks for the detailed info. Will look at it asap, but I
> >>>>
> >> am currently tied
> >>
> >>>> up with some other work...
> >>>>
> >>>> Please let me know anything else that may be relevant.
> >>>>
> >>>> Thanks,
> >>>> Rainer
> >>>>
> >>>>
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>>> bounces at lists.adiscon.com] On Behalf Of Patrick Shen
> >>>>> Sent: Monday, May 25, 2009 8:17 AM
> >>>>> To: rsyslog-users
> >>>>> Subject: Re: [rsyslog] Possible bug? 100% cpu on a server
> >>>>>
> >> thread and the
> >>
> >>>>> client stops sending data
> >>>>>
> >>>>> Hi Nathan,
> >>>>>
> >>>>> In my company, we have the similar symptom. We have 50+
> >>>>>
> >> servers which
> >>
> >>>>> send logs to the central logging servers. Both clients
> >>>>>
> >> and server use
> >>
> >>>>> rsyslog v3.20.5 right now. In the past when we use
> >>>>>
> >> v3.20.0, the symptom
> >>
> >>>>> is more common when server is under high load average.
> >>>>>
> >>>>> It's also weird for us, not all of clients were lost of
> >>>>>
> >> connections. We
> >>
> >>>>> have 2-3 clients which is running the same application
> >>>>>
> >> will looks like
> >>
> >>>>> hanging up when server is high-load. New ssh connection
> >>>>>
> >> is very slow at
> >>
> >>>>> that time. But the connected ssh session works normal (We
> >>>>>
> >> have some
> >>
> >>>>> prepared screen session on other clients which connect to
> >>>>>
> >> them via ssh).
> >>
> >>>>> When the clients are lost of connections. So far our
> >>>>>
> >> solution is to
> >>
> >>>>> restart rsyslog both on affected clients and server. Then
> >>>>>
> >> the clients
> >>
> >>>>> get back and look normal.
> >>>>>
> >>>>> We've suffered it for a while, but I didn't take chance
> >>>>>
> >> to analyze it
> >>
> >>>>> like you.
> >>>>>
> >>>>> Best regards,
> >>>>> Patrick
> >>>>>
> >>>>> Nathan March wrote:
> >>>>>
> >>>>>
> >>>>>> Having a weird issue here..... Testing out a new rsyslog
> >>>>>>
> >> deployment and
> >>
> >>>>>> I've got around 30 servers logging to one machine. On
> >>>>>>
> >> one of the clients
> >>
> >>>>>> we had an issue where ssh was mysteriously very slow to login and
> >>>>>> tracked it down to rsyslog causing issues.
> >>>>>>
> >> Simultaneously, one of the
> >>
> >>>>>> threads on the logging server spiked up to 100%. New log
> >>>>>>
> >> data from the
> >>
> >>>>>> client would stop showing up on the server at this poitn.
> >>>>>>
> >>>>>> On the client after attempting to gracefully stop rsyslog:
> >>>>>>
> >>>>>> vanguard etc # ps aux | grep -i rsyslog
> >>>>>> root      3764  0.0  0.1  5092 2564 ?        S    15:06   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >> /etc/rsyslog.conf
> >>
> >>>>>> root      3766  0.0  0.1  5092 2564 ?        S    15:06   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >> /etc/rsyslog.conf
> >>
> >>>>>> root      6203  0.0  0.0  1512  524 pts/3    S    15:09
> >>>>>>
> >>  0:00 grep -i
> >>
> >>>>>> rsyslog
> >>>>>> vanguard etc # strace -p 3764
> >>>>>> Process 3764 attached - interrupt to quit
> >>>>>> send(11,
> >>>>>>
> >> "\25\3\2\1\0uE\254\304\n\311\377\204}p\237O\211\322\211"...,
> >>
> >>>>>> 261, 0 <unfinished ...>
> >>>>>> Process 3764 detached
> >>>>>>
> >>>>>> FD 11 being the socket to the logging server, it never
> >>>>>>
> >> does anything
> >>
> >>>>>> asides from that send.
> >>>>>>
> >>>>>> On the server:
> >>>>>>
> >>>>>> ldap ~ # ps aux | grep rsyslog
> >>>>>> root      7370  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7379  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7380  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7381  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7382  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7383  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7384  0.0  0.0  83936  3380 ?        S    14:54   0:00
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> root      7385 28.3  0.0  83936  3380 ?        R    14:54   5:07
> >>>>>> /usr/sbin/rsyslogd -c3 -i /var/run/rsyslogd.pid -f
> >>>>>>
> >>>>>>
> >>>> /etc/rsyslog/rsyslog.conf
> >>>>
> >>>>
> >>>>>> Stracing the 7385 pid just shows this repeating over and
> >>>>>>
> >> over (FD 4
> >>
> >>>>>> being the network socket again):
> >>>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1
> >>>>>>
> >> EAGAIN (Resource
> >>
> >>>>>> temporarily unavailable)
> >>>>>> accept(4, 0x2b0f67658ca0, [7450515777377009792]) = -1
> >>>>>>
> >> EAGAIN (Resource
> >>
> >>>>>> temporarily unavailable)
> >>>>>>
> >>>>>> It happened 3 times within a relatively short period of
> >>>>>>
> >> time, sometimes
> >>
> >>>>>> within minutes of me restarting rsyslog on both
> >>>>>>
> >> machines. I enabled
> >>
> >>>>>> debugging and it took around 8 hours to pop up again.
> >>>>>>
> >>>>>> With debugging, I pulled this from the rsyslog server:
> >>>>>>
> >>>>>> 6640.408108000:imtcp.c: New connect on NSD 0x59ddc0.
> >>>>>> 6640.408129000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550:
> >>>>>>
> >> pszRcvBuf (nil),
> >>
> >>>>>> lenRcvBuf 0
> >>>>>> 6640.408144000:imtcp.c: hasRcvInBuffer on nsd 0x59dcd0: pszRcvBuf
> >>>>>> 0x5c11c0, lenRcvBuf -1
> >>>>>> 6640.408158000:imtcp.c: hasRcvInBuffer on nsd 0x59dc10: pszRcvBuf
> >>>>>> 0x5cb510, lenRcvBuf -1
> >>>>>> 6640.408171000:imtcp.c: hasRcvInBuffer on nsd 0x5c3d70: pszRcvBuf
> >>>>>> 0x5d45c0, lenRcvBuf -1
> >>>>>> 6640.408185000:imtcp.c: hasRcvInBuffer on nsd 0x5cf460: pszRcvBuf
> >>>>>> 0x5dd010, lenRcvBuf -1
> >>>>>> 6640.408199000:imtcp.c: hasRcvInBuffer on nsd 0x5d8600: pszRcvBuf
> >>>>>> 0x5e58b0, lenRcvBuf -1
> >>>>>> 6640.408213000:imtcp.c: hasRcvInBuffer on nsd 0x59d690: pszRcvBuf
> >>>>>> 0x5ee150, lenRcvBuf -1
> >>>>>> 6640.408227000:imtcp.c: hasRcvInBuffer on nsd 0x59d2f0: pszRcvBuf
> >>>>>> 0x5f7320, lenRcvBuf -1
> >>>>>> 6640.408241000:imtcp.c: hasRcvInBuffer on nsd 0x5e8840: pszRcvBuf
> >>>>>> 0x5fe0a0, lenRcvBuf -1
> >>>>>> 6640.408255000:imtcp.c: hasRcvInBuffer on nsd 0x6027b0: pszRcvBuf
> >>>>>> 0x608350, lenRcvBuf -1
> >>>>>> 6640.408269000:imtcp.c: hasRcvInBuffer on nsd 0x5c3660: pszRcvBuf
> >>>>>> 0x612110, lenRcvBuf -1
> >>>>>> 6640.408283000:imtcp.c: hasRcvInBuffer on nsd 0x602690: pszRcvBuf
> >>>>>> 0x615040, lenRcvBuf -1
> >>>>>> 6640.408296000:imtcp.c: hasRcvInBuffer on nsd 0x5c3b60: pszRcvBuf
> >>>>>> 0x6218f0, lenRcvBuf -1
> >>>>>> 6640.408310000:imtcp.c: hasRcvInBuffer on nsd 0x60c150: pszRcvBuf
> >>>>>> 0x62a190, lenRcvBuf -1
> >>>>>> 6640.408324000:imtcp.c: hasRcvInBuffer on nsd 0x625420: pszRcvBuf
> >>>>>> 0x634fe0, lenRcvBuf -1
> >>>>>> 6640.408338000:imtcp.c: hasRcvInBuffer on nsd 0x6117b0: pszRcvBuf
> >>>>>> 0x63b000, lenRcvBuf -1
> >>>>>> 6640.408352000:imtcp.c: hasRcvInBuffer on nsd 0x603990: pszRcvBuf
> >>>>>> 0x643c70, lenRcvBuf -1
> >>>>>> 6640.408365000:imtcp.c: hasRcvInBuffer on nsd 0x633c00: pszRcvBuf
> >>>>>> 0x64dbb0, lenRcvBuf -1
> >>>>>> 6640.408379000:imtcp.c: hasRcvInBuffer on nsd 0x62cfa0: pszRcvBuf
> >>>>>> 0x650960, lenRcvBuf -1
> >>>>>> 6640.408393000:imtcp.c: hasRcvInBuffer on nsd 0x633d10: pszRcvBuf
> >>>>>> 0x65dbc0, lenRcvBuf -1
> >>>>>> 6640.408407000:imtcp.c: hasRcvInBuffer on nsd 0x5f2860: pszRcvBuf
> >>>>>> 0x666020, lenRcvBuf -1
> >>>>>> 6640.408421000:imtcp.c: hasRcvInBuffer on nsd 0x64d250: pszRcvBuf
> >>>>>> 0x66e480, lenRcvBuf -1
> >>>>>> 6640.408435000:imtcp.c: hasRcvInBuffer on nsd 0x659b30: pszRcvBuf
> >>>>>> 0x676d20, lenRcvBuf 78
> >>>>>> 6640.408448000:imtcp.c: hasRcvInBuffer on nsd 0x669fb0: pszRcvBuf
> >>>>>> 0x67fdd0, lenRcvBuf -1
> >>>>>> 6640.408462000:imtcp.c: hasRcvInBuffer on nsd 0x59a630: pszRcvBuf
> >>>>>> 0x687f10, lenRcvBuf -1
> >>>>>> 6640.408476000:imtcp.c: hasRcvInBuffer on nsd 0x6614c0: pszRcvBuf
> >>>>>> 0x6907b0, lenRcvBuf -1
> >>>>>> 6640.408490000:imtcp.c: hasRcvInBuffer on nsd 0x60c2d0: pszRcvBuf
> >>>>>> 0x699050, lenRcvBuf -1
> >>>>>> 6640.408504000:imtcp.c: hasRcvInBuffer on nsd 0x5a0550:
> >>>>>>
> >> pszRcvBuf (nil),
> >>
> >>>>>> lenRcvBuf 0
> >>>>>>
> >>>>>> This repeats enough to generate around 29 million lines
> >>>>>>
> >> in the debug log
> >>
> >>>>>> file.
> >>>>>>
> >>>>>> In the client log, there's plenty of these which seems
> >>>>>>
> >> somewhat normal:
> >>
> >>>>>> 6509.743164000:imuxsock.c: --------imuxsock calling
> >>>>>>
> >> select, active file
> >>
> >>>>>> descriptors (max 11): 11
> >>>>>> 6509.743231000:imuxsock.c: Message from UNIX socket: #11
> >>>>>> 6509.743262000:imuxsock.c: logmsg: flags 4, from
> >>>>>>
> >> 'vanguard', msg May 20
> >>
> >>>>>> 23:15:09 bin/qmail-local[23198]: using .qmail .qmail-cfsc-forum
> >>>>>> 6509.743281000:imuxsock.c: Message has legacy syslog format.
> >>>>>> 6509.743302000:imuxsock.c: main queue: entry added, size
> >>>>>>
> >> now 579 entries
> >>
> >>>>>> 6509.743325000:imuxsock.c: wtpAdviseMaxWorkers signals busy
> >>>>>> 6509.743343000:imuxsock.c: main queue: EnqueueMsg
> >>>>>>
> >> advised worker start
> >>
> >>>>>> However comparing it to an earlier point in the log
> >>>>>>
> >> there's none of the
> >>
> >>>>>> associated tcp queue entries like this:
> >>>>>>
> >>>>>> 3587.234178000:main queue:Reg/w0: TCP sent 107 bytes,
> >>>>>>
> >> requested 107
> >>
> >>>>>> 3587.234204000:main queue:Reg/w0: main queue: entering
> >>>>>>
> >> rate limiter
> >>
> >>>>>> 3587.234229000:main queue:Reg/w0: main queue: entry
> >>>>>>
> >> deleted, state 0,
> >>
> >>>>>> size now 0 entries
> >>>>>> 3587.234255000:main queue:Reg/w0: Called action, logging
> >>>>>>
> >> to builtin-file
> >>
> >>>>>> 3587.234285000:main queue:Reg/w0:  (/var/log/maillog)
> >>>>>> 3587.234314000:main queue:Reg/w0: Called action, logging
> >>>>>>
> >> to builtin-fwd
> >>
> >>>>>> 3587.234340000:main queue:Reg/w0:  ldap.nmsrv.com
> >>>>>> 3587.234359000:main queue:Reg/w0:  ldap.nmsrv.com:10514/tcp
> >>>>>> 3587.234425000:main queue:Reg/w0: TCP sent 78 bytes, requested 78
> >>>>>> 3587.234446000:main queue:Reg/w0: main queue: entering
> >>>>>>
> >> rate limiter
> >>
> >>>>>> 3587.234471000:main queue:Reg/w0: main queue:Reg/w0: worker IDLE,
> >>>>>> waiting for work.
> >>>>>>
> >>>>>> Anyone able to shed some light on this situation? I've
> >>>>>>
> >> got the full
> >>
> >>>>>> debug logs if it's useful to anyone to take a closer look.
> >>>>>>
> >>>>>> Thanks!
> >>>>>>
> >>>>>> - Nathan
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >> --------------------------------------------------------------
> >> --------------
> >>
> >>>>
> >>>>
> >>>>> --------
> >>>>>
> >>>>>
> >>>>>> Server config file
> >>>>>>
> >>>>>>
> >>>>>>
> >> --------------------------------------------------------------
> >> --------------
> >>
> >>>>
> >>>>
> >>>>> --------
> >>>>>
> >>>>>
> >>>>>> $ModLoad immark.so # provides --MARK-- message capability
> >>>>>> $ModLoad imuxsock.so # provides support for local system
> >>>>>>
> >> logging (e.g.
> >>
> >>>>>> via logger command)
> >>>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> >>>>>>
> >>>>>> $umask 0137
> >>>>>> $DirCreateMode 0640
> >>>>>> $FileCreateMode 0640
> >>>>>> $FileOwner root
> >>>>>> $FileGroup admin
> >>>>>>
> >>>>>> $template DynFile,"/var/log/rsyslog/%HOSTNAME%/messages"
> >>>>>> $template DynFileCron,"/var/log/rsyslog/%HOSTNAME%/cron"
> >>>>>> $template DynFileSecure,"/var/log/rsyslog/%HOSTNAME%/secure"
> >>>>>> $template DynFileMail,"/var/log/rsyslog/%HOSTNAME%/maillog"
> >>>>>> $template DynFileSpool,"/var/log/rsyslog/%HOSTNAME%/spooler"
> >>>>>> $template DynFileBoot,"/var/log/rsyslog/%HOSTNAME%/boot"
> >>>>>> $template DynFileSyslog,"/var/log/rsyslog/%HOSTNAME%/syslog"
> >>>>>>
> >>>>>> :msg, contains, "no keys found for"            ~
> >>>>>> :msg, contains, "session opened for user"        ~
> >>>>>>
> >>>>>> # Log all kernel messages to the console.
> >>>>>> # Logging much else clutters up the screen.
> >>>>>> kern.*
> >>>>>>
> >> /dev/console
> >>
> >>>>>> # Log anything (except mail) of level info or higher.
> >>>>>> # Don't log private authentication messages!
> >>>>>> *.info;mail.none;cron.none                     ?DynFile
> >>>>>>
> >>>>>> *.warn;\
> >>>>>>         authpriv.none;cron.none;mail.none;news.none
> >>>>>>
> >> ?DynFileSyslog
> >>
> >>>>>> # Log all the mail messages in one place.
> >>>>>> mail.*
> >>>>>>
> >> ?DynFileMail
> >>
> >>>>>> # Log cron stuff
> >>>>>> cron.*
> >>>>>>
> >> ?DynFileCron
> >>
> >>>>>> # Everybody gets emergency messages
> >>>>>> *.emerg                                                 *
> >>>>>>
> >>>>>> # Save news errors of level crit and higher in a special file
> >>>>>> uucp,news.crit
> >>>>>>
> >> ?DynFileSpool
> >>
> >>>>>> # Save boot messages also to boot.log
> >>>>>> local7.*
> >>>>>>
> >> ?DynFileBoot
> >>
> >>>>>> # Relp config for PCI
> >>>>>> $ModLoad imrelp
> >>>>>> $InputRELPServerRun 2515
> >>>>>>
> >>>>>> # make gtls driver the default
> >>>>>> $DefaultNetstreamDriver gtls
> >>>>>>
> >>>>>> # certificate files
> >>>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
> >>>>>> $DefaultNetstreamDriverCertFile /etc/ssl/servercrt.pem
> >>>>>> $DefaultNetstreamDriverKeyFile /etc/ssl/serverkey.pem
> >>>>>>
> >>>>>> $ModLoad imtcp.so
> >>>>>>
> >>>>>> $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
> >>>>>> $InputTCPServerStreamDriverAuthMode anon # client is NOT
> >>>>>>
> >> authenticated
> >>
> >>>>>> $InputTCPServerRun 10514 # start up listener at port 10514
> >>>>>> $InputTCPMaxSessions 500
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >> --------------------------------------------------------------
> >> --------------
> >>
> >>>>
> >>>>
> >>>>> --------
> >>>>>
> >>>>>
> >>>>>> Client config file
> >>>>>>
> >>>>>>
> >>>>>>
> >> --------------------------------------------------------------
> >> --------------
> >>
> >>>>
> >>>>
> >>>>> --------
> >>>>>
> >>>>>
> >>>>>> $ModLoad imuxsock.so # provides support for local system
> >>>>>>
> >> logging (e.g.
> >>
> >>>>>> via logger command)
> >>>>>> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> >>>>>>
> >>>>>> $umask 0137
> >>>>>> $DirCreateMode 0640
> >>>>>> $FileCreateMode 0640
> >>>>>> $FileOwner root
> >>>>>> $FileGroup admin
> >>>>>>
> >>>>>>
> >>>>>> # Log anything 'info' or higher, but lower than 'warn'.
> >>>>>> # Exclude authpriv, cron, mail, and news.  These are
> >>>>>>
> >> logged elsewhere.
> >>
> >>>>>> *.info;*.!warn;\
> >>>>>>         authpriv.none;cron.none;mail.none;news.none
> >>>>>>
> >> /var/log/messages
> >>
> >>>>>> # Log anything 'warn' or higher.
> >>>>>> # Exclude authpriv, cron, mail, and news.  These are
> >>>>>>
> >> logged elsewhere.
> >>
> >>>>>> *.warn;\
> >>>>>>         authpriv.none;cron.none;mail.none;news.none
> >>>>>>
> >> /var/log/syslog
> >>
> >>>>>> # Debugging information is logged here.
> >>>>>> *.=debug
> >>>>>>
> >> /var/log/debug
> >>
> >>>>>> # Private authentication message logging:
> >>>>>> authpriv.*
> >>>>>>
> >> /var/log/secure
> >>
> >>>>>> # Cron related logs:
> >>>>>> cron.*
> >>>>>>
> >> /var/log/cron
> >>
> >>>>>> # Mail related logs:
> >>>>>> mail.*
> >>>>>>
> >> /var/log/maillog
> >>
> >>>>>> # Emergency level messages go to all users:
> >>>>>> *.emerg                                                 *
> >>>>>>
> >>>>>>
> >>>>>> ## Uncomment these lines to use RELP instead for PCI
> >>>>>>
> >> compliance (stunnel
> >>
> >>>>>> required)
> >>>>>> #$ModLoad omrelp
> >>>>>> #*.*;mail.none :omrelp:localhost:2515;RSYSLOG_ForwardFormat
> >>>>>>
> >>>>>> # certificate files - just CA for a client
> >>>>>> $DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
> >>>>>>
> >>>>>> # set up the action
> >>>>>> $DefaultNetstreamDriver gtls # use gtls netstream driver
> >>>>>> $ActionSendStreamDriverMode 1 # require TLS for the connection
> >>>>>> $ActionSendStreamDriverAuthMode anon # server is NOT
> >>>>>>
> >> authenticated
> >>
> >>>>>> *.* @@(o)ldap.nmsrv.com:10514 # send (all) messages
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> rsyslog mailing list
> >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>> http://www.rsyslog.comoften
> >>>>>>
> >>>>>>
> >>>>> --
> >>>>> Patrick Shen
> >>>>> Operations Engineer
> >>>>>
> >>>>> net mobile AG Shanghai office
> >>>>>
> >>>>> T: +86  21 61350900 - 222
> >>>>> F: +86  21 61350906
> >>>>> M: +86  13122245730
> >>>>>
> >>>>> FE55 6A7F 3192 F359 C1DF  12F6 9078 58B4 57AF 0BE3
> >>>>>
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>>>
> >>>>>
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>>>
> >>>>
> >>>
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Mon Jun 15 16:48:23 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 16:48:23 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>

Hi all,

I am having a hard time finding a suitable-fast but tickles solution to
obtaining low-precision time API under Linux. Please read here:

http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html

If I don't find a better solution, I'll probably be forced to run rsyslogd on
a tick, which would not be a good thing from a power consumption point of
view.

Comments and suggestions are highly appreciated.

Thanks,
Rainer


From epiphani at gmail.com  Mon Jun 15 16:55:57 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 15 Jun 2009 10:55:57 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
Message-ID: <e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>

I'm slightly confused - whats wrong with gettimeofday()?  It may be
higher precision, but it still
gives you the ability to find the nearest second with a few extra operations.

-Aaron

On Mon, Jun 15, 2009 at 10:48 AM, Rainer
Gerhards<rgerhards at hq.adiscon.com> wrote:
> Hi all,
>
> I am having a hard time finding a suitable-fast but tickles solution to
> obtaining low-precision time API under Linux. Please read here:
>
> http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html
>
> If I don't find a better solution, I'll probably be forced to run rsyslogd on
> a tick, which would not be a good thing from a power consumption point of
> view.
>
> Comments and suggestions are highly appreciated.
>
> Thanks,
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Mon Jun 15 16:59:21 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 16:59:21 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F937@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> Sent: Monday, June 15, 2009 4:56 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> I'm slightly confused - whats wrong with gettimeofday()?  It may be
> higher precision, but it still
> gives you the ability to find the nearest second with a few extra
operations.
> 
> -Aaron

Much too slow. For example, you can often write a sector quicker to disk than
gettimeofday() returns...

If I would base buffer-writing optimizations on gettimeofday(), the
gettimeofday() would eat up much more time than I can save with the
optimization.

Rainer

> 
> On Mon, Jun 15, 2009 at 10:48 AM, Rainer
> Gerhards<rgerhards at hq.adiscon.com> wrote:
> > Hi all,
> >
> > I am having a hard time finding a suitable-fast but tickles solution to
> > obtaining low-precision time API under Linux. Please read here:
> >
> >
http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html
> >
> > If I don't find a better solution, I'll probably be forced to run
rsyslogd
> on
> > a tick, which would not be a good thing from a power consumption point of
> > view.
> >
> > Comments and suggestions are highly appreciated.
> >
> > Thanks,
> > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Mon Jun 15 17:10:20 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 17:10:20 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>

I have updated the blogpost to hopefully provide a better description of the
situation:

http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> Sent: Monday, June 15, 2009 4:56 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> I'm slightly confused - whats wrong with gettimeofday()?  It may be
> higher precision, but it still
> gives you the ability to find the nearest second with a few extra
operations.
> 
> -Aaron
> 
> On Mon, Jun 15, 2009 at 10:48 AM, Rainer
> Gerhards<rgerhards at hq.adiscon.com> wrote:
> > Hi all,
> >
> > I am having a hard time finding a suitable-fast but tickles solution to
> > obtaining low-precision time API under Linux. Please read here:
> >
> >
http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html
> >
> > If I don't find a better solution, I'll probably be forced to run
rsyslogd
> on
> > a tick, which would not be a good thing from a power consumption point of
> > view.
> >
> > Comments and suggestions are highly appreciated.
> >
> > Thanks,
> > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From jmoyer at redhat.com  Mon Jun 15 17:13:16 2009
From: jmoyer at redhat.com (Jeff Moyer)
Date: Mon, 15 Jun 2009 11:13:16 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F937@GRFEXC.intern.adiscon.com>
	(Rainer Gerhards's message of "Mon, 15 Jun 2009 16:59:21 +0200")
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F937@GRFEXC.intern.adiscon.com>
Message-ID: <x49y6rtlecz.fsf@segfault.boston.devel.redhat.com>

"Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
>> Sent: Monday, June 15, 2009 4:56 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] high-performance, low-precision time API under
> linux?
>> 
>> I'm slightly confused - whats wrong with gettimeofday()?  It may be
>> higher precision, but it still
>> gives you the ability to find the nearest second with a few extra
> operations.
>> 
>> -Aaron
>
> Much too slow. For example, you can often write a sector quicker to disk than
> gettimeofday() returns...

Really?  That doesn't sound right to me.  Writing a sector to disk can
take 8ms or more.  One would hope gettimeofday would be faster than
that, given that it reports microseconds.  ;)  The exact resolution, of
course, depends on how your kernel is configured.  Here are the
measurements (in CPU cycles) I get running each call (time(NULL) and
gettimeofday(&tv, NULL)) 1000 times:

$ ./some-cycle-costs 
                                    fewest cycles  avg cycles
syscall(NR_time)                    2134           2721.99
syscall(NR_gettimeofday)            2156           2622.87

If I understand your original question properly, you don't need to
always have a timer running.  You can only schedule a timer when there
is I/O buffered.  I'm not sure whether your code lends itself to that or
not.

I would be interested in hearing more about what conditions cause
gettimeofday to be so slow.

Cheers,
Jeff

>> On Mon, Jun 15, 2009 at 10:48 AM, Rainer
>> Gerhards<rgerhards at hq.adiscon.com> wrote:
>> > Hi all,
>> >
>> > I am having a hard time finding a suitable-fast but tickles solution to
>> > obtaining low-precision time API under Linux. Please read here:
>> >
>> >
> http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html
>> >
>> > If I don't find a better solution, I'll probably be forced to run
> rsyslogd
>> on
>> > a tick, which would not be a good thing from a power consumption point of
>> > view.
>> >
>> > Comments and suggestions are highly appreciated.
>> >
>> > Thanks,
>> > Rainer


From epiphani at gmail.com  Mon Jun 15 17:32:13 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 15 Jun 2009 11:32:13 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
Message-ID: <e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>

I went over your post...

You shouldn't be seeing a 4ms response time on a gettimeofday() call -
this is my test:

gettimeofday({1245079355, 445434}, NULL) = 0 <0.000004>
gettimeofday({1245079355, 445503}, NULL) = 0 <0.000004>
gettimeofday({1245079355, 445560}, NULL) = 0 <0.000003>
gettimeofday({1245079355, 445629}, NULL) = 0 <0.000004>

That's 3-4 microseconds.

For the sake of argument, I checked time() as well, which is obsoleted
by gettimeofday()...

ime(NULL)                              = 1245079501 <0.000008>
time(NULL)                              = 1245079501 <0.000007>
time(NULL)                              = 1245079501 <0.000004>
time(NULL)                              = 1245079501 <0.000008>
time(NULL)                              = 1245079501 <0.000004>

About half the speed, but still in the microsecond range.

gettimeofday() is one of the fastest syscalls you can issue.
Obviously you don't want to beat on it, but I wouldn't worry about
calling it a few dozen times per second.

-Aaron


From rgerhards at hq.adiscon.com  Mon Jun 15 17:36:22 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 17:36:22 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F937@GRFEXC.intern.adiscon.com>
	<x49y6rtlecz.fsf@segfault.boston.devel.redhat.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F93B@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Jeff Moyer
> Sent: Monday, June 15, 2009 5:13 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> "Rainer Gerhards" <rgerhards at hq.adiscon.com> writes:
> 
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> >> Sent: Monday, June 15, 2009 4:56 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] high-performance, low-precision time API under
> > linux?
> >>
> >> I'm slightly confused - whats wrong with gettimeofday()?  It may be
> >> higher precision, but it still
> >> gives you the ability to find the nearest second with a few extra
> > operations.
> >>
> >> -Aaron
> >
> > Much too slow. For example, you can often write a sector quicker to disk
> than
> > gettimeofday() returns...
> 
> Really?  That doesn't sound right to me.  Writing a sector to disk can
> take 8ms or more.  One would hope gettimeofday would be faster than
> that, given that it reports microseconds.  ;)  The exact resolution, of
> course, depends on how your kernel is configured.  Here are the
> measurements (in CPU cycles) I get running each call (time(NULL) and
> gettimeofday(&tv, NULL)) 1000 times:
> 
> $ ./some-cycle-costs
>                                     fewest cycles  avg cycles
> syscall(NR_time)                    2134           2721.99
> syscall(NR_gettimeofday)            2156           2622.87

I've updated the blog post with my test program and some numbers:

http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html

(sorry for keeping this on the blog, but that makes it far easier to also ask
at other places...)

> If I understand your original question properly, you don't need to
> always have a timer running.  You can only schedule a timer when there
> is I/O buffered.  I'm not sure whether your code lends itself to that or
> not.

The code in question would not just be used for buffered writes. The problem
is also that I need to frequently re-set the timer, because it must not write
data if any new has arrived. In order to do this re-set handling properly, I
also need to know the system time.

Of course, I can schedule a specific timer while data is unwritten. But that
is just another tick processing if I don't reset the timer when there is no
need. But... all suggestions are welcome. The overall idea is that I would
like to implement something like the original callouot table. However,
callout worked on a tick. I intended to replace that with timers (more
precisely their pthreads-equivalent, which would give me some aids in awaking
the system in unexpected cases). That was the plan, but I overlooked that it
required many of the costly time() calls.

Rainer

> I would be interested in hearing more about what conditions cause
> gettimeofday to be so slow.
> 
> Cheers,
> Jeff
> 
> >> On Mon, Jun 15, 2009 at 10:48 AM, Rainer
> >> Gerhards<rgerhards at hq.adiscon.com> wrote:
> >> > Hi all,
> >> >
> >> > I am having a hard time finding a suitable-fast but tickles solution
to
> >> > obtaining low-precision time API under Linux. Please read here:
> >> >
> >> >
> >
http://blog.gerhards.net/2009/06/high-peformance-low-precision-time-api.html
> >> >
> >> > If I don't find a better solution, I'll probably be forced to run
> > rsyslogd
> >> on
> >> > a tick, which would not be a good thing from a power consumption point
of
> >> > view.
> >> >
> >> > Comments and suggestions are highly appreciated.
> >> >
> >> > Thanks,
> >> > Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From epiphani at gmail.com  Mon Jun 15 17:36:29 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 15 Jun 2009 11:36:29 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
Message-ID: <e7ca40f70906150836h38d3580fnb42aeef002145199@mail.gmail.com>

As more followup...

I would suggest running a timer per thread.  I assume every thread
would contain some type of
event-trigger loop, and I would make your timer calls in that loop,
caching for the remainder of
that iteration.  I'm not quite sure how well this would work, but you
can most certainly get away
with a fair bit of drift, especially if high-precision timers are not
required for log output.

-Aaron

On Mon, Jun 15, 2009 at 11:32 AM, Aaron Wiebe<epiphani at gmail.com> wrote:
> I went over your post...
>
> You shouldn't be seeing a 4ms response time on a gettimeofday() call -
> this is my test:
>
> gettimeofday({1245079355, 445434}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445503}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445560}, NULL) = 0 <0.000003>
> gettimeofday({1245079355, 445629}, NULL) = 0 <0.000004>
>
> That's 3-4 microseconds.
>
> For the sake of argument, I checked time() as well, which is obsoleted
> by gettimeofday()...
>
> ime(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000008>
> time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000007>
> time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000004>
> time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000008>
> time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000004>
>
> About half the speed, but still in the microsecond range.
>
> gettimeofday() is one of the fastest syscalls you can issue.
> Obviously you don't want to beat on it, but I wouldn't worry about
> calling it a few dozen times per second.
>
> -Aaron
>


From rgerhards at hq.adiscon.com  Mon Jun 15 17:45:55 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 17:45:55 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
	<e7ca40f70906150836h38d3580fnb42aeef002145199@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F93C@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> Sent: Monday, June 15, 2009 5:36 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> As more followup...
> 
> I would suggest running a timer per thread.  I assume every thread
> would contain some type of
> event-trigger loop, and I would make your timer calls in that loop,
> caching for the remainder of
> that iteration.  I'm not quite sure how well this would work, but you
> can most certainly get away
> with a fair bit of drift, especially if high-precision timers are not
> required for log output.

But that makes me running on that tick...

(more below)

> 
> -Aaron
> 
> On Mon, Jun 15, 2009 at 11:32 AM, Aaron Wiebe<epiphani at gmail.com> wrote:
> > I went over your post...
> >
> > You shouldn't be seeing a 4ms response time on a gettimeofday() call -
> > this is my test:
> >
> > gettimeofday({1245079355, 445434}, NULL) = 0 <0.000004>
> > gettimeofday({1245079355, 445503}, NULL) = 0 <0.000004>
> > gettimeofday({1245079355, 445560}, NULL) = 0 <0.000003>
> > gettimeofday({1245079355, 445629}, NULL) = 0 <0.000004>
> >
> > That's 3-4 microseconds.

Oops, sorry, yes, that is what I meant...

> > For the sake of argument, I checked time() as well, which is obsoleted
> > by gettimeofday()...
> >
> > ime(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000008>
> > time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000007>
> > time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000004>
> > time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000008>
> > time(NULL) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?= 1245079501 <0.000004>
> >
> > About half the speed, but still in the microsecond range.
> >
> > gettimeofday() is one of the fastest syscalls you can issue.
> > Obviously you don't want to beat on it, but I wouldn't worry about
> > calling it a few dozen times per second.

It's not about a few dozen times. I need to re-set the timer when I logically
write something to the file. The problem is that I don't know how long it was
since I did this the last time. So I would end up with calling time once per
logical write, that is a in the range of tenthousand to hunderedthousand
times per second.

A solution for this case may be to not reset any started timer and simply let
it fire whenever that be. So if there is a time started, don't care about the
other writes but do this only once the timer is done. That, of course, comes
at the cost of thread synchronization. Again, atomics may help here. And
while it is tempting to assume atomic writes for simple 32 bit values, this
is not always the case. So in the extreme case, and without (reliable)
atomics, one may end up with timers not being started where they should be.
Using sync primitives, however, is also very costly...

Rainer
> >
> > -Aaron
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From epiphani at gmail.com  Mon Jun 15 17:57:38 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 15 Jun 2009 11:57:38 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F93C@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
	<e7ca40f70906150836h38d3580fnb42aeef002145199@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F93C@GRFEXC.intern.adiscon.com>
Message-ID: <e7ca40f70906150857r53cd1b00yc0a748ba226c3a7c@mail.gmail.com>

On Mon, Jun 15, 2009 at 11:45 AM, Rainer
Gerhards<rgerhards at hq.adiscon.com> wrote:
>
> It's not about a few dozen times. I need to re-set the timer when I logically
> write something to the file. The problem is that I don't know how long it was
> since I did this the last time. So I would end up with calling time once per
> logical write, that is a in the range of tenthousand to hunderedthousand
> times per second.

I'll suggest this first, since it has the possibility of being the
best solution:  posix aio.  Have a look at `man 3 aio_write'.  That
should reduce your write() latency enough to not worry about IO
blocking and messing with the clock reasonably well.  I've used this
method in the past, and it generally works wonders - plus its not too
hard to implement.  The downsize is that it isn't very portable.

Otherwise, you're looking at batched writes and timer resets after the
batch completes.  There is nothing required in terms of delivering
messages to a file within a few seconds, so you could batch up writes
up to a high watermark / specified time prior to starting a write
cycle.

-Aaron


From rgerhards at hq.adiscon.com  Mon Jun 15 18:10:40 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 18:10:40 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com><e7ca40f70906150836h38d3580fnb42aeef002145199@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F93C@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150857r53cd1b00yc0a748ba226c3a7c@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F93D@GRFEXC.intern.adiscon.com>

OK, I've also had some other discussions. Looks like there is no quicker way
to do it. So I think I'll ditch the approach to have a generic capability to
do timeout processing and rather do specific code for each use case. As you
write, this is no problem for the file writer. I'll probably go to write
every 10 seconds or so (if there is a buffer flush request pending). I'll
evaluate the other use cases when I come along them.

Thanks everyone for commenting :)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe
> Sent: Monday, June 15, 2009 5:58 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> On Mon, Jun 15, 2009 at 11:45 AM, Rainer
> Gerhards<rgerhards at hq.adiscon.com> wrote:
> >
> > It's not about a few dozen times. I need to re-set the timer when I
> logically
> > write something to the file. The problem is that I don't know how long it
> was
> > since I did this the last time. So I would end up with calling time once
per
> > logical write, that is a in the range of tenthousand to hunderedthousand
> > times per second.
> 
> I'll suggest this first, since it has the possibility of being the
> best solution:  posix aio.  Have a look at `man 3 aio_write'.  That
> should reduce your write() latency enough to not worry about IO
> blocking and messing with the clock reasonably well.  I've used this
> method in the past, and it generally works wonders - plus its not too
> hard to implement.  The downsize is that it isn't very portable.
> 
> Otherwise, you're looking at batched writes and timer resets after the
> batch completes.  There is nothing required in terms of delivering
> messages to a file within a few seconds, so you could batch up writes
> up to a high watermark / specified time prior to starting a write
> cycle.
> 
> -Aaron
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From david at lang.hm  Mon Jun 15 19:01:01 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 15 Jun 2009 10:01:01 -0700 (PDT)
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906150956580.9159@asgard>

On Mon, 15 Jun 2009, Aaron Wiebe wrote:

> I went over your post...
>
> You shouldn't be seeing a 4ms response time on a gettimeofday() call -
> this is my test:
>
> gettimeofday({1245079355, 445434}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445503}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445560}, NULL) = 0 <0.000003>
> gettimeofday({1245079355, 445629}, NULL) = 0 <0.000004>
>
> That's 3-4 microseconds.
>
> For the sake of argument, I checked time() as well, which is obsoleted
> by gettimeofday()...
>
> ime(NULL)                              = 1245079501 <0.000008>
> time(NULL)                              = 1245079501 <0.000007>
> time(NULL)                              = 1245079501 <0.000004>
> time(NULL)                              = 1245079501 <0.000008>
> time(NULL)                              = 1245079501 <0.000004>
>
> About half the speed, but still in the microsecond range.
>
> gettimeofday() is one of the fastest syscalls you can issue.
> Obviously you don't want to beat on it, but I wouldn't worry about
> calling it a few dozen times per second.

this depends greatly on your specific system. linux has some wonderful 
speedups available (basicly it internally does updates only when time 
changes and the system call gets optimized to a read from memory), but 
this is not available on all systems and requires support from glibc. I 
don't remember if the limit is the CPU, 32 bit address space, kernel 
config options, or something else.

I remember seeing that without this kernel optimization a very fast 
pentium (3GHz+) took more than 1 microsecond to do a gettimeofday() call 
(tested by doing 1m calls and it took more than a second)

David Lang


From david at lang.hm  Mon Jun 15 19:12:01 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 15 Jun 2009 10:12:01 -0700 (PDT)
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906151007200.9159@asgard>

On Mon, 15 Jun 2009, Aaron Wiebe wrote:

> I went over your post...
>
> You shouldn't be seeing a 4ms response time on a gettimeofday() call -
> this is my test:
>
> gettimeofday({1245079355, 445434}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445503}, NULL) = 0 <0.000004>
> gettimeofday({1245079355, 445560}, NULL) = 0 <0.000003>
> gettimeofday({1245079355, 445629}, NULL) = 0 <0.000004>
>
> That's 3-4 microseconds.
>
> For the sake of argument, I checked time() as well, which is obsoleted
> by gettimeofday()...
>
> ime(NULL)                              = 1245079501 <0.000008>
> time(NULL)                              = 1245079501 <0.000007>
> time(NULL)                              = 1245079501 <0.000004>
> time(NULL)                              = 1245079501 <0.000008>
> time(NULL)                              = 1245079501 <0.000004>
>
> About half the speed, but still in the microsecond range.
>
> gettimeofday() is one of the fastest syscalls you can issue.
> Obviously you don't want to beat on it, but I wouldn't worry about
> calling it a few dozen times per second.

Aaron,

rsyslog is able to recieve (not fully process, but recieve into the queue) 
over 300,000 messages/second from UDP without dropping any packets (256 
byte messages over gig ethernet), without maxing out the CPU on the thread 
recieving the messages. In the next week or so I hope to be able to do 
some tests with 10G ethernet ;-)

with 4 microseconds per gettimeofday() call, you can only do 250,000 
gettimeofday() calls per second. just doing a single call per message will 
take more time than you have.

David Lang


From epiphani at gmail.com  Mon Jun 15 19:19:49 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 15 Jun 2009 13:19:49 -0400
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <alpine.DEB.1.10.0906151007200.9159@asgard>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com>
	<e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com>
	<alpine.DEB.1.10.0906151007200.9159@asgard>
Message-ID: <e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com>

Greetings David,

On Mon, Jun 15, 2009 at 1:12 PM, <david at lang.hm> wrote:

>> gettimeofday() is one of the fastest syscalls you can issue.
>> Obviously you don't want to beat on it, but I wouldn't worry about
>> calling it a few dozen times per second.
>
> Aaron,
>
> rsyslog is able to recieve (not fully process, but recieve into the queue)
> over 300,000 messages/second from UDP without dropping any packets (256
> byte messages over gig ethernet), without maxing out the CPU on the thread
> recieving the messages. In the next week or so I hope to be able to do
> some tests with 10G ethernet ;-)
>
> with 4 microseconds per gettimeofday() call, you can only do 250,000
> gettimeofday() calls per second. just doing a single call per message will
> take more time than you have.

I never meant to suggest that doing a gettimeofday() on every received
message made sense.  My point was that a gettimeofday() call is a very
lightweight system call - one of the lightest.  Using a light system
call is of course no replacement for proper application design, and
Rainers' question was a valid one.  Unfortunately, there isn't much in
the way of changes to the system calls being used that would improve
the execution.  (Though dropping in gettimeofday() instead of time()
would probably be a good idea, since time() is documented as being
obsolete to gettimeofday().)

-Aaron


From Thomas.Mieslinger at 1und1.de  Mon Jun 15 19:57:43 2009
From: Thomas.Mieslinger at 1und1.de (Thomas Mieslinger)
Date: Mon, 15 Jun 2009 19:57:43 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com><alpine.DEB.1.10.0906151007200.9159@asgard>
	<e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com>
Message-ID: <3DBDD4313FFFEA4F86E79AEF498136C3079B636D@WEBEXCHANGE.webde.local>

BTW: In the Informix DBMS the Software Architects decided to implement an OS independent timer thread to work arround (very costly in the late 90s) gettimeofday() calls.

I've run the code on some Solaris 10 boxes, because they were known for their slow gettimeofday() implementation

S11 snv_105 Pentium 4 3GHz 
time ./a.out 100000
real    0m0.087s
user    0m0.082s
sys     0m0.004s

S10 Generic_138889-03 Athlon 2,2GHz
time ./a.out 100000
real    0m0.028s
user    0m0.026s
sys     0m0.002s

ok, lets try this on an older boxes...
suse70 2.2.21 libc-2.1.3-141 2xP3 800
time ./a.out 100000

real    0m0.100s
user    0m0.080s
sys     0m0.020s

So, what are you using as a development box?


From rgerhards at hq.adiscon.com  Mon Jun 15 20:23:17 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 20:23:17 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com><alpine.DEB.1.10.0906151007200.9159@asgard><e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com>
	<3DBDD4313FFFEA4F86E79AEF498136C3079B636D@WEBEXCHANGE.webde.local>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F93E@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Thomas Mieslinger
> Sent: Monday, June 15, 2009 7:58 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> BTW: In the Informix DBMS the Software Architects decided to implement an
OS
> independent timer thread to work arround (very costly in the late 90s)
> gettimeofday() calls.
> 
> I've run the code on some Solaris 10 boxes, because they were known for
their
> slow gettimeofday() implementation
> 
> S11 snv_105 Pentium 4 3GHz
> time ./a.out 100000
> real    0m0.087s
> user    0m0.082s
> sys     0m0.004s
> 
> S10 Generic_138889-03 Athlon 2,2GHz
> time ./a.out 100000
> real    0m0.028s
> user    0m0.026s
> sys     0m0.002s
> 
> ok, lets try this on an older boxes...
> suse70 2.2.21 libc-2.1.3-141 2xP3 800
> time ./a.out 100000
> 
> real    0m0.100s
> user    0m0.080s
> sys     0m0.020s
> 
> So, what are you using as a development box?

I should have mentioned that I am typically running inside a virtualized
environment as I routinely need to run three to four machines in parallel for
testing.

If anyone is up for hardware contributions, just let me know ;)

Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From david at lang.hm  Mon Jun 15 20:27:15 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 15 Jun 2009 11:27:15 -0700 (PDT)
Subject: [rsyslog] high-performance, low-precision time API under linux?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F93E@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com><alpine.DEB.1.10.0906151007200.9159@asgard><e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com>
	<3DBDD4313FFFEA4F86E79AEF498136C3079B636D@WEBEXCHANGE.webde.local>
	<9B6E2A8877C38245BFB15CC491A11DA706F93E@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906151126180.9159@asgard>

On Mon, 15 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Thomas Mieslinger
>> Sent: Monday, June 15, 2009 7:58 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] high-performance, low-precision time API under
> linux?
>>
>> BTW: In the Informix DBMS the Software Architects decided to implement an
> OS
>> independent timer thread to work arround (very costly in the late 90s)
>> gettimeofday() calls.
>>
>> I've run the code on some Solaris 10 boxes, because they were known for
> their
>> slow gettimeofday() implementation
>>
>> S11 snv_105 Pentium 4 3GHz
>> time ./a.out 100000
>> real    0m0.087s
>> user    0m0.082s
>> sys     0m0.004s
>>
>> S10 Generic_138889-03 Athlon 2,2GHz
>> time ./a.out 100000
>> real    0m0.028s
>> user    0m0.026s
>> sys     0m0.002s
>>
>> ok, lets try this on an older boxes...
>> suse70 2.2.21 libc-2.1.3-141 2xP3 800
>> time ./a.out 100000
>>
>> real    0m0.100s
>> user    0m0.080s
>> sys     0m0.020s
>>
>> So, what are you using as a development box?
>
> I should have mentioned that I am typically running inside a virtualized
> environment as I routinely need to run three to four machines in parallel for
> testing.

ahh, virtualized boxes are especially bad for gettimeofday().
which virtualization are you using?

David Lang

> If anyone is up for hardware contributions, just let me know ;)
>
> Rainer
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Mon Jun 15 20:31:42 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 15 Jun 2009 20:31:42 +0200
Subject: [rsyslog] high-performance, low-precision time API under linux?
References: <9B6E2A8877C38245BFB15CC491A11DA706F936@GRFEXC.intern.adiscon.com><e7ca40f70906150755r4c9b6214g1e9c596efa1ca5ad@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F939@GRFEXC.intern.adiscon.com><e7ca40f70906150832s6a38bc3ayd48037acd51acba3@mail.gmail.com><alpine.DEB.1.10.0906151007200.9159@asgard><e7ca40f70906151019j88c0e8bid35d347675a2bf14@mail.gmail.com><3DBDD4313FFFEA4F86E79AEF498136C3079B636D@WEBEXCHANGE.webde.local><9B6E2A8877C38245BFB15CC491A11DA706F93E@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906151126180.9159@asgard>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F93F@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Monday, June 15, 2009 8:27 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] high-performance, low-precision time API under
linux?
> 
> On Mon, 15 Jun 2009, Rainer Gerhards wrote:
> 
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Thomas Mieslinger
> >> Sent: Monday, June 15, 2009 7:58 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] high-performance, low-precision time API under
> > linux?
> >>
> >> BTW: In the Informix DBMS the Software Architects decided to implement
an
> > OS
> >> independent timer thread to work arround (very costly in the late 90s)
> >> gettimeofday() calls.
> >>
> >> I've run the code on some Solaris 10 boxes, because they were known for
> > their
> >> slow gettimeofday() implementation
> >>
> >> S11 snv_105 Pentium 4 3GHz
> >> time ./a.out 100000
> >> real    0m0.087s
> >> user    0m0.082s
> >> sys     0m0.004s
> >>
> >> S10 Generic_138889-03 Athlon 2,2GHz
> >> time ./a.out 100000
> >> real    0m0.028s
> >> user    0m0.026s
> >> sys     0m0.002s
> >>
> >> ok, lets try this on an older boxes...
> >> suse70 2.2.21 libc-2.1.3-141 2xP3 800
> >> time ./a.out 100000
> >>
> >> real    0m0.100s
> >> user    0m0.080s
> >> sys     0m0.020s
> >>
> >> So, what are you using as a development box?
> >
> > I should have mentioned that I am typically running inside a virtualized
> > environment as I routinely need to run three to four machines in parallel
> for
> > testing.
> 
> ahh, virtualized boxes are especially bad for gettimeofday().
> which virtualization are you using?

This one is using vmware, but that doesn't really matter. Even on a real box,
as you said, the toll to pay for gettimeofday() is high compared to the rest
of the workload. So I actually do not care how much the absolute time is but
rather how it goes into the overall mix. The context switch itself is
expensive enough, so this makes everything that goes to kernel space rather
expensive.

As I had written earlier, I will probably give up the approach of a
generalized timeout handler in favor of many targeted implementations. It
would have been nice to have this callout-tablish facility, but it looks like
it is too expensive to implement.

Rainer

> 
> David Lang
> 
> > If anyone is up for hardware contributions, just let me know ;)
> >
> > Rainer
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Wed Jun 17 09:25:20 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 17 Jun 2009 09:25:20 +0200
Subject: [rsyslog] General availability of atomic operations?
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F96E@GRFEXC.intern.adiscon.com>

Hi all,

I am currently facing a situation where I can re-structure message object
represenation in a very efficient way if I have atomic operations available.
However, if they are not available, the very same implementation would have a
heavy hit on concurrency and overhead.

What I need is atomic increments and decrements (for reference counting). So
I wonder if support for these operations is available on a sufficiently large
number of platforms. I would appreciate feedback on this issue, including
pointers to place where else I could put this question.

Thanks,
Rainer	


From thomas.mieslinger at 1und1.de  Wed Jun 17 10:15:18 2009
From: thomas.mieslinger at 1und1.de (Thomas Mieslinger)
Date: Wed, 17 Jun 2009 10:15:18 +0200
Subject: [rsyslog] General availability of atomic operations?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F96E@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F96E@GRFEXC.intern.adiscon.com>
Message-ID: <4A38A616.1010408@1und1.de>

Rainer Gerhards wrote:
> What I need is atomic increments and decrements (for reference counting). So
> I wonder if support for these operations is available on a sufficiently large
> number of platforms. I would appreciate feedback on this issue, including
> pointers to place where else I could put this question.

The Apache Portable Runtime includes atomic read/set/add/inc/dec 
implementations for many platforms. But for example the sparc 
implementation has been riped out because of unclear copyrights.

-- 
Thomas Mieslinger


From martinmie at PartyGaming.com  Thu Jun 18 14:18:04 2009
From: martinmie at PartyGaming.com (Martin Mielke)
Date: Thu, 18 Jun 2009 14:18:04 +0200
Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
Message-ID: <0B1B9163138571439EAF804646F3F6AA0767973E@GIBSVWIN004X.partygaming.local>

Hi all,

I need to find a way to either convert or import syslog-ng.conf files
into rsyslog or learn the syntax to classify/store the logfiles
depending on the source.

For example, taken from a syslog-ng file:
--
source s_network { udp(); tcp(); };

filter f_systems {
host("1\.2\.3\.\d+");
};

filter f_switches {
host("5\.6\.7\.8|5\.6\.7\.9");
};

destination d_systems {file("/systems.log"); };
destination d_switches {file("/switches.log"); };

log { source(s_network); filter(f_systems); destination(d_systems); };
log { source(s_network); filter(f_switches); destination(d_switches); };
--

I have read the online documentation but I'm unable to find the right
answer...

Thanks!

Martin


This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. 

Any views or opinions are solely those of the sender.  This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk.




From rgerhards at hq.adiscon.com  Thu Jun 18 15:11:27 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 18 Jun 2009 15:11:27 +0200
Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
References: <0B1B9163138571439EAF804646F3F6AA0767973E@GIBSVWIN004X.partygaming.local>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9A4@GRFEXC.intern.adiscon.com>

FYI, there was also a forum post:

http://kb.monitorware.com/syslog-filters-and-destinations-under-rsyslog-t9858
.html

I'd appreciate if any other responses also go to the forum (so that others,
when searching, find the complete info).

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Martin Mielke
> Sent: Thursday, June 18, 2009 2:18 PM
> To: rsyslog-users
> Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
> 
> Hi all,
> 
> I need to find a way to either convert or import syslog-ng.conf files
> into rsyslog or learn the syntax to classify/store the logfiles
> depending on the source.
> 
> For example, taken from a syslog-ng file:
> --
> source s_network { udp(); tcp(); };
> 
> filter f_systems {
> host("1\.2\.3\.\d+");
> };
> 
> filter f_switches {
> host("5\.6\.7\.8|5\.6\.7\.9");
> };
> 
> destination d_systems {file("/systems.log"); };
> destination d_switches {file("/switches.log"); };
> 
> log { source(s_network); filter(f_systems); destination(d_systems); };
> log { source(s_network); filter(f_switches); destination(d_switches); };
> --
> 
> I have read the online documentation but I'm unable to find the right
> answer...
> 
> Thanks!
> 
> Martin
> 
> 
> This email and any attachments are confidential, and may be legally
privileged
> and protected by copyright. If you are not the intended recipient
> dissemination or copying of this email is prohibited. If you have received
> this in error, please notify the sender by replying by email and then
delete
> the email completely from your system.
> 
> Any views or opinions are solely those of the sender.  This communication
is
> not intended to form a binding contract unless expressly indicated to the
> contrary and properly authorised. Any actions taken on the basis of this
email
> are at the recipient's own risk.
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From alex at samad.com.au  Fri Jun 19 10:05:01 2009
From: alex at samad.com.au (Alex Samad)
Date: Fri, 19 Jun 2009 18:05:01 +1000
Subject: [rsyslog] Help with configurin rsyslog
Message-ID: <20090619080501.GB11392@samad.com.au>

Hi

I have a syslog line that looks like this

Jun 18 10:35:53 Thu Jun 18 10:42:37 2009 dlink-dir-855 System Log:
Stored configuration to non-volatile memory

notice the double dates stamps and it seems like I should be able to
rectify this with rsyslog, but I am not sure how.

I have had a quick look at the doco and the wiki, but there aren't any
example's of replacing/reformatting from specific hosts

note the above line is from a dlink dir-855 router/access point

I also have these
Jun 18 10:55:00 127.0.0.1 time="2009-06-18 10:55:00" msg="LOGIN User
logged in on TELNET (192.168.1.100)"

from a speedtouch 546v6 then I need to munge.

Any pointers or working examples would be helpful

Thanks


From rgerhards at hq.adiscon.com  Fri Jun 19 18:14:20 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Jun 2009 18:14:20 +0200
Subject: [rsyslog] memory alloc problems - any advise?
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>

Hi all,

during testing, I found out that some strange things happen when a queue is
set to large size. If I set e.g. the main message queue to 500,000 entries
and then shuffle messages very quickly at rsyslogd, it consumes considerable
memory. This is not surprising, as it is buffering not yet-processed
messages.

What is surprising is that the memory is not given back to the OS when the
data has been processed. Well... I know that some data blocks may still be
held and thus prevent giving memory back. But that should be solved when a
few new messages come in. Still, nothing changes in that scenario. What
puzzles me even more is that when I start a new burst of messages (I send
around one million each), it looks like almost only new memory is obtained
from the OS for them. At least here I had expected that the allocator would
re-use what was already claimed.

The interesting things is that when the queue size is small (50,000 message
for example), this behavior does not occur. Then, memory allocation and
deallocation works as expected.

I have also run the valgrind memory debugger on the app, and it does not
report any leaked memory.

If someone has an explanation of what causes this behavior, I would really
love to hear about it...

Thanks,
Rainer


From aoz.syn at gmail.com  Fri Jun 19 18:26:46 2009
From: aoz.syn at gmail.com (RB)
Date: Fri, 19 Jun 2009 10:26:46 -0600
Subject: [rsyslog] memory alloc problems - any advise?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
Message-ID: <4255c2570906190926v2399fcc7y7feddb010009faf5@mail.gmail.com>

On Fri, Jun 19, 2009 at 10:14, Rainer Gerhards<rgerhards at hq.adiscon.com> wrote:
> What is surprising is that the memory is not given back to the OS when the
> data has been processed. Well... I know that some data blocks may still be
> held and thus prevent giving memory back. But that should be solved when a
> few new messages come in. Still, nothing changes in that scenario. What
> puzzles me even more is that when I start a new burst of messages (I send
> around one million each), it looks like almost only new memory is obtained
> from the OS for them. At least here I had expected that the allocator would
> re-use what was already claimed.

Can you clarify what you mean by "not given back to the OS"?  Do you
mean that the RSS of rsyslog remains the same, or that the system
'free' memory does not go back up, or something else entirely?  I'm
wondering if you're seeing caching effects.


From rgerhards at hq.adiscon.com  Fri Jun 19 18:36:05 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Jun 2009 18:36:05 +0200
Subject: [rsyslog] memory alloc problems - any advise?
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
	<4255c2570906190926v2399fcc7y7feddb010009faf5@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9E0@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of RB
> Sent: Friday, June 19, 2009 6:27 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> 
> On Fri, Jun 19, 2009 at 10:14, Rainer Gerhards<rgerhards at hq.adiscon.com>
> wrote:
> > What is surprising is that the memory is not given back to the OS when
the
> > data has been processed. Well... I know that some data blocks may still
be
> > held and thus prevent giving memory back. But that should be solved when
a
> > few new messages come in. Still, nothing changes in that scenario. What
> > puzzles me even more is that when I start a new burst of messages (I send
> > around one million each), it looks like almost only new memory is
obtained
> > from the OS for them. At least here I had expected that the allocator
would
> > re-use what was already claimed.
> 
> Can you clarify what you mean by "not given back to the OS"?  Do you
> mean that the RSS of rsyslog remains the same, or that the system
> 'free' memory does not go back up, or something else entirely?  I'm
> wondering if you're seeing caching effects.

Rss remains the same, looks like sbrk() is not called. I poked somemore
around, and it looks like I need to do a malloc_trim() call in glibc to solve
this issue. Will post some stats soon...

Rainer 


From rgerhards at hq.adiscon.com  Fri Jun 19 18:57:06 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Jun 2009 18:57:06 +0200
Subject: [rsyslog] memory alloc problems - any advise?
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com><4255c2570906190926v2399fcc7y7feddb010009faf5@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E0@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9E1@GRFEXC.intern.adiscon.com>

The promised stats. I used GNU's malloc_stats() and malloc_trim().

I queried stats every 5 seconds (lines with # are my comments):

#Right after startup:
Arena 0:
system bytes     =     245760
in use bytes     =     244624
Total (incl. mmap):
system bytes     =    4247552
in use bytes     =    4246416
max mmap regions =          1
max mmap bytes   =    4001792

#First burst:
Arena 0:
system bytes     =  108318720
in use bytes     =  107797520
Arena 1:
system bytes     =   67661824
in use bytes     =   18954784
Total (incl. mmap):
system bytes     =  179982336
in use bytes     =  130754096
max mmap regions =          1
max mmap bytes   =    4001792

#processing...
Arena 0:
system bytes     =  143970304
in use bytes     =   93414224
Arena 1:
system bytes     =   91770880
in use bytes     =       2336
Arena 2:
system bytes     =  106659840
in use bytes     =       2336
Total (incl. mmap):
system bytes     =  346402816
in use bytes     =   97420688
max mmap regions =          1
max mmap bytes   =    4001792

#idle
Arena 0:
system bytes     =  143970304
in use bytes     =     255248
Arena 1:
system bytes     =   91770880
in use bytes     =       2288
Arena 2:
system bytes     =  106659840
in use bytes     =       2288
Total (incl. mmap):
system bytes     =  346402816
in use bytes     =    4261616
max mmap regions =          1
max mmap bytes   =    4001792

#second burst - so it looks like the memory is reused (good!)
Arena 0:
system bytes     =  143970304
in use bytes     =  102348464
Arena 1:
system bytes     =   91770880
in use bytes     =   14172352
Arena 2:
system bytes     =  106659840
in use bytes     =       3840
Total (incl. mmap):
system bytes     =  346402816
in use bytes     =  120526448
max mmap regions =          1
max mmap bytes   =    4001792

# first and second burst were equal, same messages, same amount of them.
# Still not all allocs are re-used (they must have been identical, too). 
# As it looks > 50% were not reused...
Arena 0:
system bytes     =  206884864
in use bytes     =  206110592
Arena 1:
system bytes     =   91770880
in use bytes     =       2336
Arena 2:
system bytes     =  162009088
in use bytes     =       2400
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =  210117120
max mmap regions =          1
max mmap bytes   =    4001792

#idle again
Arena 0:
system bytes     =  206884864
in use bytes     =     256000
Arena 1:
system bytes     =   91770880
in use bytes     =       3456
Arena 2:
system bytes     =  162009088
in use bytes     =       2352
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =    4263600
max mmap regions =          1
max mmap bytes   =    4001792

# still idle 
Arena 0:
system bytes     =  206884864
in use bytes     =     256000
Arena 1:
system bytes     =   91770880
in use bytes     =       3456
Arena 2:
system bytes     =  162009088
in use bytes     =       2352
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =    4263600
max mmap regions =          1
max mmap bytes   =    4001792

# still idle
Arena 0:
system bytes     =  206884864
in use bytes     =     256000
Arena 1:
system bytes     =   91770880
in use bytes     =       3456
Arena 2:
system bytes     =  162009088
in use bytes     =       2352
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =    4263600
max mmap regions =          1
max mmap bytes   =    4001792

# I did a malloc_trim() every 10th stat counter and here it happens.
# The number still look the same, but in the OS system monitor, I see freed
# memory (and it actually is freed if I look at how the system then
performs).
# So it looks like only the virtual address space was kept?
malloc trim (1)!
Arena 0:
system bytes     =  206884864
in use bytes     =     256000
Arena 1:
system bytes     =   91770880
in use bytes     =       3456
Arena 2:
system bytes     =  162009088
in use bytes     =       2352
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =    4263600
max mmap regions =          1
max mmap bytes   =    4001792

# no activity, so it remains at this level...
Arena 0:
system bytes     =  206884864
in use bytes     =     256000
Arena 1:
system bytes     =   91770880
in use bytes     =       3456
Arena 2:
system bytes     =  162009088
in use bytes     =       2352
Total (incl. mmap):
system bytes     =  464666624
in use bytes     =    4263600
max mmap regions =          1
max mmap bytes   =    4001792

With that information, I'll probably better turn to a GNU mailing list. Still
I wonder if someone here has an opinion. If so, please let me know. For the
time being, it looks like I need to do some period malloc_trim() calls...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Friday, June 19, 2009 6:36 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of RB
> > Sent: Friday, June 19, 2009 6:27 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] memory alloc problems - any advise?
> >
> > On Fri, Jun 19, 2009 at 10:14, Rainer Gerhards<rgerhards at hq.adiscon.com>
> > wrote:
> > > What is surprising is that the memory is not given back to the OS when
> the
> > > data has been processed. Well... I know that some data blocks may still
> be
> > > held and thus prevent giving memory back. But that should be solved
when
> a
> > > few new messages come in. Still, nothing changes in that scenario. What
> > > puzzles me even more is that when I start a new burst of messages (I
send
> > > around one million each), it looks like almost only new memory is
> obtained
> > > from the OS for them. At least here I had expected that the allocator
> would
> > > re-use what was already claimed.
> >
> > Can you clarify what you mean by "not given back to the OS"?  Do you
> > mean that the RSS of rsyslog remains the same, or that the system
> > 'free' memory does not go back up, or something else entirely?  I'm
> > wondering if you're seeing caching effects.
> 
> Rss remains the same, looks like sbrk() is not called. I poked somemore
> around, and it looks like I need to do a malloc_trim() call in glibc to
solve
> this issue. Will post some stats soon...
> 
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From david at lang.hm  Fri Jun 19 22:14:16 2009
From: david at lang.hm (david at lang.hm)
Date: Fri, 19 Jun 2009 13:14:16 -0700 (PDT)
Subject: [rsyslog] Help with configurin rsyslog
In-Reply-To: <20090619080501.GB11392@samad.com.au>
References: <20090619080501.GB11392@samad.com.au>
Message-ID: <alpine.DEB.1.10.0906191305150.20154@asgard.lang.hm>

On Fri, 19 Jun 2009, Alex Samad wrote:

> I have a syslog line that looks like this
>
> Jun 18 10:35:53 Thu Jun 18 10:42:37 2009 dlink-dir-855 System Log:
> Stored configuration to non-volatile memory
>
> notice the double dates stamps and it seems like I should be able to
> rectify this with rsyslog, but I am not sure how.
>
> I have had a quick look at the doco and the wiki, but there aren't any
> example's of replacing/reformatting from specific hosts
>
> note the above line is from a dlink dir-855 router/access point
>
> I also have these
> Jun 18 10:55:00 127.0.0.1 time="2009-06-18 10:55:00" msg="LOGIN User
> logged in on TELNET (192.168.1.100)"
>
> from a speedtouch 546v6 then I need to munge.
>
> Any pointers or working examples would be helpful

I think that the key approach to do this is to create additional formats 
and then do conditional logging

something like the following pseudocode

if hostip = '1.1.1.1' then /var/log/messages;CustomFormat1
if hostip != '1.1.1.1' then /var/log/messages;StandardFormat

the first issue is to figure out how to filter the different messages 
seperate from each other

depending on how badly the message is misformed on the wire this may be 
hard to do. if you do not go through any relay servers you can filter 
based on the source IP of the connection to rsyslog

after you get the different logs filtered from each other you can figure 
out how to reformat each type of message.

hopefully the 'odd' messages only comes from a small number of sources so 
that you can do conditionals for the specific sources reasonably. (rsyslog 
doesn't currently have a mechanism to effectivly do table lookups)

I hope this helps.

David Lang


From david at lang.hm  Fri Jun 19 22:29:30 2009
From: david at lang.hm (david at lang.hm)
Date: Fri, 19 Jun 2009 13:29:30 -0700 (PDT)
Subject: [rsyslog] memory alloc problems - any advise?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906191324100.20154@asgard.lang.hm>

On Fri, 19 Jun 2009, Rainer Gerhards wrote:

> Hi all,
>
> during testing, I found out that some strange things happen when a queue is
> set to large size. If I set e.g. the main message queue to 500,000 entries
> and then shuffle messages very quickly at rsyslogd, it consumes considerable
> memory. This is not surprising, as it is buffering not yet-processed
> messages.
>
> What is surprising is that the memory is not given back to the OS when the
> data has been processed. Well... I know that some data blocks may still be
> held and thus prevent giving memory back. But that should be solved when a
> few new messages come in. Still, nothing changes in that scenario. What
> puzzles me even more is that when I start a new burst of messages (I send
> around one million each), it looks like almost only new memory is obtained
> from the OS for them. At least here I had expected that the allocator would
> re-use what was already claimed.
>
> The interesting things is that when the queue size is small (50,000 message
> for example), this behavior does not occur. Then, memory allocation and
> deallocation works as expected.

hmm, this doesn't match what I was seeing with the 3.x series. I had a 
machine with 32G of ram and was setting the max queue size to 1M entries.

I don't remember if it was freeing the memory as the queue emptied, but I 
know that I could hit it repeatedly with large peaks and it would not use 
additional memory.

> I have also run the valgrind memory debugger on the app, and it does not
> report any leaked memory.

remember that it will only consider it a leak if you no longer have _any_ 
refrence to the memory. if you have some pointer to it stashed away 
somewhere that you won't use it, it won't consider it a leak.

early next week I should have some new boxes in that I can do high-speed 
tests on again (64G ram with 10G network)

David Lang

> If someone has an explanation of what causes this behavior, I would really
> love to hear about it...
>
> Thanks,
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Fri Jun 19 22:34:48 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 19 Jun 2009 22:34:48 +0200
Subject: [rsyslog] memory alloc problems - any advise?
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906191324100.20154@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9E3@GRFEXC.intern.adiscon.com>


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, June 19, 2009 10:30 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> 
> On Fri, 19 Jun 2009, Rainer Gerhards wrote:
> 
> > Hi all,
> >
> > during testing, I found out that some strange things happen 
> when a queue is
> > set to large size. If I set e.g. the main message queue to 
> 500,000 entries
> > and then shuffle messages very quickly at rsyslogd, it 
> consumes considerable
> > memory. This is not surprising, as it is buffering not yet-processed
> > messages.
> >
> > What is surprising is that the memory is not given back to 
> the OS when the
> > data has been processed. Well... I know that some data 
> blocks may still be
> > held and thus prevent giving memory back. But that should 
> be solved when a
> > few new messages come in. Still, nothing changes in that 
> scenario. What
> > puzzles me even more is that when I start a new burst of 
> messages (I send
> > around one million each), it looks like almost only new 
> memory is obtained
> > from the OS for them. At least here I had expected that the 
> allocator would
> > re-use what was already claimed.
> >
> > The interesting things is that when the queue size is small 
> (50,000 message
> > for example), this behavior does not occur. Then, memory 
> allocation and
> > deallocation works as expected.
> 
> hmm, this doesn't match what I was seeing with the 3.x 
> series. I had a 
> machine with 32G of ram and was setting the max queue size to 
> 1M entries.

I checked with v3, and it had the same behavior (a little less strong, but
still).

> 
> I don't remember if it was freeing the memory as the queue 
> emptied, but I 
> know that I could hit it repeatedly with large peaks and it 
> would not use 
> additional memory.
> 
> > I have also run the valgrind memory debugger on the app, 
> and it does not
> > report any leaked memory.
> 
> remember that it will only consider it a leak if you no 
> longer have _any_ 
> refrence to the memory. if you have some pointer to it stashed away 
> somewhere that you won't use it, it won't consider it a leak.

Valgrind can print out those allocations. In short, there were nothing
unexpected (the usual 81 when rsyslogd terminates, and they pose no problem).
As you can see in my later posts, the issue seems actually to be rooted in
glibc.

> 
> early next week I should have some new boxes in that I can do 
> high-speed 
> tests on again (64G ram with 10G network)

That's good news. I did a lot of optimizations this week, and think I can
release a new version with them mid next week. I hope you will see even
further improvement. In the v5 engine, I think, I have discovered a memory
leak, but I didn't investigate today. Maybe just the same issue I've later
found wiht the malloc calls...

Rainer

> 
> David Lang
> 
> > If someone has an explanation of what causes this behavior, 
> I would really
> > love to hear about it...
> >
> > Thanks,
> > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 


From david at lang.hm  Fri Jun 19 23:36:31 2009
From: david at lang.hm (david at lang.hm)
Date: Fri, 19 Jun 2009 14:36:31 -0700 (PDT)
Subject: [rsyslog] memory alloc problems - any advise?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9E3@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906191324100.20154@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E3@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906191352480.20154@asgard.lang.hm>

On Fri, 19 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Fri, 19 Jun 2009, Rainer Gerhards wrote:
>>
>>> Hi all,
>>>
>>> during testing, I found out that some strange things happen
>> when a queue is
>>> set to large size. If I set e.g. the main message queue to
>> 500,000 entries
>>> and then shuffle messages very quickly at rsyslogd, it
>> consumes considerable
>>> memory. This is not surprising, as it is buffering not yet-processed
>>> messages.
>>>
>>> What is surprising is that the memory is not given back to
>> the OS when the
>>> data has been processed. Well... I know that some data
>> blocks may still be
>>> held and thus prevent giving memory back. But that should
>> be solved when a
>>> few new messages come in. Still, nothing changes in that
>> scenario. What
>>> puzzles me even more is that when I start a new burst of
>> messages (I send
>>> around one million each), it looks like almost only new
>> memory is obtained
>>> from the OS for them. At least here I had expected that the
>> allocator would
>>> re-use what was already claimed.
>>>
>>> The interesting things is that when the queue size is small
>> (50,000 message
>>> for example), this behavior does not occur. Then, memory
>> allocation and
>>> deallocation works as expected.
>>
>> hmm, this doesn't match what I was seeing with the 3.x
>> series. I had a
>> machine with 32G of ram and was setting the max queue size to
>> 1M entries.
>
> I checked with v3, and it had the same behavior (a little less strong, but
> still).

interesting.

when I was doing the tests that showed rsyslog able to recieve at gig-e 
line speed I would run tests repeatedly without any problems (with each 
test sending millions of logs, things sitting idle for hours, then another 
burst of millions of logs, repeat)

if I set the queue size too large for my ram the system would start paging 
and die, so I'm sure that it wasn't failing to release memory.

now, one thing could be that my tests were of messages that were all the 
same size, it could be that you are seeing some effect from fragmentation 
with different message size.

David Lang

>>
>> I don't remember if it was freeing the memory as the queue
>> emptied, but I
>> know that I could hit it repeatedly with large peaks and it
>> would not use
>> additional memory.
>>
>>> I have also run the valgrind memory debugger on the app,
>> and it does not
>>> report any leaked memory.
>>
>> remember that it will only consider it a leak if you no
>> longer have _any_
>> refrence to the memory. if you have some pointer to it stashed away
>> somewhere that you won't use it, it won't consider it a leak.
>
> Valgrind can print out those allocations. In short, there were nothing
> unexpected (the usual 81 when rsyslogd terminates, and they pose no problem).
> As you can see in my later posts, the issue seems actually to be rooted in
> glibc.
>
>>
>> early next week I should have some new boxes in that I can do
>> high-speed
>> tests on again (64G ram with 10G network)
>
> That's good news. I did a lot of optimizations this week, and think I can
> release a new version with them mid next week. I hope you will see even
> further improvement. In the v5 engine, I think, I have discovered a memory
> leak, but I didn't investigate today. Maybe just the same issue I've later
> found wiht the malloc calls...
>
> Rainer
>
>>
>> David Lang
>>
>>> If someone has an explanation of what causes this behavior,
>> I would really
>>> love to hear about it...
>>>
>>> Thanks,
>>> Rainer
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From jiri.hlinka at gmail.com  Sat Jun 20 21:47:45 2009
From: jiri.hlinka at gmail.com (=?ISO-8859-2?Q?Ji=F8=ED_Hlinka?=)
Date: Sat, 20 Jun 2009 21:47:45 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
Message-ID: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com>

Hello,
is there any info about what rsyslog version will be available in RHEL
6? I suppose it will be 3.x, but im not sure, maybe 4.x version in the
time "RHEL6 package freeze" will be in stable status?

Thank you,
Jiri H.


From rgerhards at hq.adiscon.com  Sun Jun 21 11:58:57 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 21 Jun 2009 11:58:57 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>

That a good question and I, too, would like to know the answer. In any case,
I hope it will not be v2. Giving that RHEL is a conservative distribution
(and it has to be for the target base), I would assume that if they go for a
newer version, it is v3.

Rainer 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
> Sent: Saturday, June 20, 2009 9:48 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] RSyslog version in RHEL 6
> 
> Hello,
> is there any info about what rsyslog version will be available in RHEL
> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in the
> time "RHEL6 package freeze" will be in stable status?
> 
> Thank you,
> Jiri H.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 


From david at lang.hm  Sun Jun 21 12:15:12 2009
From: david at lang.hm (david at lang.hm)
Date: Sun, 21 Jun 2009 03:15:12 -0700 (PDT)
Subject: [rsyslog] RSyslog version in RHEL 6
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906210309480.23743@asgard.lang.hm>

On Sun, 21 Jun 2009, Rainer Gerhards wrote:

> That a good question and I, too, would like to know the answer. In any case,
> I hope it will not be v2. Giving that RHEL is a conservative distribution
> (and it has to be for the target base), I would assume that if they go for a
> newer version, it is v3.

not knowing when the freeze date is it's hard to say.

remember that the huge performance benifits of v4 (and v5 when it comes 
out) would also be very useful for the target base.

not to mention all the subtle fixes you have done whiel pursuing 
performance.

David Lang

> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
>> Sent: Saturday, June 20, 2009 9:48 PM
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] RSyslog version in RHEL 6
>>
>> Hello,
>> is there any info about what rsyslog version will be available in RHEL
>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in the
>> time "RHEL6 package freeze" will be in stable status?
>>
>> Thank you,
>> Jiri H.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Sun Jun 21 14:54:49 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 21 Jun 2009 14:54:49 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906210309480.23743@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9E6@GRFEXC.intern.adiscon.com>


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Sunday, June 21, 2009 12:15 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] RSyslog version in RHEL 6
> 
> On Sun, 21 Jun 2009, Rainer Gerhards wrote:
> 
> > That a good question and I, too, would like to know the 
> answer. In any case,
> > I hope it will not be v2. Giving that RHEL is a 
> conservative distribution
> > (and it has to be for the target base), I would assume that 
> if they go for a
> > newer version, it is v3.
> 
> not knowing when the freeze date is it's hard to say.
> 
> remember that the huge performance benifits of v4 (and v5 
> when it comes 
> out) would also be very useful for the target base.
> 
> not to mention all the subtle fixes you have done whiel pursuing 
> performance.

Yeah, that's right. But anyway ... I am not even asked, so I am a bystander
as most of the rest of this list ;)

> 
> David Lang
> 
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
> >> Sent: Saturday, June 20, 2009 9:48 PM
> >> To: rsyslog at lists.adiscon.com
> >> Subject: [rsyslog] RSyslog version in RHEL 6
> >>
> >> Hello,
> >> is there any info about what rsyslog version will be 
> available in RHEL
> >> 6? I suppose it will be 3.x, but im not sure, maybe 4.x 
> version in the
> >> time "RHEL6 package freeze" will be in stable status?
> >>
> >> Thank you,
> >> Jiri H.
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 


From david at lang.hm  Sun Jun 21 23:29:11 2009
From: david at lang.hm (david at lang.hm)
Date: Sun, 21 Jun 2009 14:29:11 -0700 (PDT)
Subject: [rsyslog] RSyslog version in RHEL 6
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9E6@GRFEXC.intern.adiscon.com>
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906210309480.23743@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E6@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906211427280.23743@asgard.lang.hm>

On Sun, 21 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Sun, 21 Jun 2009, Rainer Gerhards wrote:
>>
>>> That a good question and I, too, would like to know the
>> answer. In any case,
>>> I hope it will not be v2. Giving that RHEL is a
>> conservative distribution
>>> (and it has to be for the target base), I would assume that
>> if they go for a
>>> newer version, it is v3.
>>
>> not knowing when the freeze date is it's hard to say.
>>
>> remember that the huge performance benifits of v4 (and v5
>> when it comes
>> out) would also be very useful for the target base.
>>
>> not to mention all the subtle fixes you have done whiel pursuing
>> performance.
>
> Yeah, that's right. But anyway ... I am not even asked, so I am a bystander
> as most of the rest of this list ;)

Iagree that you are not asked, but since you hae a lot of influance on the 
schedule of when 4.x moves to stable you are a little more than a 
bystander ;-)

does anyone know the RHEL schedule? I would hate to see them lock in 3.x a 
few days before 4.x becomes stable

David Lang

>>
>> David Lang
>>
>>> Rainer
>>>
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com
>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
>>>> Sent: Saturday, June 20, 2009 9:48 PM
>>>> To: rsyslog at lists.adiscon.com
>>>> Subject: [rsyslog] RSyslog version in RHEL 6
>>>>
>>>> Hello,
>>>> is there any info about what rsyslog version will be
>> available in RHEL
>>>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x
>> version in the
>>>> time "RHEL6 package freeze" will be in stable status?
>>>>
>>>> Thank you,
>>>> Jiri H.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rio at rio.st  Mon Jun 22 03:44:56 2009
From: rio at rio.st (=?ISO-2022-JP?B?GyRCTkcbKEIgGyRCRiNFRBsoQg==?=)
Date: Mon, 22 Jun 2009 10:44:56 +0900
Subject: [rsyslog] RSyslog version in RHEL 6
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
Message-ID: <44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st>

Hi all,

I guess RHEL6 will be based on Fedora 11 & 12.
Fedora 11 was shipped with rsyslog-3.21.11-1.fc11.
It's sure that RHEL6 includes v3 at least.

Feature Freeze of Fedora 12 is planned on 2009-07-28.
http://fedoraproject.org/wiki/Schedule
If v4 is stable and requested to be included in F12, v4 will be  
shipped in F12.

Best Rio.

On 2009/06/21, at 18:58, Rainer Gerhards wrote:

> That a good question and I, too, would like to know the answer. In  
> any case,
> I hope it will not be v2. Giving that RHEL is a conservative  
> distribution
> (and it has to be for the target base), I would assume that if they  
> go for a
> newer version, it is v3.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
>> Sent: Saturday, June 20, 2009 9:48 PM
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] RSyslog version in RHEL 6
>>
>> Hello,
>> is there any info about what rsyslog version will be available in  
>> RHEL
>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in  
>> the
>> time "RHEL6 package freeze" will be in stable status?
>>
>> Thank you,
>> Jiri H.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com



From rgerhards at hq.adiscon.com  Mon Jun 22 11:16:02 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 22 Jun 2009 11:16:02 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906210309480.23743@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA706F9E6@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906211427280.23743@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9EB@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Sunday, June 21, 2009 11:29 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] RSyslog version in RHEL 6
> 
> On Sun, 21 Jun 2009, Rainer Gerhards wrote:

> >
> > Yeah, that's right. But anyway ... I am not even asked, so I am a
bystander
> > as most of the rest of this list ;)
> 
> Iagree that you are not asked, but since you hae a lot of influance on the
> schedule of when 4.x moves to stable you are a little more than a
> bystander ;-)


Well, the good thing is that the current v4-beta was actually scheduled to be
declared stable "these days". Will see that I make this "tomorrow". However,
the version that has most of the optimizations needs to go through the beta
stage (and needs so for good reasons), so this is another ~ three month from
now. But it probably would be sufficient to have v4 included, hopefully this
means newer versions of that same major version will also be included...

Rainer


From rgerhards at hq.adiscon.com  Mon Jun 22 13:10:32 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 22 Jun 2009 13:10:32 +0200
Subject: [rsyslog] memory alloc problems - any advise?
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906191324100.20154@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA706F9E3@GRFEXC.intern.adiscon.com>
	<alpine.DEB.1.10.0906191352480.20154@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9EE@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, June 19, 2009 11:37 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> 
> On Fri, 19 Jun 2009, Rainer Gerhards wrote:
> 
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >>
> >> On Fri, 19 Jun 2009, Rainer Gerhards wrote:
> >>
> > I checked with v3, and it had the same behavior (a little less strong,
but
> > still).
> 
> interesting.
> 
> when I was doing the tests that showed rsyslog able to recieve at gig-e
> line speed I would run tests repeatedly without any problems (with each
> test sending millions of logs, things sitting idle for hours, then another
> burst of millions of logs, repeat)
> 
> if I set the queue size too large for my ram the system would start paging
> and die, so I'm sure that it wasn't failing to release memory.
> 
> now, one thing could be that my tests were of messages that were all the
> same size, it could be that you are seeing some effect from fragmentation
> with different message size.

My test driver produces almost identical messages, with mostly the same size.
I have not yet implemented any randomization into it. So in theory, it should
be fairly easy for the malloc() subsystem to reuse freed memory.

Anyhow, based on the malloc stats I got, this really looks like an issue with
the allocator itself. I'll for now implement an alloc_trim() every 100,000
messages or so, that should keep care of this anomaly. Interestingly, v4 now
does fewer allocs than v3. It almost looks like the alloc subsystem also has
a periodic "garbage collection" counter, which now seems to be triggered less
often.

For v5, I have different memory alloc on my mind. My long-term goal is to
have almost no alloc/free pre message (in the magnitude of 1/1024 per
message). But this is rather intrusive and will not have any immediate hard
benefit. Thus I am not targeting this is something that needs to be done now.
For v4, however, I intend to do some more optimizations, some of which
further reduce the number of alloc/free calls needed per message.

I would definitely be a perfect match if you could test out these things on a
high-performance machine.

Rainer



From pvrabec at redhat.com  Mon Jun 22 15:53:10 2009
From: pvrabec at redhat.com (Peter Vrabec)
Date: Mon, 22 Jun 2009 15:53:10 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
In-Reply-To: <44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st>
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com>
	<44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st>
Message-ID: <200906221553.10791.pvrabec@redhat.com>

Hi folks,

rsyslog maintainer is on vacation at the moment so I'm answering this Q. :)

At first we need v4 stable. If we have stable v4 we will push it into rawhide 
and F11. This must be done ASAP, I see deadline in mid-July.

Peter.


On Monday 22 June 2009 03:44:56 am ? ?? wrote:
> Hi all,
>
> I guess RHEL6 will be based on Fedora 11 & 12.
> Fedora 11 was shipped with rsyslog-3.21.11-1.fc11.
> It's sure that RHEL6 includes v3 at least.
>
> Feature Freeze of Fedora 12 is planned on 2009-07-28.
> http://fedoraproject.org/wiki/Schedule
> If v4 is stable and requested to be included in F12, v4 will be
> shipped in F12.
>
> Best Rio.
>
> On 2009/06/21, at 18:58, Rainer Gerhards wrote:
> > That a good question and I, too, would like to know the answer. In
> > any case,
> > I hope it will not be v2. Giving that RHEL is a conservative
> > distribution
> > (and it has to be for the target base), I would assume that if they
> > go for a
> > newer version, it is v3.
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
> >> Sent: Saturday, June 20, 2009 9:48 PM
> >> To: rsyslog at lists.adiscon.com
> >> Subject: [rsyslog] RSyslog version in RHEL 6
> >>
> >> Hello,
> >> is there any info about what rsyslog version will be available in
> >> RHEL
> >> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in
> >> the
> >> time "RHEL6 package freeze" will be in stable status?
> >>
> >> Thank you,
> >> Jiri H.
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com



From rgerhards at hq.adiscon.com  Mon Jun 22 14:24:13 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 22 Jun 2009 14:24:13 +0200
Subject: [rsyslog] RSyslog version in RHEL 6
References: <f427baa80906201247t8fc5a9andd2179e108812dce@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com><44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st>
	<200906221553.10791.pvrabec@redhat.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com>

Hi Peter,

that sounds quite interesting. I'll see that I get the v4-stable out
tomorrow.

Thanks for following up...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Peter Vrabec
> Sent: Monday, June 22, 2009 3:53 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] RSyslog version in RHEL 6
> 
> Hi folks,
> 
> rsyslog maintainer is on vacation at the moment so I'm answering this Q. :)
> 
> At first we need v4 stable. If we have stable v4 we will push it into
rawhide
> and F11. This must be done ASAP, I see deadline in mid-July.
> 
> Peter.
> 
> 
> On Monday 22 June 2009 03:44:56 am ? ?? wrote:
> > Hi all,
> >
> > I guess RHEL6 will be based on Fedora 11 & 12.
> > Fedora 11 was shipped with rsyslog-3.21.11-1.fc11.
> > It's sure that RHEL6 includes v3 at least.
> >
> > Feature Freeze of Fedora 12 is planned on 2009-07-28.
> > http://fedoraproject.org/wiki/Schedule
> > If v4 is stable and requested to be included in F12, v4 will be
> > shipped in F12.
> >
> > Best Rio.
> >
> > On 2009/06/21, at 18:58, Rainer Gerhards wrote:
> > > That a good question and I, too, would like to know the answer. In
> > > any case,
> > > I hope it will not be v2. Giving that RHEL is a conservative
> > > distribution
> > > (and it has to be for the target base), I would assume that if they
> > > go for a
> > > newer version, it is v3.
> > >
> > > Rainer
> > >
> > >> -----Original Message-----
> > >> From: rsyslog-bounces at lists.adiscon.com
> > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka
> > >> Sent: Saturday, June 20, 2009 9:48 PM
> > >> To: rsyslog at lists.adiscon.com
> > >> Subject: [rsyslog] RSyslog version in RHEL 6
> > >>
> > >> Hello,
> > >> is there any info about what rsyslog version will be available in
> > >> RHEL
> > >> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in
> > >> the
> > >> time "RHEL6 package freeze" will be in stable status?
> > >>
> > >> Thank you,
> > >> Jiri H.
> > >> _______________________________________________
> > >> rsyslog mailing list
> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> http://www.rsyslog.com
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Mon Jun 22 19:04:35 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 22 Jun 2009 19:04:35 +0200
Subject: [rsyslog] memory alloc problems - any advise?
References: <9B6E2A8877C38245BFB15CC491A11DA706F9DF@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906191324100.20154@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA706F9E3@GRFEXC.intern.adiscon.com><alpine.DEB.1.10.0906191352480.20154@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706F9EE@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706F9F6@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Monday, June 22, 2009 1:11 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> My test driver produces almost identical messages, with mostly the same
size.
> I have not yet implemented any randomization into it. So in theory, it
should
> be fairly easy for the malloc() subsystem to reuse freed memory.
> 
> Anyhow, based on the malloc stats I got, this really looks like an issue
with
> the allocator itself. I'll for now implement an alloc_trim() every 100,000
> messages or so, that should keep care of this anomaly. Interestingly, v4
now
> does fewer allocs than v3. It almost looks like the alloc subsystem also
has
> a periodic "garbage collection" counter, which now seems to be triggered
less
> often.

Just FYI, I have implemented this algorithm now and it works pretty well.
Also, if there is an impact on performance, it is barely noticeable. 

I have also fixed a memory leak in v5 today, and run the same tests with v5
that I ran against v4. It is really impressive how much better it performs
(factor two, at least). On my low-end lab virtual machine, I am currently
processing around 150,00 tcp msgs/s (around 70,000 with v4) and this is also
close to sustained rate (that probably is around 10% below for v5).

Just thought I let you know.

Rainer


From tbergfeld at hq.adiscon.com  Tue Jun 23 11:25:52 2009
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Tue, 23 Jun 2009 11:25:52 +0200
Subject: [rsyslog] rsyslog 4.2.0 (v4-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA01@GRFEXC.intern.adiscon.com>

Hi all,
We have released the first version of rsyslog v4-stable today. It is the next
major step after v3 most importantly offering large performance improvements.
This version went through a long beta-phase and we did not have any specifc
bug reports within the last couple of weeks, so we honestly think this
release is ready for production use.

Download:

http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-161.phtml

Changelog:

http://www.rsyslog.com/Article376.phtml

As always, feedback is appreciated.

Best regards,
Tom Bergfeld

--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From sachulinux at gmail.com  Tue Jun 23 13:59:20 2009
From: sachulinux at gmail.com (Sachin Sawant)
Date: Tue, 23 Jun 2009 17:29:20 +0530
Subject: [rsyslog] template for file input module in Rsyslog
Message-ID: <edc9be030906230459q4090383sc8427501d8a413a7@mail.gmail.com>

Hi All,



I would like to monitor PostgreSQL log files using file input module in
Rsyslog. PostgreSQL create log files based on date, so is it possible to
monitor file without using absolute path?



Regards,

* *

*Sachin Sawant

*


From rgerhards at hq.adiscon.com  Tue Jun 23 14:47:34 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 23 Jun 2009 14:47:34 +0200
Subject: [rsyslog] template for file input module in Rsyslog
References: <edc9be030906230459q4090383sc8427501d8a413a7@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA0D@GRFEXC.intern.adiscon.com>

I think this is not currently supported..

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Sachin Sawant
> Sent: Tuesday, June 23, 2009 1:59 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] template for file input module in Rsyslog
> 
> Hi All,
> 
> 
> 
> I would like to monitor PostgreSQL log files using file input module in
> Rsyslog. PostgreSQL create log files based on date, so is it possible to
> monitor file without using absolute path?
> 
> 
> 
> Regards,
> 
> * *
> 
> *Sachin Sawant
> 
> *
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From martinmie at PartyGaming.com  Tue Jun 23 17:07:35 2009
From: martinmie at PartyGaming.com (Martin Mielke)
Date: Tue, 23 Jun 2009 17:07:35 +0200
Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9A4@GRFEXC.intern.adiscon.com>
References: <0B1B9163138571439EAF804646F3F6AA0767973E@GIBSVWIN004X.partygaming.local>
	<9B6E2A8877C38245BFB15CC491A11DA706F9A4@GRFEXC.intern.adiscon.com>
Message-ID: <0B1B9163138571439EAF804646F3F6AA07878F4E@GIBSVWIN004X.partygaming.local>

Sorry for the noise.

It seems that the post is pending on some reply ;-)


Cheers,


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: 18 June 2009 15:11
> To: rsyslog-users
> Subject: Re: [rsyslog] syslog-ng filters and destinations under
rsyslog
> 
> FYI, there was also a forum post:
> 
>
http://kb.monitorware.com/syslog-filters-and-destinations-under-rsyslog-
> t9858
> .html
> 
> I'd appreciate if any other responses also go to the forum (so that
> others,
> when searching, find the complete info).
> 
> Rainer
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Martin Mielke
> > Sent: Thursday, June 18, 2009 2:18 PM
> > To: rsyslog-users
> > Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
> >
> > Hi all,
> >
> > I need to find a way to either convert or import syslog-ng.conf
files
> > into rsyslog or learn the syntax to classify/store the logfiles
> > depending on the source.
> >
> > For example, taken from a syslog-ng file:
> > --
> > source s_network { udp(); tcp(); };
> >
> > filter f_systems {
> > host("1\.2\.3\.\d+");
> > };
> >
> > filter f_switches {
> > host("5\.6\.7\.8|5\.6\.7\.9");
> > };
> >
> > destination d_systems {file("/systems.log"); };
> > destination d_switches {file("/switches.log"); };
> >
> > log { source(s_network); filter(f_systems); destination(d_systems);
};
> > log { source(s_network); filter(f_switches);
destination(d_switches); };
> > --
> >
> > I have read the online documentation but I'm unable to find the
right
> > answer...
> >
> > Thanks!
> >
> > Martin
> >
> >
> > This email and any attachments are confidential, and may be legally
> privileged
> > and protected by copyright. If you are not the intended recipient
> > dissemination or copying of this email is prohibited. If you have
> received
> > this in error, please notify the sender by replying by email and
then
> delete
> > the email completely from your system.
> >
> > Any views or opinions are solely those of the sender.  This
> communication
> is
> > not intended to form a binding contract unless expressly indicated
to
> the
> > contrary and properly authorised. Any actions taken on the basis of
this
> email
> > are at the recipient's own risk.
> >
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> yslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Tue Jun 23 17:23:05 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 23 Jun 2009 17:23:05 +0200
Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
References: <0B1B9163138571439EAF804646F3F6AA0767973E@GIBSVWIN004X.partygaming.local><9B6E2A8877C38245BFB15CC491A11DA706F9A4@GRFEXC.intern.adiscon.com>
	<0B1B9163138571439EAF804646F3F6AA07878F4E@GIBSVWIN004X.partygaming.local>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA11@GRFEXC.intern.adiscon.com>

Thanks. Overlooked, you now have a reply ;)

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Martin Mielke
> Sent: Tuesday, June 23, 2009 5:08 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] syslog-ng filters and destinations under rsyslog
> 
> Sorry for the noise.
> 
> It seems that the post is pending on some reply ;-)
> 
> 
> Cheers,
> 
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> > Sent: 18 June 2009 15:11
> > To: rsyslog-users
> > Subject: Re: [rsyslog] syslog-ng filters and destinations under
> rsyslog
> >
> > FYI, there was also a forum post:
> >
> >
> http://kb.monitorware.com/syslog-filters-and-destinations-under-rsyslog-
> > t9858
> > .html
> >
> > I'd appreciate if any other responses also go to the forum (so that
> > others,
> > when searching, find the complete info).
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > bounces at lists.adiscon.com] On Behalf Of Martin Mielke
> > > Sent: Thursday, June 18, 2009 2:18 PM
> > > To: rsyslog-users
> > > Subject: [rsyslog] syslog-ng filters and destinations under rsyslog
> > >
> > > Hi all,
> > >
> > > I need to find a way to either convert or import syslog-ng.conf
> files
> > > into rsyslog or learn the syntax to classify/store the logfiles
> > > depending on the source.
> > >
> > > For example, taken from a syslog-ng file:
> > > --
> > > source s_network { udp(); tcp(); };
> > >
> > > filter f_systems {
> > > host("1\.2\.3\.\d+");
> > > };
> > >
> > > filter f_switches {
> > > host("5\.6\.7\.8|5\.6\.7\.9");
> > > };
> > >
> > > destination d_systems {file("/systems.log"); };
> > > destination d_switches {file("/switches.log"); };
> > >
> > > log { source(s_network); filter(f_systems); destination(d_systems);
> };
> > > log { source(s_network); filter(f_switches);
> destination(d_switches); };
> > > --
> > >
> > > I have read the online documentation but I'm unable to find the
> right
> > > answer...
> > >
> > > Thanks!
> > >
> > > Martin
> > >
> > >
> > > This email and any attachments are confidential, and may be legally
> > privileged
> > > and protected by copyright. If you are not the intended recipient
> > > dissemination or copying of this email is prohibited. If you have
> > received
> > > this in error, please notify the sender by replying by email and
> then
> > delete
> > > the email completely from your system.
> > >
> > > Any views or opinions are solely those of the sender.  This
> > communication
> > is
> > > not intended to form a binding contract unless expressly indicated
> to
> > the
> > > contrary and properly authorised. Any actions taken on the basis of
> this
> > email
> > > are at the recipient's own risk.
> > >
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> > yslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Tue Jun 23 17:57:11 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 23 Jun 2009 17:57:11 +0200
Subject: [rsyslog] template for file input module in Rsyslog
References: <edc9be030906230459q4090383sc8427501d8a413a7@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706FA0D@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA17@GRFEXC.intern.adiscon.com>

I just checked, it is NOT supported.

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Tuesday, June 23, 2009 2:48 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] template for file input module in Rsyslog
> 
> I think this is not currently supported..
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Sachin Sawant
> > Sent: Tuesday, June 23, 2009 1:59 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: [rsyslog] template for file input module in Rsyslog
> >
> > Hi All,
> >
> >
> >
> > I would like to monitor PostgreSQL log files using file input module in
> > Rsyslog. PostgreSQL create log files based on date, so is it possible to
> > monitor file without using absolute path?
> >
> >
> >
> > Regards,
> >
> > * *
> >
> > *Sachin Sawant
> >
> > *
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From tbergfeld at hq.adiscon.com  Wed Jun 24 12:15:05 2009
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Wed, 24 Jun 2009 12:15:05 +0200
Subject: [rsyslog] rsyslog 4.3.2 (beta) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA31@GRFEXC.intern.adiscon.com>

Hi all,
The new rsyslog beta 4.3.2 has been released. This release consists of
several enhancements like the added generic network stream server for syslog,
capability to rebind the send socket of the UDP output section, a improved
testbench with a config switch and some bugfixes. This release is recommended
for all users of the beta branch.

Download:

http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-162.phtml

Changelog:

http://www.rsyslog.com/Article378.phtml

As always, feedback is appreciated.

Best regards,
Tom Bergfeld

Send this story to someone

--
Support
=======

Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html .


From rgerhards at hq.adiscon.com  Wed Jun 24 14:31:04 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 24 Jun 2009 14:31:04 +0200
Subject: [rsyslog] segfault in v5-devel git branch
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>

Hi all,

I just wanted to let you know that the v5-devel git branch currently does not
work correctly when running with multiple worker threads. It will segfault
rather quickly. I am currently working on that and hope to resolve it soon.
Still, I thought it is better to tell you before someone begins to test with
that version.

Rainer


From rgerhards at hq.adiscon.com  Thu Jun 25 16:30:27 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 25 Jun 2009 16:30:27 +0200
Subject: [rsyslog] segfault in v5-devel git branch
References: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA5A@GRFEXC.intern.adiscon.com>

Hi all,

I just wanted to let you know that I hopefully have fixed this issue - at
least in my lab as well as under the threading debugging no problem shows up.
But of course, things like these require confirmation from the field.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Wednesday, June 24, 2009 2:31 PM
> To: rsyslog-users
> Subject: [rsyslog] segfault in v5-devel git branch
> 
> Hi all,
> 
> I just wanted to let you know that the v5-devel git branch currently does
not
> work correctly when running with multiple worker threads. It will segfault
> rather quickly. I am currently working on that and hope to resolve it soon.
> Still, I thought it is better to tell you before someone begins to test
with
> that version.
> 
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From kenneho.ndu at gmail.com  Fri Jun 26 14:30:09 2009
From: kenneho.ndu at gmail.com (Kenneth Holter)
Date: Fri, 26 Jun 2009 14:30:09 +0200
Subject: [rsyslog] rsyslogd:too many tcp sessions - dropping incoming request
Message-ID: <c25f25140906260530q7d102391k1154b1e5dca7a385@mail.gmail.com>

Hello list.


I'm running rsyslog 2.0.6, and have suddenly started to get this error
message:


rsyslogd:too many tcp sessions - dropping incoming request

I edited the /etc/sysconfig/rsyslog file and changed "-t200" to "-t200,500".
Is there any (performance) reasons or something that says that max number of
sessions should be 200 (default). If not, I'll stick with 500..


Regards,
Kenneth Holter


From rgerhards at hq.adiscon.com  Fri Jun 26 14:39:45 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 26 Jun 2009 14:39:45 +0200
Subject: [rsyslog] rsyslogd:too many tcp sessions - dropping incoming
	request
References: <c25f25140906260530q7d102391k1154b1e5dca7a385@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA68@GRFEXC.intern.adiscon.com>

No, I don't think so (in recent builds it is not). If I remember correctly,
the ",500" sets the accept queue but not the max # of sessions.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kenneth Holter
> Sent: Friday, June 26, 2009 2:30 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] rsyslogd:too many tcp sessions - dropping incoming
request
> 
> Hello list.
> 
> 
> I'm running rsyslog 2.0.6, and have suddenly started to get this error
> message:
> 
> 
> rsyslogd:too many tcp sessions - dropping incoming request
> 
> I edited the /etc/sysconfig/rsyslog file and changed "-t200" to
"-t200,500".
> Is there any (performance) reasons or something that says that max number
of
> sessions should be 200 (default). If not, I'll stick with 500..
> 
> 
> Regards,
> Kenneth Holter
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From mbiebl at gmail.com  Fri Jun 26 14:44:29 2009
From: mbiebl at gmail.com (Michael Biebl)
Date: Fri, 26 Jun 2009 14:44:29 +0200
Subject: [rsyslog] segfault in v5-devel git branch
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>
Message-ID: <aa76cdac0906260544l24ddecexb7c3b35436b05875@mail.gmail.com>

2009/6/24 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Hi all,
>
> I just wanted to let you know that the v5-devel git branch currently does not
> work correctly when running with multiple worker threads. It will segfault
> rather quickly. I am currently working on that and hope to resolve it soon.
> Still, I thought it is better to tell you before someone begins to test with
> that version.

I'm not even able to compile the v5-devel branch :-/

  CC    stringbuf.o
In file included from zlibw.h:28,
                 from stream.h:72,
                 from obj.h:50,
                 from rsyslog.h:441,
                 from stringbuf.c:39:
/usr/include/zlib.h:1366: error: expected ?=?, ?,?, ?;?, ?asm? or
?__attribute__? before ?gzseek64?
/usr/include/zlib.h:1367: error: expected ?=?, ?,?, ?;?, ?asm? or
?__attribute__? before ?gztell64?
/usr/include/zlib.h:1368: error: expected declaration specifiers or
?...? before ?off64_t?
/usr/include/zlib.h:1369: error: expected declaration specifiers or
?...? before ?off64_t?
make[2]: *** [librsyslog_la-stringbuf.lo] Fehler 1
make[1]: *** [all-recursive] Fehler 1
make: *** [all] Fehler 2
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From rgerhards at hq.adiscon.com  Fri Jun 26 15:19:28 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 26 Jun 2009 15:19:28 +0200
Subject: [rsyslog] segfault in v5-devel git branch
References: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>
	<aa76cdac0906260544l24ddecexb7c3b35436b05875@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA6A@GRFEXC.intern.adiscon.com>

Can you please let me know what the offending line in your zlib.h include
file is? I guess there is a name conflict in one of the types used.

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Friday, June 26, 2009 2:44 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] segfault in v5-devel git branch
> 
> 2009/6/24 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Hi all,
> >
> > I just wanted to let you know that the v5-devel git branch currently does
> not
> > work correctly when running with multiple worker threads. It will
segfault
> > rather quickly. I am currently working on that and hope to resolve it
soon.
> > Still, I thought it is better to tell you before someone begins to test
with
> > that version.
> 
> I'm not even able to compile the v5-devel branch :-/
> 
>   CC    stringbuf.o
> In file included from zlibw.h:28,
>                  from stream.h:72,
>                  from obj.h:50,
>                  from rsyslog.h:441,
>                  from stringbuf.c:39:
> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or
> '__attribute__' before 'gzseek64'
> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or
> '__attribute__' before 'gztell64'
> /usr/include/zlib.h:1368: error: expected declaration specifiers or
> '...' before 'off64_t'
> /usr/include/zlib.h:1369: error: expected declaration specifiers or
> '...' before 'off64_t'
> make[2]: *** [librsyslog_la-stringbuf.lo] Fehler 1
> make[1]: *** [all-recursive] Fehler 1
> make: *** [all] Fehler 2
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From mbiebl at gmail.com  Fri Jun 26 15:36:18 2009
From: mbiebl at gmail.com (Michael Biebl)
Date: Fri, 26 Jun 2009 15:36:18 +0200
Subject: [rsyslog] segfault in v5-devel git branch
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA6A@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706FA3C@GRFEXC.intern.adiscon.com>
	<aa76cdac0906260544l24ddecexb7c3b35436b05875@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA706FA6A@GRFEXC.intern.adiscon.com>
Message-ID: <aa76cdac0906260636ia9a3cc2gba9a4bfc6fecdec4@mail.gmail.com>

#ifdef _LARGEFILE64_SOURCE
   ZEXTERN gzFile ZEXPORT gzopen64 OF((const char *, const char *));
   ZEXTERN off64_t ZEXPORT gzseek64 OF((gzFile, off64_t, int));
   ZEXTERN off64_t ZEXPORT gztell64 OF((gzFile));
   ZEXTERN uLong ZEXPORT adler32_combine64 OF((uLong, uLong, off64_t));
   ZEXTERN uLong ZEXPORT crc32_combine64 OF((uLong, uLong, off64_t));
#endif


2009/6/26 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Can you please let me know what the offending line in your zlib.h include
> file is? I guess there is a name conflict in one of the types used.
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
>> Sent: Friday, June 26, 2009 2:44 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] segfault in v5-devel git branch
>>
>> 2009/6/24 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> > Hi all,
>> >
>> > I just wanted to let you know that the v5-devel git branch currently does
>> not
>> > work correctly when running with multiple worker threads. It will
> segfault
>> > rather quickly. I am currently working on that and hope to resolve it
> soon.
>> > Still, I thought it is better to tell you before someone begins to test
> with
>> > that version.
>>
>> I'm not even able to compile the v5-devel branch :-/
>>
>> ? CC ? ?stringbuf.o
>> In file included from zlibw.h:28,
>> ? ? ? ? ? ? ? ? ?from stream.h:72,
>> ? ? ? ? ? ? ? ? ?from obj.h:50,
>> ? ? ? ? ? ? ? ? ?from rsyslog.h:441,
>> ? ? ? ? ? ? ? ? ?from stringbuf.c:39:
>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or
>> '__attribute__' before 'gzseek64'
>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or
>> '__attribute__' before 'gztell64'
>> /usr/include/zlib.h:1368: error: expected declaration specifiers or
>> '...' before 'off64_t'
>> /usr/include/zlib.h:1369: error: expected declaration specifiers or
>> '...' before 'off64_t'
>> make[2]: *** [librsyslog_la-stringbuf.lo] Fehler 1
>> make[1]: *** [all-recursive] Fehler 1
>> make: *** [all] Fehler 2
>> --
>> Why is it that all of the instruments seeking intelligent life in the
>> universe are pointed away from Earth?
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>



-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


From iamsayan at gmail.com  Mon Jun 29 18:25:18 2009
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Mon, 29 Jun 2009 12:25:18 -0400
Subject: [rsyslog] How to test rsyslog performance
Message-ID: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com>

Hi,
I saw the announcement of the 4.2.0 stable version of rsyslog. I was
wondering if there is a way I can test the performance of rsyslog with my
own rsyslog.conf.
The way I am trying to use rsyslog is on a local system, which has following
rules

logging different facilities to different files with different formats.
forwarding certain facilities to an external server.
receive messages from one(and only one) other machine over udp.

Are there any tool/scripts available which I can use to perform the tests
myself?
Basically what I am thinking of is running a script which writes is bunch of
messages to syslog, but then how do I make sure all those messages were
actually logged and not lost? It may be very painful to look through the
entire log file 1 message at a time :)
Is there a stat/counter based api which shows me how many messages were
logged?

Regards,
Sayan


From david at lang.hm  Mon Jun 29 18:46:53 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 29 Jun 2009 09:46:53 -0700 (PDT)
Subject: [rsyslog] How to test rsyslog performance
In-Reply-To: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com>
References: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906290943480.9159@asgard>

On Mon, 29 Jun 2009, Sayan Chowdhury wrote:

> Hi,
> I saw the announcement of the 4.2.0 stable version of rsyslog. I was
> wondering if there is a way I can test the performance of rsyslog with my
> own rsyslog.conf.
> The way I am trying to use rsyslog is on a local system, which has following
> rules
>
> logging different facilities to different files with different formats.
> forwarding certain facilities to an external server.
> receive messages from one(and only one) other machine over udp.
>
> Are there any tool/scripts available which I can use to perform the tests
> myself?
> Basically what I am thinking of is running a script which writes is bunch of
> messages to syslog, but then how do I make sure all those messages were
> actually logged and not lost? It may be very painful to look through the
> entire log file 1 message at a time :)
> Is there a stat/counter based api which shows me how many messages were
> logged?

what I do is to send a known quantity of messages and include a identifier 
string in the message ('rsyslog-test-message' for example), and then grep 
the resulting files to make sure that the right number of messages arrive.

to send messages to rsyslog I have used tcpreplay to replay UDP messages. 
I first setup a system to send the messages via UDP, start tcpdump, send 
the messages to the local box (which then sends them via UDP to a remote 
server), then I stop tcpdump and can use the resulting capture file with 
tcpreplay to replay the messages at different speeds.

David Lang


From iamsayan at gmail.com  Mon Jun 29 19:55:15 2009
From: iamsayan at gmail.com (Sayan Chowdhury)
Date: Mon, 29 Jun 2009 13:55:15 -0400
Subject: [rsyslog] How to test rsyslog performance
In-Reply-To: <alpine.DEB.1.10.0906290943480.9159@asgard>
References: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com>
	<alpine.DEB.1.10.0906290943480.9159@asgard>
Message-ID: <87a1ae540906291055n7625b369naf02a1576e8ee187@mail.gmail.com>

Hello David,
Thanks a bunch ..
I will try this, I was also thinking in similar lines, in the sense that I
will log 1 line of message, set the file size to be huge, and then count the
number of lines on the file.
I just read the numbers on Rainer's blog, and I think I will need the
tcprelay approach as well, there is no way I can generate so much load with
a shell script.
Thanks for the tip.
Regadrs,
Sayan


On Mon, Jun 29, 2009 at 12:46 PM, <david at lang.hm> wrote:

> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
>
> > Hi,
> > I saw the announcement of the 4.2.0 stable version of rsyslog. I was
> > wondering if there is a way I can test the performance of rsyslog with my
> > own rsyslog.conf.
> > The way I am trying to use rsyslog is on a local system, which has
> following
> > rules
> >
> > logging different facilities to different files with different formats.
> > forwarding certain facilities to an external server.
> > receive messages from one(and only one) other machine over udp.
> >
> > Are there any tool/scripts available which I can use to perform the tests
> > myself?
> > Basically what I am thinking of is running a script which writes is bunch
> of
> > messages to syslog, but then how do I make sure all those messages were
> > actually logged and not lost? It may be very painful to look through the
> > entire log file 1 message at a time :)
> > Is there a stat/counter based api which shows me how many messages were
> > logged?
>
> what I do is to send a known quantity of messages and include a identifier
> string in the message ('rsyslog-test-message' for example), and then grep
> the resulting files to make sure that the right number of messages arrive.
>
> to send messages to rsyslog I have used tcpreplay to replay UDP messages.
> I first setup a system to send the messages via UDP, start tcpdump, send
> the messages to the local box (which then sends them via UDP to a remote
> server), then I stop tcpdump and can use the resulting capture file with
> tcpreplay to replay the messages at different speeds.
>
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From david at lang.hm  Mon Jun 29 20:03:33 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 29 Jun 2009 11:03:33 -0700 (PDT)
Subject: [rsyslog] How to test rsyslog performance
In-Reply-To: <87a1ae540906291055n7625b369naf02a1576e8ee187@mail.gmail.com>
References: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com>
	<alpine.DEB.1.10.0906290943480.9159@asgard>
	<87a1ae540906291055n7625b369naf02a1576e8ee187@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.0906291100010.9159@asgard>

On Mon, 29 Jun 2009, Sayan Chowdhury wrote:

> Hello David,
> Thanks a bunch ..
> I will try this, I was also thinking in similar lines, in the sense that I
> will log 1 line of message, set the file size to be huge, and then count the
> number of lines on the file.
> I just read the numbers on Rainer's blog, and I think I will need the
> tcprelay approach as well, there is no way I can generate so much load with
> a shell script.
> Thanks for the tip.

with an earlier version of the 4.x series I was seeing numbers like the 
following

with 256 byte messages

recieving via UDP and storing in the queue, >300K/sec (wire speed, gig-E)

recieving via UDP and writing to disk, ~78K/sec

recieving via UDP, writing to disk and sending to another machine, ~30k/sec

since then there has been a lot of work to optimize the queue functions 
that appeared to be the bottleneck in my testing, and so I am very 
interested in seeing the new results (I expect to be testing shortly, but 
will be out much of this week, one of my 'hobbies' is doing professional 
fireworks, and so I will be busy preparing for the July 4th shows)

David Lang


> Regadrs,
> Sayan
>
>
> On Mon, Jun 29, 2009 at 12:46 PM, <david at lang.hm> wrote:
>
>> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
>>
>>> Hi,
>>> I saw the announcement of the 4.2.0 stable version of rsyslog. I was
>>> wondering if there is a way I can test the performance of rsyslog with my
>>> own rsyslog.conf.
>>> The way I am trying to use rsyslog is on a local system, which has
>> following
>>> rules
>>>
>>> logging different facilities to different files with different formats.
>>> forwarding certain facilities to an external server.
>>> receive messages from one(and only one) other machine over udp.
>>>
>>> Are there any tool/scripts available which I can use to perform the tests
>>> myself?
>>> Basically what I am thinking of is running a script which writes is bunch
>> of
>>> messages to syslog, but then how do I make sure all those messages were
>>> actually logged and not lost? It may be very painful to look through the
>>> entire log file 1 message at a time :)
>>> Is there a stat/counter based api which shows me how many messages were
>>> logged?
>>
>> what I do is to send a known quantity of messages and include a identifier
>> string in the message ('rsyslog-test-message' for example), and then grep
>> the resulting files to make sure that the right number of messages arrive.
>>
>> to send messages to rsyslog I have used tcpreplay to replay UDP messages.
>> I first setup a system to send the messages via UDP, start tcpdump, send
>> the messages to the local box (which then sends them via UDP to a remote
>> server), then I stop tcpdump and can use the resulting capture file with
>> tcpreplay to replay the messages at different speeds.
>>
>> David Lang
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From rgerhards at hq.adiscon.com  Mon Jun 29 21:33:54 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 29 Jun 2009 21:33:54 +0200
Subject: [rsyslog] How to test rsyslog performance
References: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com><alpine.DEB.1.10.0906290943480.9159@asgard><87a1ae540906291055n7625b369naf02a1576e8ee187@mail.gmail.com>
	<alpine.DEB.1.10.0906291100010.9159@asgard>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA8A@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Monday, June 29, 2009 8:04 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] How to test rsyslog performance
> 
> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
> 
> > Hello David,
> > Thanks a bunch ..
> > I will try this, I was also thinking in similar lines, in 
> the sense that I
> > will log 1 line of message, set the file size to be huge, 
> and then count the
> > number of lines on the file.
> > I just read the numbers on Rainer's blog, and I think I 
> will need the
> > tcprelay approach as well, there is no way I can generate 
> so much load with
> > a shell script.
> > Thanks for the tip.
> 
> with an earlier version of the 4.x series I was seeing 
> numbers like the 
> following
> 
> with 256 byte messages
> 
> recieving via UDP and storing in the queue, >300K/sec (wire 
> speed, gig-E)
> 
> recieving via UDP and writing to disk, ~78K/sec
> 
> recieving via UDP, writing to disk and sending to another 
> machine, ~30k/sec
> 
> since then there has been a lot of work to optimize the queue 
> functions 
> that appeared to be the bottleneck in my testing,

In the mean time, I have identified at least one more bottleneck, and this
was alloc/free. These come with implicit synchroniation primitives (to
protect the heap) and it turned out that they cause quite some performance
loss as soon as we have enough threads running concurrently. I have removed
much of these alloc calls during the past two weeks, and the results - at
least in my non-high-end lab - are quite pleasant. This work is still not
100% finished, but I think I have reched the mark where further reduction of
alloc does not case any notable performance increase. I'll still continue to
finish up some things (so that the code base is consistent), but I don't
expect much performance increase from that. 

All in all, the "new v4" engine should perform around twice as fast as the
"old v4" engine (depending a bit on configuration). The v5 engine offers more
advanced optimizations and will receive even more of them. At this time, the
v5 engine in my lab (with TCP!) is at least twice as fast as the "new v4"
engine, probably a bit less with UDP.

There is still ample of tuning potential. Two things that immediately come to
my mind is that the inputs still use select() [originally used for
portability reasons] and the inputs themselves are single-threaded. For v5 I
can also envision a new malloc subsytem, that will do 1/1024th malloc per
message (right now, we do around 5 mallocs per message, tomorrow evening
probably only 3).

If someone is looking into performance, one should definitely look into the
v5 engine. This version is experimental, but if nobody tries it out, it can't
improve (everything I could do in my lab I have done, but I know from past
experience that the real world and high-end hardware is a different story).
But even the test with the threading debugger went quite well. I'll release a
new v5 shortly, and that one is definitely worth a try (if you need it
urgent, just let me know).

> and so I am very 
> interested in seeing the new results (I expect to be testing 
> shortly, but 
> will be out much of this week, one of my 'hobbies' is doing 
> professional 
> fireworks, and so I will be busy preparing for the July 4th shows)

Cool - wish I were there ;)

Rainer 

> David Lang
> 
> 
> > Regadrs,
> > Sayan
> >
> >
> > On Mon, Jun 29, 2009 at 12:46 PM, <david at lang.hm> wrote:
> >
> >> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
> >>
> >>> Hi,
> >>> I saw the announcement of the 4.2.0 stable version of 
> rsyslog. I was
> >>> wondering if there is a way I can test the performance of 
> rsyslog with my
> >>> own rsyslog.conf.
> >>> The way I am trying to use rsyslog is on a local system, which has
> >> following
> >>> rules
> >>>
> >>> logging different facilities to different files with 
> different formats.
> >>> forwarding certain facilities to an external server.
> >>> receive messages from one(and only one) other machine over udp.
> >>>
> >>> Are there any tool/scripts available which I can use to 
> perform the tests
> >>> myself?
> >>> Basically what I am thinking of is running a script which 
> writes is bunch
> >> of
> >>> messages to syslog, but then how do I make sure all those 
> messages were
> >>> actually logged and not lost? It may be very painful to 
> look through the
> >>> entire log file 1 message at a time :)
> >>> Is there a stat/counter based api which shows me how many 
> messages were
> >>> logged?
> >>
> >> what I do is to send a known quantity of messages and 
> include a identifier
> >> string in the message ('rsyslog-test-message' for 
> example), and then grep
> >> the resulting files to make sure that the right number of 
> messages arrive.
> >>
> >> to send messages to rsyslog I have used tcpreplay to 
> replay UDP messages.
> >> I first setup a system to send the messages via UDP, start 
> tcpdump, send
> >> the messages to the local box (which then sends them via 
> UDP to a remote
> >> server), then I stop tcpdump and can use the resulting 
> capture file with
> >> tcpreplay to replay the messages at different speeds.
> >>
> >> David Lang
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> 


From david at lang.hm  Mon Jun 29 22:18:50 2009
From: david at lang.hm (david at lang.hm)
Date: Mon, 29 Jun 2009 13:18:50 -0700 (PDT)
Subject: [rsyslog] How to test rsyslog performance
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA8A@GRFEXC.intern.adiscon.com>
References: <87a1ae540906290925s2652f647wa90f9cd2455ec202@mail.gmail.com><alpine.DEB.1.10.0906290943480.9159@asgard><87a1ae540906291055n7625b369naf02a1576e8ee187@mail.gmail.com>
	<alpine.DEB.1.10.0906291100010.9159@asgard>
	<9B6E2A8877C38245BFB15CC491A11DA706FA8A@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906291317300.9159@asgard>

On Mon, 29 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
>>
>>> Hello David,
>>> Thanks a bunch ..
>>> I will try this, I was also thinking in similar lines, in
>> the sense that I
>>> will log 1 line of message, set the file size to be huge,
>> and then count the
>>> number of lines on the file.
>>> I just read the numbers on Rainer's blog, and I think I
>> will need the
>>> tcprelay approach as well, there is no way I can generate
>> so much load with
>>> a shell script.
>>> Thanks for the tip.
>>
>> with an earlier version of the 4.x series I was seeing
>> numbers like the
>> following
>>
>> with 256 byte messages
>>
>> recieving via UDP and storing in the queue, >300K/sec (wire
>> speed, gig-E)
>>
>> recieving via UDP and writing to disk, ~78K/sec
>>
>> recieving via UDP, writing to disk and sending to another
>> machine, ~30k/sec
>>
>> since then there has been a lot of work to optimize the queue
>> functions
>> that appeared to be the bottleneck in my testing,
>
> In the mean time, I have identified at least one more bottleneck, and this
> was alloc/free. These come with implicit synchroniation primitives (to
> protect the heap) and it turned out that they cause quite some performance
> loss as soon as we have enough threads running concurrently. I have removed
> much of these alloc calls during the past two weeks, and the results - at
> least in my non-high-end lab - are quite pleasant. This work is still not
> 100% finished, but I think I have reched the mark where further reduction of
> alloc does not case any notable performance increase. I'll still continue to
> finish up some things (so that the code base is consistent), but I don't
> expect much performance increase from that.
>
> All in all, the "new v4" engine should perform around twice as fast as the
> "old v4" engine (depending a bit on configuration). The v5 engine offers more
> advanced optimizations and will receive even more of them. At this time, the
> v5 engine in my lab (with TCP!) is at least twice as fast as the "new v4"
> engine, probably a bit less with UDP.
>
> There is still ample of tuning potential. Two things that immediately come to
> my mind is that the inputs still use select() [originally used for
> portability reasons] and the inputs themselves are single-threaded. For v5 I
> can also envision a new malloc subsytem, that will do 1/1024th malloc per
> message (right now, we do around 5 mallocs per message, tomorrow evening
> probably only 3).
>
> If someone is looking into performance, one should definitely look into the
> v5 engine. This version is experimental, but if nobody tries it out, it can't
> improve (everything I could do in my lab I have done, but I know from past
> experience that the real world and high-end hardware is a different story).
> But even the test with the threading debugger went quite well. I'll release a
> new v5 shortly, and that one is definitely worth a try (if you need it
> urgent, just let me know).

I tried to compile the v5 branch over the weekend, but got a compile 
error, so I just moved to the 4.2.0 release. on the ultra-reliable testbed 
it's getting ~8500/sec.

David Lang

>> and so I am very
>> interested in seeing the new results (I expect to be testing
>> shortly, but
>> will be out much of this week, one of my 'hobbies' is doing
>> professional
>> fireworks, and so I will be busy preparing for the July 4th shows)
>
> Cool - wish I were there ;)
>
> Rainer
>
>> David Lang
>>
>>
>>> Regadrs,
>>> Sayan
>>>
>>>
>>> On Mon, Jun 29, 2009 at 12:46 PM, <david at lang.hm> wrote:
>>>
>>>> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
>>>>
>>>>> Hi,
>>>>> I saw the announcement of the 4.2.0 stable version of
>> rsyslog. I was
>>>>> wondering if there is a way I can test the performance of
>> rsyslog with my
>>>>> own rsyslog.conf.
>>>>> The way I am trying to use rsyslog is on a local system, which has
>>>> following
>>>>> rules
>>>>>
>>>>> logging different facilities to different files with
>> different formats.
>>>>> forwarding certain facilities to an external server.
>>>>> receive messages from one(and only one) other machine over udp.
>>>>>
>>>>> Are there any tool/scripts available which I can use to
>> perform the tests
>>>>> myself?
>>>>> Basically what I am thinking of is running a script which
>> writes is bunch
>>>> of
>>>>> messages to syslog, but then how do I make sure all those
>> messages were
>>>>> actually logged and not lost? It may be very painful to
>> look through the
>>>>> entire log file 1 message at a time :)
>>>>> Is there a stat/counter based api which shows me how many
>> messages were
>>>>> logged?
>>>>
>>>> what I do is to send a known quantity of messages and
>> include a identifier
>>>> string in the message ('rsyslog-test-message' for
>> example), and then grep
>>>> the resulting files to make sure that the right number of
>> messages arrive.
>>>>
>>>> to send messages to rsyslog I have used tcpreplay to
>> replay UDP messages.
>>>> I first setup a system to send the messages via UDP, start
>> tcpdump, send
>>>> the messages to the local box (which then sends them via
>> UDP to a remote
>>>> server), then I stop tcpdump and can use the resulting
>> capture file with
>>>> tcpreplay to replay the messages at different speeds.
>>>>
>>>> David Lang
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>


From sfzhi at yahoo.com  Tue Jun 30 11:11:14 2009
From: sfzhi at yahoo.com (Sergei Zhirikov)
Date: Tue, 30 Jun 2009 11:11:14 +0200
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
Message-ID: <h2ckri$16j$1@ger.gmane.org>

Rainer Gerhards wrote:
> Hi all,
> 
> The most prominent use case is to size-limit files. $OutChannel provides the
> ability to execute a command when a certain file size is reached. The command
> may then rotate the file. After command execution, omfile checks the file
> size again. If it still is above the configured limit, this file will be
> disabled (no longer be written to until restart).
> 
> If you use this for size-limiting, please let me know how you do it. Please
> also let me know if you would like to see things handled differently.
> 
> Looking forward to your feedback.
> 
> Thanks,
> Rainer
> 

Hi Rainer,

I do use $OutChannel for size-limiting and it would be a disappointment 
to me if that feature disappeared. As a matter of fact, it was the main 
reason why I chose to use rsyslog in the first place.

I find the "classic" way of using cron+logrotate+SIGHUP quite ugly and 
in some cases unacceptable. I have a small home gateway with read-only 
main file system, so all the log files are written into a tmpfs mount. 
It happened a couple of times that some misconfigured or malfunctioning 
software produced a lot of logging and filled in most of the available 
RAM before the next scheduled run of logrotate. In this situation the 
size limiting feature of rsyslog is exactly what I need in order to be 
able to avoid such this kind of problems.

Kind regards,
Sergei.



From david at lang.hm  Tue Jun 30 12:35:58 2009
From: david at lang.hm (david at lang.hm)
Date: Tue, 30 Jun 2009 03:35:58 -0700 (PDT)
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <h2ckri$16j$1@ger.gmane.org>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
	<h2ckri$16j$1@ger.gmane.org>
Message-ID: <alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>

On Tue, 30 Jun 2009, Sergei Zhirikov wrote:

> Rainer Gerhards wrote:
>> Hi all,
>>
>> The most prominent use case is to size-limit files. $OutChannel provides the
>> ability to execute a command when a certain file size is reached. The command
>> may then rotate the file. After command execution, omfile checks the file
>> size again. If it still is above the configured limit, this file will be
>> disabled (no longer be written to until restart).
>>
>> If you use this for size-limiting, please let me know how you do it. Please
>> also let me know if you would like to see things handled differently.
>>
>> Looking forward to your feedback.
>>
>> Thanks,
>> Rainer
>>
>
> Hi Rainer,
>
> I do use $OutChannel for size-limiting and it would be a disappointment
> to me if that feature disappeared. As a matter of fact, it was the main
> reason why I chose to use rsyslog in the first place.
>
> I find the "classic" way of using cron+logrotate+SIGHUP quite ugly and
> in some cases unacceptable. I have a small home gateway with read-only
> main file system, so all the log files are written into a tmpfs mount.
> It happened a couple of times that some misconfigured or malfunctioning
> software produced a lot of logging and filled in most of the available
> RAM before the next scheduled run of logrotate. In this situation the
> size limiting feature of rsyslog is exactly what I need in order to be
> able to avoid such this kind of problems.

this is a very legitimate use. I suspect the biggest issue is how to give 
you this capability without slowing down the people who don't use it.

I wonder if rainerscript could be extended to where it could be used to do 
this?

something like:

for each message logged, add the length of the message to a counter, and 
have an if statement that calls the external program to roll the log and 
reset the counter.

David Lang


From rgerhards at hq.adiscon.com  Tue Jun 30 12:43:14 2009
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 30 Jun 2009 12:43:14 +0200
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com><h2ckri$16j$1@ger.gmane.org>
	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, June 30, 2009 12:36 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] omfile redesign / $OutChannel will go away!
> 
> On Tue, 30 Jun 2009, Sergei Zhirikov wrote:
> 
> > Rainer Gerhards wrote:
> >> Hi all,
> >>
> >> The most prominent use case is to size-limit files. $OutChannel provides
> the
> >> ability to execute a command when a certain file size is reached. The
> command
> >> may then rotate the file. After command execution, omfile checks the
file
> >> size again. If it still is above the configured limit, this file will be
> >> disabled (no longer be written to until restart).
> >>
> >> If you use this for size-limiting, please let me know how you do it.
Please
> >> also let me know if you would like to see things handled differently.
> >>
> >> Looking forward to your feedback.
> >>
> >> Thanks,
> >> Rainer
> >>
> >
> > Hi Rainer,
> >
> > I do use $OutChannel for size-limiting and it would be a disappointment
> > to me if that feature disappeared. As a matter of fact, it was the main
> > reason why I chose to use rsyslog in the first place.
> >
> > I find the "classic" way of using cron+logrotate+SIGHUP quite ugly and
> > in some cases unacceptable. I have a small home gateway with read-only
> > main file system, so all the log files are written into a tmpfs mount.
> > It happened a couple of times that some misconfigured or malfunctioning
> > software produced a lot of logging and filled in most of the available
> > RAM before the next scheduled run of logrotate. In this situation the
> > size limiting feature of rsyslog is exactly what I need in order to be
> > able to avoid such this kind of problems.
> 
> this is a very legitimate use. I suspect the biggest issue is how to give
> you this capability without slowing down the people who don't use it.
> 
> I wonder if rainerscript could be extended to where it could be used to do
> this?
> 
> something like:
> 
> for each message logged, add the length of the message to a counter, and
> have an if statement that calls the external program to roll the log and
> reset the counter.

I probably should have phrased the initial question less agressive, but than
I'd probably not received any feedback. From the feedback I received so far,
there seem to be two use cases:

a) circular logs
b) stop logging when file hits max size (was that really what this post
described?)

In any case, these two seem to be the only things that are done with
outchannel. So when $outchannel goes away (and it will at some time), the
"regular" file writer will have options to support these use cases - probably
more elegant than before. For example, for case a), circular logs, I do not
need to involve an external process, so I can remove that feature without
harm, as long as I replace it with something suitable (and think that omfile
now uses the stream writer, which *already does* circular "logging" for disk
queues).

So functionality in actual need will not go away, as long as people tell what
they need.

If I look at the b) case, I suspect that this is not even the intent of the
original poster. I think he would be much better served with a circular log
that records as many log records as fit into a finite, operator-specified log
file size. Or am I wrong here? (If nobody tells me I am wrong, I'll probably
remove the b) use case in the long term!).

Rainer



From david at lang.hm  Tue Jun 30 13:03:54 2009
From: david at lang.hm (david at lang.hm)
Date: Tue, 30 Jun 2009 04:03:54 -0700 (PDT)
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com><h2ckri$16j$1@ger.gmane.org>
	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.1.10.0906300354370.11336@asgard.lang.hm>

On Tue, 30 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Tue, 30 Jun 2009, Sergei Zhirikov wrote:
>>
>>> Rainer Gerhards wrote:
>>>> Hi all,
>>>>
>>>> The most prominent use case is to size-limit files. $OutChannel provides
>> the
>>>> ability to execute a command when a certain file size is reached. The
>> command
>>>> may then rotate the file. After command execution, omfile checks the
> file
>>>> size again. If it still is above the configured limit, this file will be
>>>> disabled (no longer be written to until restart).
>>>>
>>>> If you use this for size-limiting, please let me know how you do it.
> Please
>>>> also let me know if you would like to see things handled differently.
>>>>
>>>> Looking forward to your feedback.
>>>>
>>>> Thanks,
>>>> Rainer
>>>>
>>>
>>> Hi Rainer,
>>>
>>> I do use $OutChannel for size-limiting and it would be a disappointment
>>> to me if that feature disappeared. As a matter of fact, it was the main
>>> reason why I chose to use rsyslog in the first place.
>>>
>>> I find the "classic" way of using cron+logrotate+SIGHUP quite ugly and
>>> in some cases unacceptable. I have a small home gateway with read-only
>>> main file system, so all the log files are written into a tmpfs mount.
>>> It happened a couple of times that some misconfigured or malfunctioning
>>> software produced a lot of logging and filled in most of the available
>>> RAM before the next scheduled run of logrotate. In this situation the
>>> size limiting feature of rsyslog is exactly what I need in order to be
>>> able to avoid such this kind of problems.
>>
>> this is a very legitimate use. I suspect the biggest issue is how to give
>> you this capability without slowing down the people who don't use it.
>>
>> I wonder if rainerscript could be extended to where it could be used to do
>> this?
>>
>> something like:
>>
>> for each message logged, add the length of the message to a counter, and
>> have an if statement that calls the external program to roll the log and
>> reset the counter.
>
> I probably should have phrased the initial question less agressive, but than
> I'd probably not received any feedback. From the feedback I received so far,
> there seem to be two use cases:
>
> a) circular logs
> b) stop logging when file hits max size (was that really what this post
> described?)

no, this is a third case
c) limit the size of the logfile that is being added to, and when it hits 
this limit roll it off to permanent storage (at which point the file 
becomes read-only)

this is a need at both the low-end and the high-end

for this poster, he needs it at the low end as his 'working' files are in 
ram (his /var/log/messages), but then he rolls it off to network storage.

at this high end you can have large volumes of logs arriving, and want to 
have ver fast (but expensive) storage to handle the high file modification 
rate, but want to make sure that you don't run out of space there by 
rolling the logs off to slower, but larger disks so that they don't 
fillthe fast drive and cause you to loose logs.

> In any case, these two seem to be the only things that are done with
> outchannel. So when $outchannel goes away (and it will at some time), the
> "regular" file writer will have options to support these use cases - probably
> more elegant than before. For example, for case a), circular logs, I do not
> need to involve an external process, so I can remove that feature without
> harm, as long as I replace it with something suitable (and think that omfile
> now uses the stream writer, which *already does* circular "logging" for disk
> queues).

that sounds reasonable

> So functionality in actual need will not go away, as long as people tell what
> they need.
>
> If I look at the b) case, I suspect that this is not even the intent of the
> original poster. I think he would be much better served with a circular log
> that records as many log records as fit into a finite, operator-specified log
> file size. Or am I wrong here? (If nobody tells me I am wrong, I'll probably
> remove the b) use case in the long term!).

no, it's not that total space for logs is limited to a fixed amount, it's 
that you have two spaces for logs, a small fast space and a large slow 
space, and you need to not run out of the fast space by moving logs to the 
slow space 'soon enough'


from a security point of view there are two arguments about option b

on the one hand, if a bad guy can run you out of space he can then do new 
things that aren't logged.

on the other hand, if you have a circular buffer, the bad buy can generate 
enough logs to cause you to overwrite the evidence of him breaking in to 
your system

there are places for both.

David Lang


From sfzhi at yahoo.com  Tue Jun 30 15:24:53 2009
From: sfzhi at yahoo.com (Sergei Zhirikov)
Date: Tue, 30 Jun 2009 15:24:53 +0200
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com><h2ckri$16j$1@ger.gmane.org>	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
Message-ID: <h2d3n6$cp5$1@ger.gmane.org>

> From the feedback I received so far, there seem to be two use cases:
> 
> a) circular logs
> b) stop logging when file hits max size (was that really what this post
> described?)
> 
> If I look at the b) case, I suspect that this is not even the intent of the
> original poster. I think he would be much better served with a circular log
> that records as many log records as fit into a finite, operator-specified log
> file size. Or am I wrong here? (If nobody tells me I am wrong, I'll probably
> remove the b) use case in the long term!).
> 
> Rainer
> 

In the case of log flood neither of the two options is perfect. The 
difference is only in what you lose, the oldest or the newest log 
entries. What I really like about the current rsyslog functionality is 
the possibility to call an external process to deal with the oversized 
logs. Stopping logging I consider only a safeguard in case that external 
process fails to do its job for whatever reason.

Having an external process makes the whole setup a lot more flexible, 
because it gives almost unlimited opportunities, for example:

   * gzip the older logs rather than delete them. In case of log flood 
gzipping is usually very efficient, because the same message is repeated 
in the log multiple times, so there is no need to delete anything.
   * filter the older logs based on conditions that involve analyzing 
more than one log entry at the same time. This can be useful, for 
example, do detect break-in attempts if desired.
   * measure the average speed of logging and send an email or generate 
another kind of alert when the logs are growing unusually fast.
   * make a decision whether to gzip the older logs, to move them to 
another location (e.g. a network storage, if available), or to delete 
them based on conditions outside of control of rsyslog (e.g. available 
system resources, availability of alternative storage, etc.)

The list can go on infinitely. I believe the possibility to call an 
external process to handle log rotation gives the flexibility that can 
not be provided within rsyslog itself, because there are so many 
different things one might want to do. And I would very much like to see 
this functionality stay in one or another form. Hope I'm not the only 
one who thinks that way.

Kind regards,
Sergei.



From epiphani at gmail.com  Tue Jun 30 15:38:11 2009
From: epiphani at gmail.com (Aaron Wiebe)
Date: Tue, 30 Jun 2009 09:38:11 -0400
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <h2d3n6$cp5$1@ger.gmane.org>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
	<h2ckri$16j$1@ger.gmane.org>
	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
	<h2d3n6$cp5$1@ger.gmane.org>
Message-ID: <e7ca40f70906300638n2b7dd90eo49429daec8caddc2@mail.gmail.com>

On Tue, Jun 30, 2009 at 9:24 AM, Sergei Zhirikov<sfzhi at yahoo.com> wrote:
>
> ? * gzip the older logs rather than delete them. In case of log flood
> gzipping is usually very efficient, because the same message is repeated
> in the log multiple times, so there is no need to delete anything.

One thing worth noting is that the new omfile branch contains changes
that will allow gzipping of files automatically out of rsyslogd, which
removes the requirement to do it after writing to disk.  :)

-Aaron


From aoz.syn at gmail.com  Tue Jun 30 16:44:55 2009
From: aoz.syn at gmail.com (RB)
Date: Tue, 30 Jun 2009 08:44:55 -0600
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>
	<h2ckri$16j$1@ger.gmane.org>
	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
Message-ID: <4255c2570906300744h5db55712m135c67aa2d15bcea@mail.gmail.com>

On Tue, Jun 30, 2009 at 04:43, Rainer Gerhards<rgerhards at hq.adiscon.com> wrote:
> a) circular logs

This.  I keep getting distracted with other projects, but have had on
my TODO list to add an 'omclog' output module that both handles the
*BSD clog format and extends it for some particular shortcomings.
Specifically, I'm thinking of embedded concerns here - compression,
ringing on number of records, and a few miscallaneous implementation
oversights it (clog) has.

> b) stop logging when file hits max size (was that really what this post
> described?)

As noted, most people want this use-case to cut off logs when they hit
a particular size so they may be archived/processed/etc.  I honestly
couldn't care less whether rsyslog has internal compression
capability, as long as it closes the handle and releases it to me to
deal with.  Maybe a hook to call a program when the size trigger is
hit, but even with [di]notify that becomes less necessary.


From sfzhi at yahoo.com  Tue Jun 30 18:12:28 2009
From: sfzhi at yahoo.com (Sergei Zhirikov)
Date: Tue, 30 Jun 2009 18:12:28 +0200
Subject: [rsyslog] omfile redesign / $OutChannel will go away!
In-Reply-To: <4255c2570906300744h5db55712m135c67aa2d15bcea@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA706F86B@GRFEXC.intern.adiscon.com>	<h2ckri$16j$1@ger.gmane.org>	<alpine.DEB.1.10.0906300332570.11336@asgard.lang.hm>	<9B6E2A8877C38245BFB15CC491A11DA706FA90@GRFEXC.intern.adiscon.com>
	<4255c2570906300744h5db55712m135c67aa2d15bcea@mail.gmail.com>
Message-ID: <h2ddhf$8q5$1@ger.gmane.org>

RB wrote:
> I honestly couldn't care less whether rsyslog has internal compression
> capability, as long as it closes the handle and releases it to me to
> deal with.  Maybe a hook to call a program when the size trigger is
> hit, but even with [di]notify that becomes less necessary.

I completely agree.

By the way, I use the inotify method to handle logs written by Samba 
server, which does have a log size limit, but does not call an external 
program when that limit is exceeded. So far I did not have any problems 
with that setup.

--
Sergei.



From david at lang.hm  Tue Jun 30 23:46:38 2009
From: david at lang.hm (david at lang.hm)
Date: Tue, 30 Jun 2009 14:46:38 -0700 (PDT)
Subject: [rsyslog] v5-devel won't compile
Message-ID: <alpine.DEB.1.10.0906301445160.9159@asgard>

ubuntu 9.04

zlib version
/* zlib.h -- interface of the 'zlib' general purpose compression library
   version 1.2.3.3, October 2nd, 2006

ii  zlib1g-dev                          1:1.2.3.3.dfsg-12 
compression library - development


error

In file included from zlibw.h:28,
                  from stream.h:72,
                  from obj.h:50,
                  from rsyslog.h:482,
                  from stringbuf.c:39:
/usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or 
'__attribute__' before 'gzseek64'
/usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or 
'__attribute__' before 'gztell64'
/usr/include/zlib.h:1368: error: expected declaration specifiers or '...' 
before 'off64_t'
/usr/include/zlib.h:1369: error: expected declaration specifiers or '...' 
before 'off64_t'
make[2]: *** [librsyslog_la-stringbuf.lo] Error 1
make[2]: Leaving directory `/usr/src/rsyslog/runtime'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/usr/src/rsyslog'
make: *** [all] Error 2


David Lang