[rsyslog] rsyslog stats

Rainer Gerhards rgerhards at hq.adiscon.com
Tue Sep 7 23:22:52 CEST 2010

Finding the age of the oldest message is not necessarily cheap - think about out of order processing and delays. The one at queue top is not necessarily the oldest one...


----- Ursprüngliche Nachricht -----
Von: david at lang.hm <david at lang.hm>
Gesendet: Dienstag, 7. September 2010 22:58
An: rsyslog-users <rsyslog at lists.adiscon.com>
Betreff: Re: [rsyslog] rsyslog stats

On Tue, 7 Sep 2010, Joe Williams wrote:

> On Sep 7, 2010, at 11:11 AM, Aaron Wiebe wrote:
>> On Tue, Sep 7, 2010 at 2:07 PM,  <david at lang.hm> wrote:
>>> 1. stats that are cheap to gather, generate
>>>  examples  current length of queues, age of oldest message in each queue,
>>> number of messages processed (possibly even per output)
>>> 2. stats that are expensive to gather or generate
>>>  examples: average length of queues, average time to process messages
>> Expensive stats shouldn't be the responsibility of rsyslog though -
>> trending over time and the like should be the responsibility of a
>> third party product (ie, cacti/mrtg).  As long as the stats are easily
>> made available through snmp or other methods, the trending should be
>> done outside rsyslog.

> Agreed, rsyslog should just report the current stats and any 
> trending/analysis should be done externally. No different than running 
> iostat or other tools.

iostat and vmstat actually provide some of these expensive stats.

an example of an expensive stat would be the average age of messages in 
the queue.

finding the age of the oldest message is relativly cheap (one lookup to 
the oldest message in the queue, that lookup may delay something 
processing the message so it's only 'relativly' cheap)

but finding the average age of all messages in the queue will require 
looking up the age of every message in the queue, and probably freezing 
the queue while you do so to make sure that it's not changing under you.

> One other timing stat I would love to see is the amount of time in each 
> step of the pipeline (time to read the message from the socket, time 
> processing it through any rules, time spent in any queues, time spent 
> writing to disk, etc). Haproxy has a good example of this in it's 
> logging (8.2.3. HTTP log format - 
> http://haproxy.1wt.eu/download/1.4/doc/configuration.txt).

message processing in rsyslog is done at such a high rate that you can get 
a noticable performance improvement by doing one gettimeofday call for 
every 10 messages instead of one call per message. looking up the time at 
different stages within rsyslog definantly qualifies as an expensive stat 
to gather.

cheap things to gather are things that can be gathered by modifying a 
per-thread variable.

relativly cheap things to gather are things that require locking to gather 
(either to update a central variable, or to prevent other threads from 
modifying what you are reading)

expensive things to gather include things that require syscalls 
(for example, gettimeofday()), or that require looking at a lot of data 
(average age of all messages in a queue)

David Lang
rsyslog mailing list

More information about the rsyslog mailing list