[rsyslog] How to test rsyslog performance

[david at lang.hm]

On Mon, 29 Jun 2009, Sayan Chowdhury wrote:

> Hello David,
> Thanks a bunch ..
> I will try this, I was also thinking in similar lines, in the sense that I
> will log 1 line of message, set the file size to be huge, and then count the
> number of lines on the file.
> I just read the numbers on Rainer's blog, and I think I will need the
> tcprelay approach as well, there is no way I can generate so much load with
> a shell script.
> Thanks for the tip.

with an earlier version of the 4.x series I was seeing numbers like the 
following

with 256 byte messages

recieving via UDP and storing in the queue, >300K/sec (wire speed, gig-E)

recieving via UDP and writing to disk, ~78K/sec

recieving via UDP, writing to disk and sending to another machine, ~30k/sec

since then there has been a lot of work to optimize the queue functions 
that appeared to be the bottleneck in my testing, and so I am very 
interested in seeing the new results (I expect to be testing shortly, but 
will be out much of this week, one of my 'hobbies' is doing professional 
fireworks, and so I will be busy preparing for the July 4th shows)

David Lang


> Regadrs,
> Sayan
>
>
> On Mon, Jun 29, 2009 at 12:46 PM, <david at lang.hm> wrote:
>
>> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
>>
>>> Hi,
>>> I saw the announcement of the 4.2.0 stable version of rsyslog. I was
>>> wondering if there is a way I can test the performance of rsyslog with my
>>> own rsyslog.conf.
>>> The way I am trying to use rsyslog is on a local system, which has
>> following
>>> rules
>>>
>>> logging different facilities to different files with different formats.
>>> forwarding certain facilities to an external server.
>>> receive messages from one(and only one) other machine over udp.
>>>
>>> Are there any tool/scripts available which I can use to perform the tests
>>> myself?
>>> Basically what I am thinking of is running a script which writes is bunch
>> of
>>> messages to syslog, but then how do I make sure all those messages were
>>> actually logged and not lost? It may be very painful to look through the
>>> entire log file 1 message at a time :)
>>> Is there a stat/counter based api which shows me how many messages were
>>> logged?
>>
>> what I do is to send a known quantity of messages and include a identifier
>> string in the message ('rsyslog-test-message' for example), and then grep
>> the resulting files to make sure that the right number of messages arrive.
>>
>> to send messages to rsyslog I have used tcpreplay to replay UDP messages.
>> I first setup a system to send the messages via UDP, start tcpdump, send
>> the messages to the local box (which then sends them via UDP to a remote
>> server), then I stop tcpdump and can use the resulting capture file with
>> tcpreplay to replay the messages at different speeds.
>>
>> David Lang
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>