[rsyslog] How to test rsyslog performance

[david at lang.hm]

On Mon, 29 Jun 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
>>
>>> Hello David,
>>> Thanks a bunch ..
>>> I will try this, I was also thinking in similar lines, in
>> the sense that I
>>> will log 1 line of message, set the file size to be huge,
>> and then count the
>>> number of lines on the file.
>>> I just read the numbers on Rainer's blog, and I think I
>> will need the
>>> tcprelay approach as well, there is no way I can generate
>> so much load with
>>> a shell script.
>>> Thanks for the tip.
>>
>> with an earlier version of the 4.x series I was seeing
>> numbers like the
>> following
>>
>> with 256 byte messages
>>
>> recieving via UDP and storing in the queue, >300K/sec (wire
>> speed, gig-E)
>>
>> recieving via UDP and writing to disk, ~78K/sec
>>
>> recieving via UDP, writing to disk and sending to another
>> machine, ~30k/sec
>>
>> since then there has been a lot of work to optimize the queue
>> functions
>> that appeared to be the bottleneck in my testing,
>
> In the mean time, I have identified at least one more bottleneck, and this
> was alloc/free. These come with implicit synchroniation primitives (to
> protect the heap) and it turned out that they cause quite some performance
> loss as soon as we have enough threads running concurrently. I have removed
> much of these alloc calls during the past two weeks, and the results - at
> least in my non-high-end lab - are quite pleasant. This work is still not
> 100% finished, but I think I have reched the mark where further reduction of
> alloc does not case any notable performance increase. I'll still continue to
> finish up some things (so that the code base is consistent), but I don't
> expect much performance increase from that.
>
> All in all, the "new v4" engine should perform around twice as fast as the
> "old v4" engine (depending a bit on configuration). The v5 engine offers more
> advanced optimizations and will receive even more of them. At this time, the
> v5 engine in my lab (with TCP!) is at least twice as fast as the "new v4"
> engine, probably a bit less with UDP.
>
> There is still ample of tuning potential. Two things that immediately come to
> my mind is that the inputs still use select() [originally used for
> portability reasons] and the inputs themselves are single-threaded. For v5 I
> can also envision a new malloc subsytem, that will do 1/1024th malloc per
> message (right now, we do around 5 mallocs per message, tomorrow evening
> probably only 3).
>
> If someone is looking into performance, one should definitely look into the
> v5 engine. This version is experimental, but if nobody tries it out, it can't
> improve (everything I could do in my lab I have done, but I know from past
> experience that the real world and high-end hardware is a different story).
> But even the test with the threading debugger went quite well. I'll release a
> new v5 shortly, and that one is definitely worth a try (if you need it
> urgent, just let me know).

I tried to compile the v5 branch over the weekend, but got a compile 
error, so I just moved to the 4.2.0 release. on the ultra-reliable testbed 
it's getting ~8500/sec.

David Lang

>> and so I am very
>> interested in seeing the new results (I expect to be testing
>> shortly, but
>> will be out much of this week, one of my 'hobbies' is doing
>> professional
>> fireworks, and so I will be busy preparing for the July 4th shows)
>
> Cool - wish I were there ;)
>
> Rainer
>
>> David Lang
>>
>>
>>> Regadrs,
>>> Sayan
>>>
>>>
>>> On Mon, Jun 29, 2009 at 12:46 PM, <david at lang.hm> wrote:
>>>
>>>> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
>>>>
>>>>> Hi,
>>>>> I saw the announcement of the 4.2.0 stable version of
>> rsyslog. I was
>>>>> wondering if there is a way I can test the performance of
>> rsyslog with my
>>>>> own rsyslog.conf.
>>>>> The way I am trying to use rsyslog is on a local system, which has
>>>> following
>>>>> rules
>>>>>
>>>>> logging different facilities to different files with
>> different formats.
>>>>> forwarding certain facilities to an external server.
>>>>> receive messages from one(and only one) other machine over udp.
>>>>>
>>>>> Are there any tool/scripts available which I can use to
>> perform the tests
>>>>> myself?
>>>>> Basically what I am thinking of is running a script which
>> writes is bunch
>>>> of
>>>>> messages to syslog, but then how do I make sure all those
>> messages were
>>>>> actually logged and not lost? It may be very painful to
>> look through the
>>>>> entire log file 1 message at a time :)
>>>>> Is there a stat/counter based api which shows me how many
>> messages were
>>>>> logged?
>>>>
>>>> what I do is to send a known quantity of messages and
>> include a identifier
>>>> string in the message ('rsyslog-test-message' for
>> example), and then grep
>>>> the resulting files to make sure that the right number of
>> messages arrive.
>>>>
>>>> to send messages to rsyslog I have used tcpreplay to
>> replay UDP messages.
>>>> I first setup a system to send the messages via UDP, start
>> tcpdump, send
>>>> the messages to the local box (which then sends them via
>> UDP to a remote
>>>> server), then I stop tcpdump and can use the resulting
>> capture file with
>>>> tcpreplay to replay the messages at different speeds.
>>>>
>>>> David Lang
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>