[rsyslog] How to test rsyslog performance

[Rainer Gerhards]

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Monday, June 29, 2009 8:04 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] How to test rsyslog performance
> 
> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
> 
> > Hello David,
> > Thanks a bunch ..
> > I will try this, I was also thinking in similar lines, in 
> the sense that I
> > will log 1 line of message, set the file size to be huge, 
> and then count the
> > number of lines on the file.
> > I just read the numbers on Rainer's blog, and I think I 
> will need the
> > tcprelay approach as well, there is no way I can generate 
> so much load with
> > a shell script.
> > Thanks for the tip.
> 
> with an earlier version of the 4.x series I was seeing 
> numbers like the 
> following
> 
> with 256 byte messages
> 
> recieving via UDP and storing in the queue, >300K/sec (wire 
> speed, gig-E)
> 
> recieving via UDP and writing to disk, ~78K/sec
> 
> recieving via UDP, writing to disk and sending to another 
> machine, ~30k/sec
> 
> since then there has been a lot of work to optimize the queue 
> functions 
> that appeared to be the bottleneck in my testing,

In the mean time, I have identified at least one more bottleneck, and this
was alloc/free. These come with implicit synchroniation primitives (to
protect the heap) and it turned out that they cause quite some performance
loss as soon as we have enough threads running concurrently. I have removed
much of these alloc calls during the past two weeks, and the results - at
least in my non-high-end lab - are quite pleasant. This work is still not
100% finished, but I think I have reched the mark where further reduction of
alloc does not case any notable performance increase. I'll still continue to
finish up some things (so that the code base is consistent), but I don't
expect much performance increase from that. 

All in all, the "new v4" engine should perform around twice as fast as the
"old v4" engine (depending a bit on configuration). The v5 engine offers more
advanced optimizations and will receive even more of them. At this time, the
v5 engine in my lab (with TCP!) is at least twice as fast as the "new v4"
engine, probably a bit less with UDP.

There is still ample of tuning potential. Two things that immediately come to
my mind is that the inputs still use select() [originally used for
portability reasons] and the inputs themselves are single-threaded. For v5 I
can also envision a new malloc subsytem, that will do 1/1024th malloc per
message (right now, we do around 5 mallocs per message, tomorrow evening
probably only 3).

If someone is looking into performance, one should definitely look into the
v5 engine. This version is experimental, but if nobody tries it out, it can't
improve (everything I could do in my lab I have done, but I know from past
experience that the real world and high-end hardware is a different story).
But even the test with the threading debugger went quite well. I'll release a
new v5 shortly, and that one is definitely worth a try (if you need it
urgent, just let me know).

> and so I am very 
> interested in seeing the new results (I expect to be testing 
> shortly, but 
> will be out much of this week, one of my 'hobbies' is doing 
> professional 
> fireworks, and so I will be busy preparing for the July 4th shows)

Cool - wish I were there ;)

Rainer 

> David Lang
> 
> 
> > Regadrs,
> > Sayan
> >
> >
> > On Mon, Jun 29, 2009 at 12:46 PM, <david at lang.hm> wrote:
> >
> >> On Mon, 29 Jun 2009, Sayan Chowdhury wrote:
> >>
> >>> Hi,
> >>> I saw the announcement of the 4.2.0 stable version of 
> rsyslog. I was
> >>> wondering if there is a way I can test the performance of 
> rsyslog with my
> >>> own rsyslog.conf.
> >>> The way I am trying to use rsyslog is on a local system, which has
> >> following
> >>> rules
> >>>
> >>> logging different facilities to different files with 
> different formats.
> >>> forwarding certain facilities to an external server.
> >>> receive messages from one(and only one) other machine over udp.
> >>>
> >>> Are there any tool/scripts available which I can use to 
> perform the tests
> >>> myself?
> >>> Basically what I am thinking of is running a script which 
> writes is bunch
> >> of
> >>> messages to syslog, but then how do I make sure all those 
> messages were
> >>> actually logged and not lost? It may be very painful to 
> look through the
> >>> entire log file 1 message at a time :)
> >>> Is there a stat/counter based api which shows me how many 
> messages were
> >>> logged?
> >>
> >> what I do is to send a known quantity of messages and 
> include a identifier
> >> string in the message ('rsyslog-test-message' for 
> example), and then grep
> >> the resulting files to make sure that the right number of 
> messages arrive.
> >>
> >> to send messages to rsyslog I have used tcpreplay to 
> replay UDP messages.
> >> I first setup a system to send the messages via UDP, start 
> tcpdump, send
> >> the messages to the local box (which then sends them via 
> UDP to a remote
> >> server), then I stop tcpdump and can use the resulting 
> capture file with
> >> tcpreplay to replay the messages at different speeds.
> >>
> >> David Lang
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>