[rsyslog] optimization questions
david at lang.hm
david at lang.hm
Mon Sep 15 09:01:40 CEST 2008
On Mon, 15 Sep 2008, Rainer Gerhards wrote:
> Hi David,
> thanks for your message and interest. This is very welcome as I would
> really like to get hold of more insight into rsyslog's performance in
> real-world and extreme scenarios. So far, I have unfortunately been
> unable to do that, because I had no funding for the required hardware
> and time to conduct the testing. I would *really* appreciate if you
> could help with that and I would be very willing to tweak the code for
> optimum performance (actually, I am always very concerned about high
> performance and I was a bit sad about the fact that I could not ensure
> Having said that, I'd first of all would like to have a look at your
> rsyslog.conf, so that we are on common ground. Also, I think, this can
> become quite lengthy and also of interest for others. May it be useful
> to discuss this on the web forum, so that we can easier keep track of
> things. If you think this is useful, I recommend this one here:
done, I created a topic in the developers corner
> some more comments inline:
> On Mon, 2008-09-15 at 07:23 +0200, david at lang.hm wrote:
>> I've got a couple projects I'm working on that rsyslog is looking like
>> good answer to. I'm going to send questions about them in seperate
>> the first project is a traditional syslog server. I've been testing
>> various syslog implementations to find out what sort of performance
>> can sustain.
>> I've gotten the (almost) standard sysklogd to sustail almost 30,000
>> messages/sec (udp) before it starts loosing significant numbers of
>> rsyslog is able to handle bursts of around 150,000 messages/sec, but
>> doesn't seem to write them out very fast, and over time seems to be
>> limited to ~22,000 messages/sec
>> the hardware I am running this test on is insanely overpowered (four
>> dual-core opterons, 16G of ram, battery-backed cache on a raid card
>> high-speed SAS drives). in production I will be working with slower
>> hardware, but I'm trying to find the limits of the software before I
>> introducing lower hardware limits.
>> what can I look at doing to speed up the writing of data from the
>> does rsyslog write the messages one at a time, or does it have an
>> for writing them in batches (pulling a bunch off of the queue and
>> them all at once)?
> The design is that each individual message is pushed to the output. The
> output than writes the message. There has been discussion about a lazy
> write, but nothing in this regard has yet been implemented.
> Note that this is a good time to request new things. I am finished with
> the last major thing (TLS) support and in front of the next one
> (enhanced scripting), but scripting, I think, could be moved to a later
> time in favour of something more important (like performance ;)).
>> some sort of batch mode would be critical for writes to a database
>> you can frequently get 1000x speedups if you do the inserts as a
>> transaction as opposed to individual transactions.
> I am not (any longer) so much a database guy. Can I really do a batch
> insert with them? Can you point me to an API?
> Looking forward to your reply!
> rsyslog mailing list
More information about the rsyslog