[rsyslog] optimization questions

[david at lang.hm]

I've got a couple projects I'm working on that rsyslog is looking like a 
good answer to. I'm going to send questions about them in seperate 
messages.

the first project is a traditional syslog server. I've been testing 
various syslog implementations to find out what sort of performance they 
can sustain.

I've gotten the (almost) standard sysklogd to sustail almost 30,000 
messages/sec (udp) before it starts loosing significant numbers of logs

rsyslog is able to handle bursts of around 150,000 messages/sec, but it 
doesn't seem to write them out very fast, and over time seems to be 
limited to ~22,000 messages/sec

the hardware I am running this test on is insanely overpowered (four 
dual-core opterons, 16G of ram, battery-backed cache on a raid card with 
high-speed SAS drives). in production I will be working with slower 
hardware, but I'm trying to find the limits of the software before I start 
introducing lower hardware limits.

what can I look at doing to speed up the writing of data from the queues? 
does rsyslog write the messages one at a time, or does it have an option 
for writing them in batches (pulling a bunch off of the queue and sending 
them all at once)?

some sort of batch mode would be critical for writes to a database where 
you can frequently get 1000x speedups if you do the inserts as a single 
transaction as opposed to individual transactions.

David Lang