[rsyslog] performance optimization (milestone) done

[david at lang.hm]

On Wed, 1 Jul 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Wednesday, July 01, 2009 6:59 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] performance optimization (milestone) done
>>
>> On Wed, 1 Jul 2009, Rainer Gerhards wrote:
>>
>>> Hi all,
>>>
>>> I just wanted to let you know that I have reached a
>> milestone with my
>>> optimizations. My goal of reducing malloc/free calls has
>> now been reached. I
>>> need to revisit priorities, but I will probably move away
>> from performance
>>> optimization for a while now (maybe some smaller things
>> that I stumble
>>> across, but not a real effort).
>>>
>>> As I have written recently, there is still ample potential
>> for optimization.
>>> I will "just" probably not explore it at this point in time.
>>>
>>> However, I would like to ask one question that could lead
>> to a new effort (in
>>> v5). Looking at the queue engine, I see that managing the
>> queue, especially
>>> in ultra-reliable mode, requires a lot of effort, what also
>> means a lot of
>>> CPU time.
>>>
>>> If we relax reliability conditions, we could definitely
>> save some time here
>>> (probably a real lot!). So my question is what would be the
>> least reliability
>>> that you need for a non-audit, high-performance scenario. I
>> can envision that
>>> it is sufficient to have this:
>>>
>>> All messages handed over to the queue will be processed if
>> no system failure
>>> occurs and if there is space available inside the queue.
>> All messages are
>>> queued in main memory. If messages are tried to be enqueued
>> when the queue is
>>> full, these messages will be lost. Message loss victims
>> will be strictly
>>> based on order of enqueue (queue full condition) and not
>> severity or any
>>> other property (evaluating that would take time, again). Also, it is
>>> acceptable to lose unprocessed messages during rsyslogd
>> shutdown, if they
>>> cannot be processed within the (configurable) shutdown intervals.
>>>
>>> Would this make sense for sufficiently important use cases?
>> If so, I could
>>> see that (over time!) I implement a very fast queue driver
>> based on that
>>> relaxed criteria. With it, I think I could also create a
>> lock-free version
>>> (but not immediately), which would definitely be a big
>> performance gain.
>>
>> how is this different than what happens today?
>
> Puuuuhhh... In various ways, we have all this go to disk, low/high watermark,
> persistence, reliability, discard messages based on priority, slow down
> sender, rate limiting ... stuff. All nice and useful, but takes time...
>
> What I described above would be the sole capability, (almost) none of these
> bells and whistles (except those that are handled in the action engine, I do
> not know out of my head which these are, but a low subset of what I said).

what you described as the new, limited capabilities sounds like what I 
would expect from a syslog daemon.

David Lang