[rsyslog] performance optimization (milestone) done

[david at lang.hm]

On Wed, 1 Jul 2009, Rainer Gerhards wrote:

> Hi all,
>
> I just wanted to let you know that I have reached a milestone with my
> optimizations. My goal of reducing malloc/free calls has now been reached. I
> need to revisit priorities, but I will probably move away from performance
> optimization for a while now (maybe some smaller things that I stumble
> across, but not a real effort).
>
> As I have written recently, there is still ample potential for optimization.
> I will "just" probably not explore it at this point in time.
>
> However, I would like to ask one question that could lead to a new effort (in
> v5). Looking at the queue engine, I see that managing the queue, especially
> in ultra-reliable mode, requires a lot of effort, what also means a lot of
> CPU time.
>
> If we relax reliability conditions, we could definitely save some time here
> (probably a real lot!). So my question is what would be the least reliability
> that you need for a non-audit, high-performance scenario. I can envision that
> it is sufficient to have this:
>
> All messages handed over to the queue will be processed if no system failure
> occurs and if there is space available inside the queue. All messages are
> queued in main memory. If messages are tried to be enqueued when the queue is
> full, these messages will be lost. Message loss victims will be strictly
> based on order of enqueue (queue full condition) and not severity or any
> other property (evaluating that would take time, again). Also, it is
> acceptable to lose unprocessed messages during rsyslogd shutdown, if they
> cannot be processed within the (configurable) shutdown intervals.
>
> Would this make sense for sufficiently important use cases? If so, I could
> see that (over time!) I implement a very fast queue driver based on that
> relaxed criteria. With it, I think I could also create a lock-free version
> (but not immediately), which would definitely be a big performance gain.

how is this different than what happens today?

David Lang

> Feedback is appreciated.
>
> The optimized builds are currently in the git master and v5-devel branches,
> new releases are due soon (but will see that I do at least a bit more doc
> work before releasing them).
>
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>