[rsyslog] performance optimization (milestone) done

[Rainer Gerhards]

Hi all,

I just wanted to let you know that I have reached a milestone with my
optimizations. My goal of reducing malloc/free calls has now been reached. I
need to revisit priorities, but I will probably move away from performance
optimization for a while now (maybe some smaller things that I stumble
across, but not a real effort).

As I have written recently, there is still ample potential for optimization.
I will "just" probably not explore it at this point in time.

However, I would like to ask one question that could lead to a new effort (in
v5). Looking at the queue engine, I see that managing the queue, especially
in ultra-reliable mode, requires a lot of effort, what also means a lot of
CPU time.

If we relax reliability conditions, we could definitely save some time here
(probably a real lot!). So my question is what would be the least reliability
that you need for a non-audit, high-performance scenario. I can envision that
it is sufficient to have this:

All messages handed over to the queue will be processed if no system failure
occurs and if there is space available inside the queue. All messages are
queued in main memory. If messages are tried to be enqueued when the queue is
full, these messages will be lost. Message loss victims will be strictly
based on order of enqueue (queue full condition) and not severity or any
other property (evaluating that would take time, again). Also, it is
acceptable to lose unprocessed messages during rsyslogd shutdown, if they
cannot be processed within the (configurable) shutdown intervals.

Would this make sense for sufficiently important use cases? If so, I could
see that (over time!) I implement a very fast queue driver based on that
relaxed criteria. With it, I think I could also create a lock-free version
(but not immediately), which would definitely be a big performance gain.

Feedback is appreciated.

The optimized builds are currently in the git master and v5-devel branches,
new releases are due soon (but will see that I do at least a bit more doc
work before releasing them).

Rainer