[rsyslog] performance optimization (milestone) done

[Rainer Gerhards]

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Wednesday, July 01, 2009 6:59 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] performance optimization (milestone) done
> 
> On Wed, 1 Jul 2009, Rainer Gerhards wrote:
> 
> > Hi all,
> >
> > I just wanted to let you know that I have reached a 
> milestone with my
> > optimizations. My goal of reducing malloc/free calls has 
> now been reached. I
> > need to revisit priorities, but I will probably move away 
> from performance
> > optimization for a while now (maybe some smaller things 
> that I stumble
> > across, but not a real effort).
> >
> > As I have written recently, there is still ample potential 
> for optimization.
> > I will "just" probably not explore it at this point in time.
> >
> > However, I would like to ask one question that could lead 
> to a new effort (in
> > v5). Looking at the queue engine, I see that managing the 
> queue, especially
> > in ultra-reliable mode, requires a lot of effort, what also 
> means a lot of
> > CPU time.
> >
> > If we relax reliability conditions, we could definitely 
> save some time here
> > (probably a real lot!). So my question is what would be the 
> least reliability
> > that you need for a non-audit, high-performance scenario. I 
> can envision that
> > it is sufficient to have this:
> >
> > All messages handed over to the queue will be processed if 
> no system failure
> > occurs and if there is space available inside the queue. 
> All messages are
> > queued in main memory. If messages are tried to be enqueued 
> when the queue is
> > full, these messages will be lost. Message loss victims 
> will be strictly
> > based on order of enqueue (queue full condition) and not 
> severity or any
> > other property (evaluating that would take time, again). Also, it is
> > acceptable to lose unprocessed messages during rsyslogd 
> shutdown, if they
> > cannot be processed within the (configurable) shutdown intervals.
> >
> > Would this make sense for sufficiently important use cases? 
> If so, I could
> > see that (over time!) I implement a very fast queue driver 
> based on that
> > relaxed criteria. With it, I think I could also create a 
> lock-free version
> > (but not immediately), which would definitely be a big 
> performance gain.
> 
> how is this different than what happens today?

Puuuuhhh... In various ways, we have all this go to disk, low/high watermark,
persistence, reliability, discard messages based on priority, slow down
sender, rate limiting ... stuff. All nice and useful, but takes time...

What I described above would be the sole capability, (almost) none of these
bells and whistles (except those that are handled in the action engine, I do
not know out of my head which these are, but a low subset of what I said).

Rainer
> 
> David Lang
> 
> > Feedback is appreciated.
> >
> > The optimized builds are currently in the git master and 
> v5-devel branches,
> > new releases are due soon (but will see that I do at least 
> a bit more doc
> > work before releasing them).
> >
> > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>