[rsyslog] rsyslog - what's next?

[Rainer Gerhards]

One important point to stress is that rsyslog does NOT yet have a real DNS
cache! Let me quote my mail from July, 9th[1]:

==
And another point to stress: rsyslogd does *not* yet have a high-performance
cache. All it does is cache the last host (and only the last host). This
works exceptionally well if a large bunch of messages arrive from the same
host, but "cache" performance can easily be thrashed with multiple senders.
All in all, in practice, it works reasonably well, as on a busy system at
least a couple of messages are usually from the same sender. I plan to add a
real cache some time later, personally I would hope to see it this summer.
==

Adding a the cache is not really a focus activity - I think it could be done
in a couple of days. I also personally do not see an issue with obying TTL -
or discarding entries every five minutes or so. In my POV, it doesn't really
matter if they are quickly discarded. If there is low traffic, it doesn't
make a difference at all. If we handle hundereds of thousands message per
second, one lockup every few billion of messages doesn't really matter
either. The compare operation is relatively fast and will not add
significantly to the runtime (at least I'd guess that).

In regard to imudp, I think there is a couple of more optimizations possible.
I'd expect that the biggest hit (after DNS lookup) is switching from select
to epoll. Re-using property copying is another one (reduces expensive memory
writes). A batching interface seems also possible. Enhanced pipelining is
another possibility. I also think that the current solution will gain very
different results based on the communication patterns. So rather than
(educated) guessing, I would prefer to have some good instrumentation to get
a grip on all this variants.

Also, imudp is just a sample, there are more areas that could benefit from a
focussed analytic view. Plus that performance tools can be used to
stress-testing and thus finding otherwise hard to find bugs. Thus I think it
is worth spending some time on an improved toolset (and will pay off in the
long term).

Rainer

[1] http://lists.adiscon.net/pipermail/rsyslog/2009-July/002432.html
 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of 
> Kenneth Marshall
> Sent: Monday, July 13, 2009 9:38 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog - what's next?
> 
> On Mon, Jul 13, 2009 at 10:09:32AM -0700, david at lang.hm wrote:
> > On Mon, 13 Jul 2009, Rainer Gerhards wrote:
> > 
> > > Hi all,
> > >
> > > I think we made some  really good progress with rsyslog's 
> capabilities
> > > (feature set & performance) in the past couple of month. 
> I also think I have
> > > reached a milestone. As such, I've taken a bit time off 
> to think about where
> > > to head to next. And as I'd like to have this available 
> as a handy reference,
> > > I've blogged about.
> > >
> > > So if you are interested in what I intend to focus on the 
> next time (maybe
> > > two to three month), please have a look at my blog post:
> > >
> > > http://blog.gerhards.net/2009/07/rsyslog-where-are-we.html
> > >
> > > Of course, urgent things will have priority over the 
> general goal, but I am
> > > pretty sure the planned effort will have a very positive 
> effect on rsyslog's
> > > capabilities and qualities even in the medium term.
> > >
> > > As always, comments and questions are welcome.
> > 
> > I don't think that you need to do much for UDP recieve 
> performance. as 
> > long as it doesn't need to do DNS lookups it can recieve 
> and insert into a 
> > memory queue at full gig-E speeds. the only think you may 
> want to do here 
> > is to extend the DNS cache so that instead of only caching 
> the last think 
> > you looked up, you cache everything until a restart or HUP 
> (ideally the 
> > HUP should be a configurable option)
> > 
> > the machines I had on order with the 10G interfaces were 
> misconfigured by 
> > the vendor, so I don't have the 10G cards yet :-( when I 
> get them I'll do 
> > more testing and see how much faster I can push rsyslog.
> > 
> > David Lang
> 
> I agree with David. DNS lookups can really sap both bandwidth and CPU.
> I am not sure that caching everything since a restart is a good idea
> though. Even if you could do a lookup, cache the TTL and IP and then
> relookup after the TTL that would be great. Even having it re-query
> the DNS in a worker thread to sweep through the cached values 
> periodically
> would really help performance.
> 
> Regards,
> Ken
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>