[rsyslog] rsyslog - what's next?

[Kenneth Marshall]

On Mon, Jul 13, 2009 at 10:09:32AM -0700, david at lang.hm wrote:
> On Mon, 13 Jul 2009, Rainer Gerhards wrote:
> 
> > Hi all,
> >
> > I think we made some  really good progress with rsyslog's capabilities
> > (feature set & performance) in the past couple of month. I also think I have
> > reached a milestone. As such, I've taken a bit time off to think about where
> > to head to next. And as I'd like to have this available as a handy reference,
> > I've blogged about.
> >
> > So if you are interested in what I intend to focus on the next time (maybe
> > two to three month), please have a look at my blog post:
> >
> > http://blog.gerhards.net/2009/07/rsyslog-where-are-we.html
> >
> > Of course, urgent things will have priority over the general goal, but I am
> > pretty sure the planned effort will have a very positive effect on rsyslog's
> > capabilities and qualities even in the medium term.
> >
> > As always, comments and questions are welcome.
> 
> I don't think that you need to do much for UDP recieve performance. as 
> long as it doesn't need to do DNS lookups it can recieve and insert into a 
> memory queue at full gig-E speeds. the only think you may want to do here 
> is to extend the DNS cache so that instead of only caching the last think 
> you looked up, you cache everything until a restart or HUP (ideally the 
> HUP should be a configurable option)
> 
> the machines I had on order with the 10G interfaces were misconfigured by 
> the vendor, so I don't have the 10G cards yet :-( when I get them I'll do 
> more testing and see how much faster I can push rsyslog.
> 
> David Lang

I agree with David. DNS lookups can really sap both bandwidth and CPU.
I am not sure that caching everything since a restart is a good idea
though. Even if you could do a lookup, cache the TTL and IP and then
relookup after the TTL that would be great. Even having it re-query
the DNS in a worker thread to sweep through the cached values periodically
would really help performance.

Regards,
Ken