[rsyslog] DNS performance - was: RE: rsyslog - what's next?

[Rainer Gerhards]

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of RB
> Sent: Tuesday, July 14, 2009 5:13 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog - what's next?
> 
> On Mon, Jul 13, 2009 at 15:58, <david at lang.hm> wrote:
> > at the time I did quite a number of strace dumps to show how much time
was
> > being eaten up in the system calls.
> 
> Hopefully some of the tooling Rainer is considering will better enable
> performance monitoring and we can test the effect of some of these
> suggested changes.  I don't completely discount strace as a
> performance testing tool, but it's inherently intrusive, making
> Heisenberg quite happy.

That's exactly part of the "new tools" idea, but obviously this
instrumentation will also affect the overall picture. But I think I can do
the base counters with a few atomic increments, so the effect should be not
that bad. The real work shall then be done in the front-end.

> 
> > well, if you are really wanting audit-grade logging, will you use
anything
> > other than the IP address (or what is already in the message)? any lookup
> > that you do is a potential for corruption.
> >
> > I'm fine with the default being to do a lookup for each item. I just want
> > a way to avoid it, and if I can do that with a cache instead of having to
> > disable DNS, I will do so.
> >
> > it's very common for servers to need to disable DNS lookups for their
> > logs. do a quick search for apache dns performance and you will find that
> > it's a very common problem people have there if they enable lookups on
the
> > sender's IP address
> 
> Indeed; I've generally eschewed DNS for such critical pieces of
> infrastructure, instead opting for resolution during analysis and
> correlation with other logs.  That largely stems from network security
> work and the associated disdain for using DNS entries for rules.  Even
> so, I'm curious whether a properly tuned library-integrated cache like
> nscd (as opposed to a local caching resolver) would provide a
> sufficient performance increase in the interim.
> 
> I spent a couple of hours this evening digging through the related
> code with the intent of starting work on delayed resolution, moving
> the lookup to runtime/msg.c:getHOSTNAME.  It might not be the right
> place, but would have the advantage of only resolving hosts you
> explicitly request it for.  Unfortunately, the AllowedSender pragma is
> a sticking point, forcing per-packet resolution; otherwise, it seemed
> a pretty trivial conversion.  I don't know how much (if any)
> performance increase that approach would grant, but it would at least
> move a blocking call out of that single thread.  I may experiment with
> adding a "NoAllowedSenderDNS" later, but it's beyond my time
> flexibility right now.

That's indeed an interesting thought. The allowedSenders is not really that
much of a problem. First of all, not all inputs support them (one may argue
they should, but one may also argue that a firewall is a much better place to
block traffic...). Secondly, $AllowedSender is not used in most
configurations. Rsyslog can detect that the sender name is not needed and in
this case can actually defer the name lookup. Of course, that doesn't help in
situations where the lookup is required, but it helps in many configurations.
This is also the spirit of some of my recent performance enhancements: not
all work equally well under all circumstances. But I have tried to make only
those suffer that actually need the feature that bears the cost. I hope to
extend that idea to the queue, where an ultra-fast mode may automatically be
selected if the parameters indicate that is possible. But, again, some
tooling to test this and its affects would be quite useful.

Rainer