[rsyslog] rsyslog - what's next?

[david at lang.hm]

On Mon, 13 Jul 2009, RB wrote:

> On Mon, Jul 13, 2009 at 11:09, <david at lang.hm> wrote:
>> is to extend the DNS cache so that instead of only caching the last think
>> you looked up, you cache everything until a restart or HUP (ideally the
>> HUP should be a configurable option)
>
> I haven't looked at the DNS cache code, but unless you're also caching
> and handling the records' TTL, blindly caching DNS records within the
> app is a dark road to go down.  Some apps (namely browsers) do it
> anyway, but get away with it by setting their internal TTL
> significantly lower than that of a typical record.

for Internet access I completely agree with you, but for a log server you 
don't have IPs changing names very frequently. As a result it becomes 
practical to cache the lookup unconditionally until a restart/HUP. even in 
a 'highly dynamic' environment you are usually adding servers instead of 
changing the names of IP addresses.

note that caching the lookup at all can be disabled via a config option

> What kind of performance hit are you actually seeing with DNS lookups?

6 months ago it was a factor of 10x to 100x (it's probably more now since 
rsyslog has gotten faster)

the problem is that to do a name lookup requires a LOT of steps

first the name resolver library looks several places on the filesystem for 
various config files

then it reads /etc/hosts, parses it, and looks for the name/IP

then it makes a network connection to the nameserver, sends a message, 
waits for the reciever to parse the message and look it up in it's 
datastore, then send the message back to the requester who needs to parse 
the message

and if you network packet gets dropped due to congestion on the network, 
you wait 30 seconds and retry.


doing all of this for every UDP syslog packet that you recieve results in 
a _lot_ of system calls, and a lot of delays.

even if you have the name in the /etc/hosts file, the overhead of looking 
in the filesystem and parsing the file is significant.

without doing DNS lookups, rsyslog is able to recieve UDP logs at almost 
400,000 per second (effectivly Gig-E wire speed with 256 byte log 
messages), _very_ few DNS servers can handle requests at that speed. in 
fact, at that request rate, you use about half of a Gig-E connection just 
for the DNS requests (more if you request more data, like what the TTL 
would be, or don't give it the best name the first time and need to make 
multiple requests with different domains attached or something like that)


David Lang