[rsyslog] rsyslog - what's next?

[david at lang.hm]

On Mon, 13 Jul 2009, RB wrote:

> On Mon, Jul 13, 2009 at 14:38, <david at lang.hm> wrote:
>> HUP does not interrupt services.
>
> I'm sorry; last October you noted that for certain configurations
> SIGHUP will drop memory-queued messages.  In addition to the other
> notes in the thread (tearing down TCP connections, etc.), it sounds an
> awful lot like a service interruption.  Has that since changed for 3.x
> or 4.2?

yes, for 4.2 there is the option of 'HUPisRestart no', which makes HUP not 
do a complete teardown, halt, and restart, but instead just closes and 
re-opens files and connections so that no long messages are lost in a 
restart.

this matches the traditional use of HUP to get syslogd to release the 
files it's writing to so that they can be rotated away.

it doesn't re-read the config file due to the fact that the rsyslog config 
file is so complex and can significantly alter the software by loading 
modules.

>> the system calls to access it (and to first lookup if it should be
>> accessed at all) take enough time to be significant at these speeds.
>
> May I gently prod you to substantiate this statement?  I don't doubt
> that it makes calls to external libraries, and that those calls are
> likely more expensive than internal resolution, but "significant" is
> not significant without numbers to back it up.  Even a simple speed
> comparison for a few million packets between 'no resolution' and 'in
> /etc/hosts with a cold cache' would be useful.

I would have to do a new set of tests to get you concrete numbers, but 
back in roughly the october timeframe last year I was doing extensive 
tests on this and with hosts files vs. no resolution I was seeing 4x or 
more differences (with a tiny, 5-line hosts file to parse)

at the time I think I discussed it on a long performance thread on the 
website.

at the time I did quite a number of strace dumps to show how much time was 
being eaten up in the system calls. RG has done a LOT of cleanup since 
then (to the point that the failsafe audit mode is in the ballpark of 
being as fast as the in-memory mode was back then)

>> you want the ability to cache the name lookup no matter what method is
>> used. only DNS provides a TTL, hosts files, NIS, LDAP, etc do not include
>> a TTL.
>
> I have no problem expanding the scope of discussion to resolution
> methods beyond DNS (which was the original premise), but if a name
> service does not provide an explicit TTL, it must be assumed as an
> implicit '0', which blind caching will break.  That said, each of
> those methods still provides some [internal] method of caching and
> invalidation that, to be audit-grade correct, rsyslog will either have
> to replicate or trust.  I still can't see a problem with having
> something to the effect of a "$BreakNSCache" configuration option, but
> by default a syslog daemon should play as strictly by the rules as
> possible.

well, if you are really wanting audit-grade logging, will you use anything 
other than the IP address (or what is already in the message)? any lookup 
that you do is a potential for corruption.

I'm fine with the default being to do a lookup for each item. I just want 
a way to avoid it, and if I can do that with a cache instead of having to 
disable DNS, I will do so.

it's very common for servers to need to disable DNS lookups for their 
logs. do a quick search for apache dns performance and you will find that 
it's a very common problem people have there if they enable lookups on the 
sender's IP address.

> Then again, I'm RB, not RG.  ;)

more people speaking up is good. I have strong opinions, and real problems 
to address, but RG needs to hear from people other than me.

David Lang