[rsyslog] rsyslog - what's next?

[Rainer Gerhards]

I am digesting this thread (just got up ;)), but let me spell out something
explicitely: the traditional perception of what is fast and what is slow is
no longer valid if you aim at a rate of several hunderd-thousand messages per
second. For example, it then makes a difference if you (memory) write a
single 32 byte buffer per record or not. It may be in the region of 100 to
500 records per second, but these differences sum up. So trying to avoid such
writes, even a the costs of (currently much cheaper) reads is something to
aim at. I'll see that I get together a more elaborate description inside my
blog, probably it is useful. I learned a lot of lessons the past nine month
;)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, July 14, 2009 6:55 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog - what's next?
> 
> On Mon, 13 Jul 2009, RB wrote:
> 
> > On Mon, Jul 13, 2009 at 15:58, <david at lang.hm> wrote:
> >> at the time I did quite a number of strace dumps to show 
> how much time was
> >> being eaten up in the system calls.
> >
> > Hopefully some of the tooling Rainer is considering will 
> better enable
> > performance monitoring and we can test the effect of some of these
> > suggested changes.  I don't completely discount strace as a
> > performance testing tool, but it's inherently intrusive, making
> > Heisenberg quite happy.
> 
> absolutly, especially when you enable timing of the calls !!! (a 
> gettimeofday call before and after evey syscall)
> 
> but if the program hasn't been examined to try and reduce the 
> number of 
> syscalls it makes, strace is a good place to start.
> 
> once you get down to a minimum of nessasary syscalls, strace 
> stops being 
> useful.
> 
> >> well, if you are really wanting audit-grade logging, will 
> you use anything
> >> other than the IP address (or what is already in the 
> message)? any lookup
> >> that you do is a potential for corruption.
> >>
> >> I'm fine with the default being to do a lookup for each 
> item. I just want
> >> a way to avoid it, and if I can do that with a cache 
> instead of having to
> >> disable DNS, I will do so.
> >>
> >> it's very common for servers to need to disable DNS 
> lookups for their
> >> logs. do a quick search for apache dns performance and you 
> will find that
> >> it's a very common problem people have there if they 
> enable lookups on the
> >> sender's IP address
> >
> > Indeed; I've generally eschewed DNS for such critical pieces of
> > infrastructure, instead opting for resolution during analysis and
> > correlation with other logs.  That largely stems from 
> network security
> > work and the associated disdain for using DNS entries for 
> rules.  Even
> > so, I'm curious whether a properly tuned library-integrated 
> cache like
> > nscd (as opposed to a local caching resolver) would provide a
> > sufficient performance increase in the interim.
> 
> I'll try and get a chance to look into that
> 
> > I spent a couple of hours this evening digging through the related
> > code with the intent of starting work on delayed resolution, moving
> > the lookup to runtime/msg.c:getHOSTNAME.  It might not be the right
> > place, but would have the advantage of only resolving hosts you
> > explicitly request it for.  Unfortunately, the 
> AllowedSender pragma is
> > a sticking point, forcing per-packet resolution; otherwise, 
> it seemed
> > a pretty trivial conversion.  I don't know how much (if any)
> > performance increase that approach would grant, but it 
> would at least
> > move a blocking call out of that single thread.  I may 
> experiment with
> > adding a "NoAllowedSenderDNS" later, but it's beyond my time
> > flexibility right now.
> 
> interesting thought.
> 
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>