[rsyslog] memory alloc problems - any advise?

[Rainer Gerhards]

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, June 19, 2009 10:30 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> 
> On Fri, 19 Jun 2009, Rainer Gerhards wrote:
> 
> > Hi all,
> >
> > during testing, I found out that some strange things happen 
> when a queue is
> > set to large size. If I set e.g. the main message queue to 
> 500,000 entries
> > and then shuffle messages very quickly at rsyslogd, it 
> consumes considerable
> > memory. This is not surprising, as it is buffering not yet-processed
> > messages.
> >
> > What is surprising is that the memory is not given back to 
> the OS when the
> > data has been processed. Well... I know that some data 
> blocks may still be
> > held and thus prevent giving memory back. But that should 
> be solved when a
> > few new messages come in. Still, nothing changes in that 
> scenario. What
> > puzzles me even more is that when I start a new burst of 
> messages (I send
> > around one million each), it looks like almost only new 
> memory is obtained
> > from the OS for them. At least here I had expected that the 
> allocator would
> > re-use what was already claimed.
> >
> > The interesting things is that when the queue size is small 
> (50,000 message
> > for example), this behavior does not occur. Then, memory 
> allocation and
> > deallocation works as expected.
> 
> hmm, this doesn't match what I was seeing with the 3.x 
> series. I had a 
> machine with 32G of ram and was setting the max queue size to 
> 1M entries.

I checked with v3, and it had the same behavior (a little less strong, but
still).

> 
> I don't remember if it was freeing the memory as the queue 
> emptied, but I 
> know that I could hit it repeatedly with large peaks and it 
> would not use 
> additional memory.
> 
> > I have also run the valgrind memory debugger on the app, 
> and it does not
> > report any leaked memory.
> 
> remember that it will only consider it a leak if you no 
> longer have _any_ 
> refrence to the memory. if you have some pointer to it stashed away 
> somewhere that you won't use it, it won't consider it a leak.

Valgrind can print out those allocations. In short, there were nothing
unexpected (the usual 81 when rsyslogd terminates, and they pose no problem).
As you can see in my later posts, the issue seems actually to be rooted in
glibc.

> 
> early next week I should have some new boxes in that I can do 
> high-speed 
> tests on again (64G ram with 10G network)

That's good news. I did a lot of optimizations this week, and think I can
release a new version with them mid next week. I hope you will see even
further improvement. In the v5 engine, I think, I have discovered a memory
leak, but I didn't investigate today. Maybe just the same issue I've later
found wiht the malloc calls...

Rainer

> 
> David Lang
> 
> > If someone has an explanation of what causes this behavior, 
> I would really
> > love to hear about it...
> >
> > Thanks,
> > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>