[rsyslog] memory alloc problems - any advise?

[david at lang.hm]

On Fri, 19 Jun 2009, Rainer Gerhards wrote:

> Hi all,
>
> during testing, I found out that some strange things happen when a queue is
> set to large size. If I set e.g. the main message queue to 500,000 entries
> and then shuffle messages very quickly at rsyslogd, it consumes considerable
> memory. This is not surprising, as it is buffering not yet-processed
> messages.
>
> What is surprising is that the memory is not given back to the OS when the
> data has been processed. Well... I know that some data blocks may still be
> held and thus prevent giving memory back. But that should be solved when a
> few new messages come in. Still, nothing changes in that scenario. What
> puzzles me even more is that when I start a new burst of messages (I send
> around one million each), it looks like almost only new memory is obtained
> from the OS for them. At least here I had expected that the allocator would
> re-use what was already claimed.
>
> The interesting things is that when the queue size is small (50,000 message
> for example), this behavior does not occur. Then, memory allocation and
> deallocation works as expected.

hmm, this doesn't match what I was seeing with the 3.x series. I had a 
machine with 32G of ram and was setting the max queue size to 1M entries.

I don't remember if it was freeing the memory as the queue emptied, but I 
know that I could hit it repeatedly with large peaks and it would not use 
additional memory.

> I have also run the valgrind memory debugger on the app, and it does not
> report any leaked memory.

remember that it will only consider it a leak if you no longer have _any_ 
refrence to the memory. if you have some pointer to it stashed away 
somewhere that you won't use it, it won't consider it a leak.

early next week I should have some new boxes in that I can do high-speed 
tests on again (64G ram with 10G network)

David Lang

> If someone has an explanation of what causes this behavior, I would really
> love to hear about it...
>
> Thanks,
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>