[rsyslog] memory alloc problems - any advise?

[Rainer Gerhards]

Hi all,

during testing, I found out that some strange things happen when a queue is
set to large size. If I set e.g. the main message queue to 500,000 entries
and then shuffle messages very quickly at rsyslogd, it consumes considerable
memory. This is not surprising, as it is buffering not yet-processed
messages.

What is surprising is that the memory is not given back to the OS when the
data has been processed. Well... I know that some data blocks may still be
held and thus prevent giving memory back. But that should be solved when a
few new messages come in. Still, nothing changes in that scenario. What
puzzles me even more is that when I start a new burst of messages (I send
around one million each), it looks like almost only new memory is obtained
from the OS for them. At least here I had expected that the allocator would
re-use what was already claimed.

The interesting things is that when the queue size is small (50,000 message
for example), this behavior does not occur. Then, memory allocation and
deallocation works as expected.

I have also run the valgrind memory debugger on the app, and it does not
report any leaked memory.

If someone has an explanation of what causes this behavior, I would really
love to hear about it...

Thanks,
Rainer