[rsyslog] memory alloc problems - any advise?

[Rainer Gerhards]

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, June 19, 2009 11:37 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] memory alloc problems - any advise?
> 
> On Fri, 19 Jun 2009, Rainer Gerhards wrote:
> 
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >>
> >> On Fri, 19 Jun 2009, Rainer Gerhards wrote:
> >>
> > I checked with v3, and it had the same behavior (a little less strong,
but
> > still).
> 
> interesting.
> 
> when I was doing the tests that showed rsyslog able to recieve at gig-e
> line speed I would run tests repeatedly without any problems (with each
> test sending millions of logs, things sitting idle for hours, then another
> burst of millions of logs, repeat)
> 
> if I set the queue size too large for my ram the system would start paging
> and die, so I'm sure that it wasn't failing to release memory.
> 
> now, one thing could be that my tests were of messages that were all the
> same size, it could be that you are seeing some effect from fragmentation
> with different message size.

My test driver produces almost identical messages, with mostly the same size.
I have not yet implemented any randomization into it. So in theory, it should
be fairly easy for the malloc() subsystem to reuse freed memory.

Anyhow, based on the malloc stats I got, this really looks like an issue with
the allocator itself. I'll for now implement an alloc_trim() every 100,000
messages or so, that should keep care of this anomaly. Interestingly, v4 now
does fewer allocs than v3. It almost looks like the alloc subsystem also has
a periodic "garbage collection" counter, which now seems to be triggered less
often.

For v5, I have different memory alloc on my mind. My long-term goal is to
have almost no alloc/free pre message (in the magnitude of 1/1024 per
message). But this is rather intrusive and will not have any immediate hard
benefit. Thus I am not targeting this is something that needs to be done now.
For v4, however, I intend to do some more optimizations, some of which
further reduce the number of alloc/free calls needed per message.

I would definitely be a perfect match if you could test out these things on a
high-performance machine.

Rainer