[rsyslog] Out of memory
david at lang.hm
Mon Jan 23 19:46:23 CET 2017
On Thu, 19 Jan 2017, mostolog--- via rsyslog wrote:
> Really interesting and instructive. Let me see if I understood properly:
> Considering the below explanation, an idle (not going to receive more
> messages the next 7 days) imrelp rsyslog...shouldn't memory be freed
> from current 512MB usage in the near future?
remember that in your case, this is not the cause of the memory usage, you have
a memory leak where the memory remains allocated.
normally this is a small amount of memory (and since the memory isn't used by by
rsyslog, it can be paged out at no cost to the system)
> You mentioned forcing each 100.000 messages, but that's never gonna
> happen if we are still at 20k, and no more messages coming.
> El 19/01/17 a las 11:16, Rainer Gerhards escribió:
>> 2017-01-19 10:19 GMT+01:00 mostolog--- via rsyslog
>> <rsyslog at lists.adiscon.com <mailto:rsyslog at lists.adiscon.com>>:
>> with very few exceptions, rsyslog releases the memory as it
>> goes, there should not be any significant amount of memory
>> freed by rsyslog after it's been idle for a while.
>> But being idle for 15m should release memory from 512MB to a few
>> KB if they aren't used, isnt it?
>> Memory alloc is not as simple as it seems ;-)
>> First, clib does not always do a proper cleanup. I guess it seems not
>> to consolidate free space in all cases. There is a clib call to force
>> this, and rsyslog does it from time to time (I think every 100,000
>> messages). We do not do it very frequently, because it is an expensive
>> operation. Also note that memory is reusable internally, so even
>> though it is not returned to the OS, further alloc requests inside
>> rsyslog can use this memory and do so. Returning memory to the OS and
>> re-claiming it is expensive. Thus you do want to keep some memory
>> allocated but internally unused to avoid doing this operation too
>> Secondly, memory alloc from the OS is done by sbrk IIRC. The
>> important point is that we need to alloc and free memory in sequence.
>> This means if you alloc 100MB, than alloc 1MB, you have the following
>> memory layout
>> with the break at BASE+101MB. If you now free the 100MB chunk, you
>> have this layout:
>> 100MB free
>> if you want to return to the OS, you'd need to copy down the 1MB to
>> immediatly after BASE, because otherwise you cannot reset the break to
>> BASE+1MB. The allocator does not do this, it would totally wrek
>> performance (note: that is not rsyslog specific, that is how the *C
>> runtime* works). More importantly, it would mean all pointers to it
>> would need to be updated. And the runtime does not know where these
>> pointers are located. So it would not only costly, compaction is
>> simply impossible. Let's assume we now alloc another 10 MB. Then we have
>> 90MB free
>> because the allocator uses the free, but still allocated mem. Now,
>> let's assume we free the 1MB chunk, we get:
>> 91MB free
>> Now the free space is at the end of the data segment. So the alloc
>> subsystem has the choice to reduce memory alloc from the OS. It may or
>> may not dealloc. I don't know the exact rules, but the important thing
>> is that the alloc system uses some heuristic (plus the call I
>> mentioned) to decide if to dealloc. Let's assume it does. Then it
>> reduceds the data segement size and we get to
>> effectively reducing rss by 91 MB.
>> IMHO the alloc system strongly works on the assumption that memory
>> allocated (from the OS) but free internally does not really hurt, as
>> it is just virtual address space, which, if actually unused, is paged
>> out to disk once and then doesn't matter at all until it get's reused
>> OF COURSE if we have constantly growing memory, the app seems not to
>> free some chunks, so the alloc system doesn't know they are free. This
>> has nothing to do with what I explained. What I explained just means
>> that an app may do proper free()'s, but the rss size doesn't reflect
>> this. The typical (and often visible) effect of that is that the app
>> grows to a certain size and then remains at it (no growth, no
>> decrease). This is where, via the clib call, we force free.
>> There is one important point, though: if we have a memory block at the
>> far end of the data segment, we cannot return mem to the OS until this
>> block has been freed. In rsyslog, you typical see this if
>> a) old-style "last message repeated n times" is active
>> b) an infrequently written-to output stores a message object when
>> queues are very full
>> c) no different messages are written to that output for a long time
>> What here is very probably that in b) a high memory address is used
>> for that memory block. As due to c) we do not have any traffic, that
>> high address is kept in use for hours, maybe days. Once a different
>> message arrives at the output, the old object is processed and freed.
>> This can result in a very sharp decrease of rss size, especially if
>> the system has low load during such times. Let's assume this layout:
>> 10MB used
>> 1.5GB free, but allocated from OS
>> 5k msg object
>> If we now free those 5k, most probably the alloc system heuristic will
>> immediately return 1.5GB+5K to the OS, reducing the rss size accordingly.
>> Bottom line: not everything that looks like a memory leak actually is one.
>>  https://linux.die.net/man/2/sbrk
> rsyslog mailing list
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
More information about the rsyslog