[rsyslog] Out of memory

David Lang david at lang.hm
Mon Jan 23 19:46:23 CET 2017


On Thu, 19 Jan 2017, mostolog--- via rsyslog wrote:

> Really interesting and instructive. Let me see if I understood properly:
>
> Considering the below explanation, an idle (not going to receive more 
> messages the next 7 days) imrelp rsyslog...shouldn't memory be freed 
> from current 512MB usage in the near future?

remember that in your case, this is not the cause of the memory usage, you have 
a memory leak where the memory remains allocated.

normally this is a small amount of memory (and since the memory isn't used by by 
rsyslog, it can be paged out at no cost to the system)

David Lang

> You mentioned forcing each 100.000 messages, but that's never gonna 
> happen if we are still at 20k, and no more messages coming.
>
> Regards
>
>
> El 19/01/17 a las 11:16, Rainer Gerhards escribió:
>> 2017-01-19 10:19 GMT+01:00 mostolog--- via rsyslog 
>> <rsyslog at lists.adiscon.com <mailto:rsyslog at lists.adiscon.com>>:
>>
>>         with very few exceptions, rsyslog releases the memory as it
>>         goes, there should not be any significant amount of memory
>>         freed by rsyslog after it's been idle for a while.
>>
>>     But being idle for 15m should release memory from 512MB to a few
>>     KB if they aren't used, isnt it?
>>
>>
>> Memory alloc is not as simple as it seems ;-)
>>
>> First, clib does not always do a proper cleanup. I guess it seems not 
>> to consolidate free space in all cases. There is a clib call to force 
>> this, and rsyslog does it from time to time (I think every 100,000 
>> messages). We do not do it very frequently, because it is an expensive 
>> operation. Also note that memory is reusable internally, so even 
>> though it is not returned to the OS, further alloc requests inside 
>> rsyslog can use this memory and do so. Returning memory to the OS and 
>> re-claiming it is expensive. Thus you do want to keep some memory 
>> allocated but internally unused to avoid doing this operation too 
>> frequently.
>>
>> Secondly, memory alloc from the OS is done by sbrk[1] IIRC. The 
>> important point is that we need to alloc and free memory in sequence. 
>> This means if you alloc 100MB, than alloc 1MB, you have the following 
>> memory layout
>>
>> BASE
>> 100MB
>> 1MB
>>
>> with the break at BASE+101MB. If you now free the 100MB chunk, you 
>> have this layout:
>>
>> BASE
>> 100MB free
>> 1MB
>>
>> if you want to return to the OS, you'd need to copy down the 1MB to 
>> immediatly after BASE, because otherwise you cannot reset the break to 
>> BASE+1MB. The allocator does not do this, it would totally wrek 
>> performance (note: that is not rsyslog specific, that is how the *C 
>> runtime* works). More importantly, it would mean all pointers to it 
>> would need to be updated. And the runtime does not know where these 
>> pointers are located. So it would not only costly, compaction is 
>> simply impossible. Let's assume we now alloc another 10 MB. Then we have
>>
>> BASE
>> 10MB
>> 90MB free
>> 1MB
>>
>> because the allocator uses the free, but still allocated mem. Now, 
>> let's assume we free the 1MB chunk, we get:
>>
>> BASE
>> 10MB
>> 91MB free
>>
>> Now the free space is at the end of the data segment. So the alloc 
>> subsystem has the choice to reduce memory alloc from the OS. It may or 
>> may not dealloc. I don't know the exact rules, but the important thing 
>> is that the alloc system uses some heuristic (plus the call I 
>> mentioned) to decide if to dealloc. Let's assume it does. Then it 
>> reduceds the data segement size and we get to
>>
>> BASE
>> 10MB
>>
>> effectively reducing rss by 91 MB.
>>
>> IMHO the alloc system strongly works on the assumption that memory 
>> allocated (from the OS) but free internally does not really hurt, as 
>> it is just virtual address space, which, if actually unused, is paged 
>> out to disk once and then doesn't matter at all until it get's reused 
>> again.
>>
>> OF COURSE if we have constantly growing memory, the app seems not to 
>> free some chunks, so the alloc system doesn't know they are free. This 
>> has nothing to do with what I explained. What I explained just means 
>> that an app may do proper free()'s, but the rss size doesn't reflect 
>> this. The typical (and often visible) effect of that is that the app 
>> grows to a certain size and then remains at it (no growth, no 
>> decrease). This is where, via the clib call, we force free.
>>
>> There is one important point, though: if we have a memory block at the 
>> far end of the data segment, we cannot return mem to the OS until this 
>> block has been freed. In rsyslog, you typical see this if
>>
>> a) old-style "last message repeated n times" is active
>> b) an infrequently written-to output stores a message object when 
>> queues are very full
>> c) no different messages are written to that output for a long time
>>
>> What here is very probably that in b) a high memory address is used 
>> for that memory block. As due to c) we do not have any traffic, that 
>> high address is kept in use for hours, maybe days. Once a different 
>> message arrives at the output, the old object is processed and freed. 
>> This can result in a very sharp decrease of rss size, especially if 
>> the system has low load during such times. Let's assume this layout:
>>
>> BASE
>> 10MB used
>> 1.5GB free, but allocated from OS
>> 5k msg object
>>
>> If we now free those 5k, most probably the alloc system heuristic will 
>> immediately return 1.5GB+5K to the OS, reducing the rss size accordingly.
>>
>> Bottom line: not everything that looks like a memory leak actually is one.
>>
>> HTH
>> [1] https://linux.die.net/man/2/sbrk
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.


More information about the rsyslog mailing list