[rsyslog] Out of memory

mostolog at gmail.com mostolog at gmail.com
Wed Jan 18 10:16:16 CET 2017


Hi

This might be not enough yet to diagnose, but here's an spoiler. We have 
managed to review configuration and improve memory footprint, but still 
growing over time and ends being killed by oom:

    module(load="impstats" log.file="/data/stats.log")
    syslog.=debug /data/rsyslog-stats

    global(
         MaxMessageSize="32k"
         workDirectory="/data"
    parser.escapeControlCharactersOnReceive="off"
    )
    input(
         type="imrelp"
         ruleset="relp"
    )
    ruleset(
         name="relp"
         queue.filename="relp"
         queue.maxdiskspace="1G"
         queue.SaveOnShutdown="on"
         queue.type="LinkedList"
         ) {
         action(
             type="mmjsonparse"
         )
         if $parsesuccess == "FAIL" then {
             call error #to write error.log
             stop
         }
         action(
             type="mmnormalize"
             #actually it's doing nothing, cause liblognorm issues.
         )
         set some variables;
         if $!app != $!app then {
             call unknown #to write unk.log
             stop
         }
         #this line include files with if's to re-set variables
         $IncludeConfig /etc/rsyslog.d/apps/conf/3*.conf
         call clean
         set a few variables;
         action(
             type="omelasticsearch"
         )
    }
    #include other rulesets to handle other things, like error or unknown
    $IncludeConfig /etc/rsyslog.d/apps/conf/4*.conf


Wed Jan 18 09:47:57 2017: global: origin=dynstats
Wed Jan 18 09:47:57 2017: action 0: origin=core.action processed=0 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: omelasticsearch: origin=omelasticsearch 
submitted=235 failed.http=0 failed.httprequests=0 failed.checkConn=0 
failed.es=0
Wed Jan 18 09:47:57 2017: json: origin=core.action processed=235 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: norm: origin=core.action processed=235 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: elastic: origin=core.action processed=235 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: error: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: unk: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: imrelp[20514]: origin=imrelp submitted=235
Wed Jan 18 09:47:57 2017: resource-usage: origin=impstats utime=168000 
stime=124000 maxrss=7828 minflt=1383 majflt=0 inblock=10 oublock=16 
nvcsw=1628 nivcsw=44
Wed Jan 18 09:47:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0 
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 09:47:57 2017: relp: origin=core.queue size=0 enqueued=235 
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 09:47:57 2017: main Q: origin=core.queue size=9 enqueued=12 
full=0 discarded.full=0 discarded.nf=0 maxqsize=9
Wed Jan 18 09:52:57 2017: global: origin=dynstats
Wed Jan 18 09:52:57 2017: action 0: origin=core.action processed=0 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: omelasticsearch: origin=omelasticsearch 
submitted=418 failed.http=0 failed.httprequests=0 failed.checkConn=0 
failed.es=0
Wed Jan 18 09:52:57 2017: json: origin=core.action processed=418 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: norm: origin=core.action processed=418 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: elastic: origin=core.action processed=418 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: error: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: unk: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: imrelp[20514]: origin=imrelp submitted=418
Wed Jan 18 09:52:57 2017: resource-usage: origin=impstats utime=236000 
stime=232000 maxrss=7828 minflt=1385 majflt=0 inblock=10 oublock=18 
nvcsw=2710 nivcsw=59
Wed Jan 18 09:52:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0 
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 09:52:57 2017: relp: origin=core.queue size=0 enqueued=418 
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 09:52:57 2017: main Q: origin=core.queue size=0 enqueued=25 
full=0 discarded.full=0 discarded.nf=0 maxqsize=9
Wed Jan 18 09:57:57 2017: global: origin=dynstats
Wed Jan 18 09:57:57 2017: action 0: origin=core.action processed=0 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:57:57 2017: omelasticsearch: origin=omelasticsearch 
submitted=574 failed.http=0 failed.httprequests=0 failed.checkConn=0 
failed.es=0
Wed Jan 18 09:57:57 2017: json: origin=core.action processed=574 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:57:57 2017: norm: origin=core.action processed=574 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:57:57 2017: elastic: origin=core.action processed=574 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:57:57 2017: error: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:57:57 2017: unk: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:57:57 2017: imrelp[20514]: origin=imrelp submitted=574
Wed Jan 18 09:57:57 2017: resource-usage: origin=impstats utime=332000 
stime=288000 maxrss=7828 minflt=1385 majflt=0 inblock=10 oublock=21 
nvcsw=3634 nivcsw=67
Wed Jan 18 09:57:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0 
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 09:57:57 2017: relp: origin=core.queue size=0 enqueued=574 
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 09:57:57 2017: main Q: origin=core.queue size=0 enqueued=38 
full=0 discarded.full=0 discarded.nf=0 maxqsize=9
Wed Jan 18 10:02:57 2017: global: origin=dynstats
Wed Jan 18 10:02:57 2017: action 0: origin=core.action processed=0 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:02:57 2017: omelasticsearch: origin=omelasticsearch 
submitted=755 failed.http=0 failed.httprequests=0 failed.checkConn=0 
failed.es=0
Wed Jan 18 10:02:57 2017: json: origin=core.action processed=755 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:02:57 2017: norm: origin=core.action processed=755 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:02:57 2017: elastic: origin=core.action processed=755 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:02:57 2017: error: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:02:57 2017: unk: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:02:57 2017: imrelp[20514]: origin=imrelp submitted=755
Wed Jan 18 10:02:57 2017: resource-usage: origin=impstats utime=416000 
stime=376000 maxrss=7828 minflt=1388 majflt=0 inblock=10 oublock=24 
nvcsw=4689 nivcsw=75
Wed Jan 18 10:02:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0 
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:02:57 2017: relp: origin=core.queue size=0 enqueued=755 
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 10:02:57 2017: main Q: origin=core.queue size=0 enqueued=51 
full=0 discarded.full=0 discarded.nf=0 maxqsize=9
Wed Jan 18 10:07:57 2017: global: origin=dynstats
Wed Jan 18 10:07:57 2017: action 0: origin=core.action processed=0 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:07:57 2017: omelasticsearch: origin=omelasticsearch 
submitted=934 failed.http=0 failed.httprequests=0 failed.checkConn=0 
failed.es=0
Wed Jan 18 10:07:57 2017: json: origin=core.action processed=934 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:07:57 2017: norm: origin=core.action processed=934 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:07:57 2017: elastic: origin=core.action processed=934 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:07:57 2017: error: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:07:57 2017: unk: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:07:57 2017: imrelp[20514]: origin=imrelp submitted=934
Wed Jan 18 10:07:57 2017: resource-usage: origin=impstats utime=520000 
stime=448000 maxrss=7828 minflt=1398 majflt=0 inblock=10 oublock=27 
nvcsw=5741 nivcsw=86
Wed Jan 18 10:07:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0 
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:07:57 2017: relp: origin=core.queue size=0 enqueued=934 
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 10:07:57 2017: main Q: origin=core.queue size=0 enqueued=64 
full=0 discarded.full=0 discarded.nf=0 maxqsize=9
Wed Jan 18 10:12:57 2017: global: origin=dynstats
Wed Jan 18 10:12:57 2017: action 0: origin=core.action processed=0 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:12:57 2017: omelasticsearch: origin=omelasticsearch 
submitted=1114 failed.http=0 failed.httprequests=0 failed.checkConn=0 
failed.es=0
Wed Jan 18 10:12:57 2017: json: origin=core.action processed=1114 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:12:57 2017: norm: origin=core.action processed=1114 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:12:57 2017: elastic: origin=core.action processed=1114 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:12:57 2017: error: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:12:57 2017: unk: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:12:57 2017: imrelp[20514]: origin=imrelp submitted=1114
Wed Jan 18 10:12:57 2017: resource-usage: origin=impstats utime=608000 
stime=528000 maxrss=7828 minflt=1411 majflt=0 inblock=10 oublock=30 
nvcsw=6812 nivcsw=93
Wed Jan 18 10:12:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0 
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:12:57 2017: relp: origin=core.queue size=0 enqueued=1114 
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 10:12:57 2017: main Q: origin=core.queue size=0 enqueued=77 
full=0 discarded.full=0 discarded.nf=0 maxqsize=9

Feedback much appreciated.


More information about the rsyslog mailing list