[rsyslog] Out of memory

Rainer Gerhards rgerhards at hq.adiscon.com
Wed Jan 18 17:05:19 CET 2017


If this is a test environment, I would strongly recommend to run it in a
terminal session under valgrind control. If it is an actual leak, valgrind
can provide very good diagnostic information.

That would be very helpful and clarify what is going on.

Rainer

2017-01-18 16:16 GMT+01:00 mostolog--- via rsyslog <
rsyslog at lists.adiscon.com>:

>
> Below an impstats trace showing a few tests:
>
>  * receiving some idle messages
>  * 1 message per second
>  * 10m/sec
>  * 100m/s
>  * 1000m/s
>  * 2000m/s
>
> None of the above created any queue files when using omfile as output.
>
> *Still, memory was slowly increasing (RGROW=1056K with 100m/s input)
> according to atop. Shall we have a look on this?
> *
>
> *omfile/without omelasticsearch*
>
>    Wed Jan 18 13:28:46 2017: global: origin=dynstats
>    Wed Jan 18 13:28:46 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:28:46 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:28:46 2017: json: origin=core.action processed=83
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:28:46 2017: norm: origin=core.action processed=83
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:28:46 2017: tofile: origin=core.action processed=83
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:28:46 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:28:46 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:28:46 2017: imrelp[20514]: origin=imrelp submitted=83
>    Wed Jan 18 13:28:46 2017: resource-usage: origin=impstats
>    utime=40000 stime=28000 maxrss=7176 minflt=1278 majflt=0 inblock=10
>    oublock=87 nvcsw=395 nivcsw=48
>    Wed Jan 18 13:28:46 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:28:46 2017: relp: origin=core.queue size=0 enqueued=83
>    full=0 discarded.full=0 discarded.nf=0 maxqsize=53
>    Wed Jan 18 13:28:46 2017: main Q: origin=core.queue size=0
>    enqueued=12 full=0 discarded.full=0 discarded.nf=0 maxqsize=1
>    Wed Jan 18 13:30:51 2017: global: origin=dynstats
>    Wed Jan 18 13:30:51 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:30:51 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:30:51 2017: json: origin=core.action processed=89
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:30:51 2017: norm: origin=core.action processed=89
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:30:51 2017: tofile: origin=core.action processed=89
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:30:51 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:30:51 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:30:51 2017: imrelp[20514]: origin=imrelp submitted=89
>    Wed Jan 18 13:30:51 2017: resource-usage: origin=impstats
>    utime=48000 stime=36000 maxrss=7212 minflt=1318 majflt=0 inblock=10
>    oublock=92 nvcsw=439 nivcsw=55
>    Wed Jan 18 13:30:51 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:30:51 2017: relp: origin=core.queue size=0 enqueued=89
>    full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:30:51 2017: main Q: origin=core.queue size=0
>    enqueued=12 full=0 discarded.full=0 discarded.nf=0 maxqsize=4
>    Wed Jan 18 13:32:31 2017: global: origin=dynstats
>    Wed Jan 18 13:32:31 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:32:31 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:32:31 2017: json: origin=core.action processed=486
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:32:31 2017: norm: origin=core.action processed=486
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:32:31 2017: tofile: origin=core.action processed=486
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:32:31 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:32:31 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:32:31 2017: imrelp[20514]: origin=imrelp submitted=486
>    Wed Jan 18 13:32:31 2017: resource-usage: origin=impstats
>    utime=80000 stime=88000 maxrss=7212 minflt=1323 majflt=0 inblock=10
>    oublock=468 nvcsw=1019 nivcsw=63
>    Wed Jan 18 13:32:31 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:32:31 2017: relp: origin=core.queue size=0
>    enqueued=486 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:32:31 2017: main Q: origin=core.queue size=1
>    enqueued=25 full=0 discarded.full=0 discarded.nf=0 maxqsize=4
>    Wed Jan 18 13:34:11 2017: global: origin=dynstats
>    Wed Jan 18 13:34:11 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:34:11 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:34:11 2017: json: origin=core.action processed=1318
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:34:11 2017: norm: origin=core.action processed=1318
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:34:11 2017: tofile: origin=core.action processed=1318
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:34:11 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:34:11 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:34:11 2017: imrelp[20514]: origin=imrelp submitted=1318
>    Wed Jan 18 13:34:11 2017: resource-usage: origin=impstats
>    utime=176000 stime=176000 maxrss=7212 minflt=1354 majflt=0
>    inblock=10 oublock=1251 nvcsw=2327 nivcsw=103
>    Wed Jan 18 13:34:11 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:34:11 2017: relp: origin=core.queue size=0
>    enqueued=1318 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:34:11 2017: main Q: origin=core.queue size=0
>    enqueued=38 full=0 discarded.full=0 discarded.nf=0 maxqsize=4
>    Wed Jan 18 13:35:51 2017: global: origin=dynstats
>    Wed Jan 18 13:35:51 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:35:51 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:35:51 2017: json: origin=core.action processed=2188
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:35:51 2017: norm: origin=core.action processed=2188
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:35:51 2017: tofile: origin=core.action processed=2188
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:35:51 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:35:51 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:35:51 2017: imrelp[20514]: origin=imrelp submitted=2188
>    Wed Jan 18 13:35:51 2017: resource-usage: origin=impstats
>    utime=260000 stime=260000 maxrss=7476 minflt=1417 majflt=0
>    inblock=10 oublock=2068 nvcsw=3704 nivcsw=130
>    Wed Jan 18 13:35:51 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:35:51 2017: relp: origin=core.queue size=0
>    enqueued=2188 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:35:51 2017: main Q: origin=core.queue size=0
>    enqueued=51 full=0 discarded.full=0 discarded.nf=0 maxqsize=4
>    Wed Jan 18 13:37:31 2017: global: origin=dynstats
>    Wed Jan 18 13:37:31 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:37:31 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:37:31 2017: json: origin=core.action processed=2998
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:37:31 2017: norm: origin=core.action processed=2998
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:37:31 2017: tofile: origin=core.action processed=2998
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:37:31 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:37:31 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:37:31 2017: imrelp[20514]: origin=imrelp submitted=2998
>    Wed Jan 18 13:37:31 2017: resource-usage: origin=impstats
>    utime=352000 stime=340000 maxrss=7740 minflt=1475 majflt=0
>    inblock=10 oublock=2831 nvcsw=5027 nivcsw=152
>    Wed Jan 18 13:37:31 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:37:31 2017: relp: origin=core.queue size=0
>    enqueued=2998 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:37:31 2017: main Q: origin=core.queue size=0
>    enqueued=64 full=0 discarded.full=0 discarded.nf=0 maxqsize=4
>    Wed Jan 18 13:39:12 2017: global: origin=dynstats
>    Wed Jan 18 13:39:12 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:39:12 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:39:12 2017: json: origin=core.action processed=3838
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:39:12 2017: norm: origin=core.action processed=3838
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:39:12 2017: tofile: origin=core.action processed=3838
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:39:12 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:39:12 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:39:12 2017: imrelp[20514]: origin=imrelp submitted=3838
>    Wed Jan 18 13:39:12 2017: resource-usage: origin=impstats
>    utime=424000 stime=444000 maxrss=8004 minflt=1536 majflt=0
>    inblock=10 oublock=3621 nvcsw=6415 nivcsw=174
>    Wed Jan 18 13:39:12 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:39:12 2017: relp: origin=core.queue size=0
>    enqueued=3838 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:39:12 2017: main Q: origin=core.queue size=0
>    enqueued=77 full=0 discarded.full=0 discarded.nf=0 maxqsize=4
>    Wed Jan 18 13:40:52 2017: global: origin=dynstats
>    Wed Jan 18 13:40:52 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:40:52 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:40:52 2017: json: origin=core.action processed=4698
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:40:52 2017: norm: origin=core.action processed=4698
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:40:52 2017: tofile: origin=core.action processed=4698
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:40:52 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:40:52 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:40:52 2017: imrelp[20514]: origin=imrelp submitted=4698
>    Wed Jan 18 13:40:52 2017: resource-usage: origin=impstats
>    utime=520000 stime=524000 maxrss=8268 minflt=1598 majflt=0
>    inblock=10 oublock=4428 nvcsw=7742 nivcsw=216
>    Wed Jan 18 13:40:52 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:40:52 2017: relp: origin=core.queue size=0
>    enqueued=4698 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:40:52 2017: main Q: origin=core.queue size=0
>    enqueued=90 full=0 discarded.full=0 discarded.nf=0 maxqsize=4
>    Wed Jan 18 13:42:32 2017: global: origin=dynstats
>    Wed Jan 18 13:42:32 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:42:32 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:42:32 2017: json: origin=core.action processed=5513
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:42:32 2017: norm: origin=core.action processed=5513
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:42:32 2017: tofile: origin=core.action processed=5513
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:42:32 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:42:32 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:42:32 2017: imrelp[20514]: origin=imrelp submitted=5513
>    Wed Jan 18 13:42:32 2017: resource-usage: origin=impstats
>    utime=584000 stime=616000 maxrss=8532 minflt=1659 majflt=0
>    inblock=10 oublock=5197 nvcsw=9004 nivcsw=244
>    Wed Jan 18 13:42:32 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:42:32 2017: relp: origin=core.queue size=0
>    enqueued=5513 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:42:32 2017: main Q: origin=core.queue size=0
>    enqueued=103 full=0 discarded.full=0 discarded.nf=0 maxqsize=4
>    Wed Jan 18 13:44:12 2017: global: origin=dynstats
>    Wed Jan 18 13:44:12 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:44:12 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:44:12 2017: json: origin=core.action processed=6329
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:44:12 2017: norm: origin=core.action processed=6329
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:44:12 2017: tofile: origin=core.action processed=6329
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:44:12 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:44:12 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:44:12 2017: imrelp[20514]: origin=imrelp submitted=6329
>    Wed Jan 18 13:44:12 2017: resource-usage: origin=impstats
>    utime=660000 stime=708000 maxrss=8796 minflt=1716 majflt=0
>    inblock=10 oublock=5964 nvcsw=10172 nivcsw=286
>    Wed Jan 18 13:44:12 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:44:12 2017: relp: origin=core.queue size=0
>    enqueued=6329 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:44:12 2017: main Q: origin=core.queue size=0
>    enqueued=116 full=0 discarded.full=0 discarded.nf=0 maxqsize=4
>    Wed Jan 18 13:45:52 2017: global: origin=dynstats
>    Wed Jan 18 13:45:52 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:45:52 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:45:52 2017: json: origin=core.action processed=7227
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:45:52 2017: norm: origin=core.action processed=7227
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:45:52 2017: tofile: origin=core.action processed=7227
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:45:52 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:45:52 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:45:52 2017: imrelp[20514]: origin=imrelp submitted=7227
>    Wed Jan 18 13:45:52 2017: resource-usage: origin=impstats
>    utime=732000 stime=776000 maxrss=9060 minflt=1781 majflt=0
>    inblock=10 oublock=6812 nvcsw=11502 nivcsw=315
>    Wed Jan 18 13:45:52 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:45:52 2017: relp: origin=core.queue size=0
>    enqueued=7227 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:45:52 2017: main Q: origin=core.queue size=0
>    enqueued=129 full=0 discarded.full=0 discarded.nf=0 maxqsize=4
>    Wed Jan 18 13:47:32 2017: global: origin=dynstats
>    Wed Jan 18 13:47:32 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:47:32 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:47:32 2017: json: origin=core.action processed=7245
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:47:32 2017: norm: origin=core.action processed=7245
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:47:32 2017: tofile: origin=core.action processed=7245
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:47:32 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:47:32 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:47:32 2017: imrelp[20514]: origin=imrelp submitted=7245
>    Wed Jan 18 13:47:32 2017: resource-usage: origin=impstats
>    utime=736000 stime=776000 maxrss=9060 minflt=1783 majflt=0
>    inblock=10 oublock=6832 nvcsw=11541 nivcsw=318
>    Wed Jan 18 13:47:32 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:47:32 2017: relp: origin=core.queue size=0
>    enqueued=7245 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:47:32 2017: main Q: origin=core.queue size=0
>    enqueued=142 full=0 discarded.full=0 discarded.nf=0 maxqsize=4
>    Wed Jan 18 13:49:12 2017: global: origin=dynstats
>    Wed Jan 18 13:49:12 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:49:12 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:49:12 2017: json: origin=core.action processed=38503
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:49:12 2017: norm: origin=core.action processed=38503
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:49:12 2017: tofile: origin=core.action processed=38503
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:49:12 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:49:12 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:49:12 2017: imrelp[20514]: origin=imrelp submitted=38503
>    Wed Jan 18 13:49:12 2017: resource-usage: origin=impstats
>    utime=3780000 stime=3104000 maxrss=18036 minflt=4021 majflt=0
>    inblock=10 oublock=36109 nvcsw=62186 nivcsw=894
>    Wed Jan 18 13:49:12 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:49:12 2017: relp: origin=core.queue size=0
>    enqueued=38503 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:49:12 2017: main Q: origin=core.queue size=1
>    enqueued=155 full=0 discarded.full=0 discarded.nf=0 maxqsize=4
>    Wed Jan 18 13:50:52 2017: global: origin=dynstats
>    Wed Jan 18 13:50:52 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:50:52 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:50:52 2017: json: origin=core.action processed=74584
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:50:52 2017: norm: origin=core.action processed=74584
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:50:52 2017: tofile: origin=core.action processed=74584
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:50:52 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:50:52 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:50:52 2017: imrelp[20514]: origin=imrelp submitted=74584
>    Wed Jan 18 13:50:52 2017: resource-usage: origin=impstats
>    utime=7208000 stime=5528000 maxrss=28332 minflt=6599 majflt=0
>    inblock=10 oublock=69900 nvcsw=122477 nivcsw=1349
>    Wed Jan 18 13:50:52 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:50:52 2017: relp: origin=core.queue size=0
>    enqueued=74584 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:50:52 2017: main Q: origin=core.queue size=0
>    enqueued=168 full=0 discarded.full=0 discarded.nf=0 maxqsize=4
>    Wed Jan 18 13:52:32 2017: global: origin=dynstats
>    Wed Jan 18 13:52:32 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:52:32 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:52:32 2017: json: origin=core.action processed=112795
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:52:32 2017: norm: origin=core.action processed=112795
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:52:32 2017: tofile: origin=core.action
>    processed=112795 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:52:32 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:52:32 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:52:32 2017: imrelp[20514]: origin=imrelp submitted=112796
>    Wed Jan 18 13:52:32 2017: resource-usage: origin=impstats
>    utime=10676000 stime=8308000 maxrss=39296 minflt=9332 majflt=0
>    inblock=10 oublock=105688 nvcsw=187173 nivcsw=1884
>    Wed Jan 18 13:52:32 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:52:32 2017: relp: origin=core.queue size=0
>    enqueued=112796 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:52:32 2017: main Q: origin=core.queue size=0
>    enqueued=181 full=0 discarded.full=0 discarded.nf=0 maxqsize=4
>    Wed Jan 18 13:54:12 2017: global: origin=dynstats
>    Wed Jan 18 13:54:12 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:54:12 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:54:12 2017: json: origin=core.action processed=150321
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:54:12 2017: norm: origin=core.action processed=150321
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:54:12 2017: tofile: origin=core.action
>    processed=150321 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:54:12 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:54:12 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:54:12 2017: imrelp[20514]: origin=imrelp submitted=150321
>    Wed Jan 18 13:54:12 2017: resource-usage: origin=impstats
>    utime=14404000 stime=10956000 maxrss=49856 minflt=12016 majflt=0
>    inblock=10 oublock=140834 nvcsw=250888 nivcsw=2373
>    Wed Jan 18 13:54:12 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:54:12 2017: relp: origin=core.queue size=1
>    enqueued=150322 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:54:12 2017: main Q: origin=core.queue size=4
>    enqueued=194 full=0 discarded.full=0 discarded.nf=0 maxqsize=4
>    Wed Jan 18 13:55:52 2017: global: origin=dynstats
>    Wed Jan 18 13:55:52 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:55:52 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:55:52 2017: json: origin=core.action processed=188357
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:55:52 2017: norm: origin=core.action processed=188357
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:55:52 2017: tofile: origin=core.action
>    processed=188357 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:55:52 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:55:52 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:55:52 2017: imrelp[20514]: origin=imrelp submitted=188357
>    Wed Jan 18 13:55:52 2017: resource-usage: origin=impstats
>    utime=18296000 stime=13868000 maxrss=60680 minflt=14735 majflt=0
>    inblock=10 oublock=176456 nvcsw=314849 nivcsw=2957
>    Wed Jan 18 13:55:52 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:55:52 2017: relp: origin=core.queue size=0
>    enqueued=188357 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:55:52 2017: main Q: origin=core.queue size=0
>    enqueued=207 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>    Wed Jan 18 13:57:32 2017: global: origin=dynstats
>    Wed Jan 18 13:57:32 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:57:32 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:57:32 2017: json: origin=core.action processed=211610
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:57:32 2017: norm: origin=core.action processed=211610
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:57:32 2017: tofile: origin=core.action
>    processed=211610 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:57:32 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:57:32 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:57:32 2017: imrelp[20514]: origin=imrelp submitted=211610
>    Wed Jan 18 13:57:32 2017: resource-usage: origin=impstats
>    utime=20712000 stime=15788000 maxrss=67312 minflt=16396 majflt=0
>    inblock=10 oublock=198239 nvcsw=353668 nivcsw=3420
>    Wed Jan 18 13:57:32 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:57:32 2017: relp: origin=core.queue size=0
>    enqueued=211610 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:57:32 2017: main Q: origin=core.queue size=0
>    enqueued=220 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>    Wed Jan 18 13:59:12 2017: global: origin=dynstats
>    Wed Jan 18 13:59:12 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:59:12 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 13:59:12 2017: json: origin=core.action processed=211658
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:59:12 2017: norm: origin=core.action processed=211658
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:59:12 2017: tofile: origin=core.action
>    processed=211658 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:59:12 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:59:12 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 13:59:12 2017: imrelp[20514]: origin=imrelp submitted=211658
>    Wed Jan 18 13:59:12 2017: resource-usage: origin=impstats
>    utime=20720000 stime=15788000 maxrss=67312 minflt=16402 majflt=0
>    inblock=10 oublock=198287 nvcsw=353748 nivcsw=3420
>    Wed Jan 18 13:59:12 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 13:59:12 2017: relp: origin=core.queue size=0
>    enqueued=211658 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 13:59:12 2017: main Q: origin=core.queue size=0
>    enqueued=233 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>    Wed Jan 18 14:00:52 2017: global: origin=dynstats
>    Wed Jan 18 14:00:52 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:00:52 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 14:00:52 2017: json: origin=core.action processed=260651
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:00:52 2017: norm: origin=core.action processed=260651
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:00:52 2017: tofile: origin=core.action
>    processed=260651 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:00:52 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:00:52 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:00:52 2017: imrelp[20514]: origin=imrelp submitted=260651
>    Wed Jan 18 14:00:52 2017: resource-usage: origin=impstats
>    utime=25916000 stime=19604000 maxrss=81568 minflt=19920 majflt=0
>    inblock=10 oublock=244174 nvcsw=435288 nivcsw=4151
>    Wed Jan 18 14:00:52 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 14:00:52 2017: relp: origin=core.queue size=0
>    enqueued=260651 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 14:00:52 2017: main Q: origin=core.queue size=0
>    enqueued=246 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>    Wed Jan 18 14:02:32 2017: global: origin=dynstats
>    Wed Jan 18 14:02:32 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:02:32 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 14:02:32 2017: json: origin=core.action processed=323799
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:02:32 2017: norm: origin=core.action processed=323799
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:02:32 2017: tofile: origin=core.action
>    processed=323799 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:02:32 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:02:32 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:02:32 2017: imrelp[20514]: origin=imrelp submitted=323799
>    Wed Jan 18 14:02:32 2017: resource-usage: origin=impstats
>    utime=32336000 stime=24796000 maxrss=99644 minflt=24447 majflt=0
>    inblock=10 oublock=303433 nvcsw=538266 nivcsw=5075
>    Wed Jan 18 14:02:32 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 14:02:32 2017: relp: origin=core.queue size=0
>    enqueued=323799 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 14:02:32 2017: main Q: origin=core.queue size=0
>    enqueued=259 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>    Wed Jan 18 14:04:12 2017: global: origin=dynstats
>    Wed Jan 18 14:04:12 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:04:12 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 14:04:12 2017: json: origin=core.action processed=388417
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:04:12 2017: norm: origin=core.action processed=388417
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:04:12 2017: tofile: origin=core.action
>    processed=388417 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:04:12 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:04:12 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:04:12 2017: imrelp[20514]: origin=imrelp submitted=388417
>    Wed Jan 18 14:04:12 2017: resource-usage: origin=impstats
>    utime=39384000 stime=30512000 maxrss=118124 minflt=29081 majflt=0
>    inblock=10 oublock=364203 nvcsw=640038 nivcsw=6096
>    Wed Jan 18 14:04:12 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 14:04:12 2017: relp: origin=core.queue size=0
>    enqueued=388417 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 14:04:12 2017: main Q: origin=core.queue size=0
>    enqueued=272 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>    Wed Jan 18 14:05:52 2017: global: origin=dynstats
>    Wed Jan 18 14:05:52 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:05:52 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 14:05:52 2017: json: origin=core.action processed=396371
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:05:52 2017: norm: origin=core.action processed=396371
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:05:52 2017: tofile: origin=core.action
>    processed=396371 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:05:52 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:05:52 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:05:52 2017: imrelp[20514]: origin=imrelp submitted=396371
>    Wed Jan 18 14:05:52 2017: resource-usage: origin=impstats
>    utime=40196000 stime=31096000 maxrss=120236 minflt=29650 majflt=0
>    inblock=10 oublock=371686 nvcsw=653582 nivcsw=6187
>    Wed Jan 18 14:05:52 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 14:05:52 2017: relp: origin=core.queue size=0
>    enqueued=396371 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 14:05:52 2017: main Q: origin=core.queue size=0
>    enqueued=285 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>    Wed Jan 18 14:07:33 2017: global: origin=dynstats
>    Wed Jan 18 14:07:33 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:07:33 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 14:07:33 2017: json: origin=core.action processed=396389
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:07:33 2017: norm: origin=core.action processed=396389
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:07:33 2017: tofile: origin=core.action
>    processed=396389 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:07:33 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:07:33 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:07:33 2017: imrelp[20514]: origin=imrelp submitted=396389
>    Wed Jan 18 14:07:33 2017: resource-usage: origin=impstats
>    utime=40200000 stime=31100000 maxrss=120236 minflt=29653 majflt=0
>    inblock=10 oublock=371707 nvcsw=653624 nivcsw=6189
>    Wed Jan 18 14:07:33 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 14:07:33 2017: relp: origin=core.queue size=0
>    enqueued=396389 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 14:07:33 2017: main Q: origin=core.queue size=0
>    enqueued=298 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>    Wed Jan 18 14:09:13 2017: global: origin=dynstats
>    Wed Jan 18 14:09:13 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:09:13 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 14:09:13 2017: json: origin=core.action processed=412439
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:09:13 2017: norm: origin=core.action processed=412439
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:09:13 2017: tofile: origin=core.action
>    processed=412439 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:09:13 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:09:13 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:09:13 2017: imrelp[20514]: origin=imrelp submitted=412439
>    Wed Jan 18 14:09:13 2017: resource-usage: origin=impstats
>    utime=41512000 stime=31976000 maxrss=125148 minflt=30820 majflt=0
>    inblock=10 oublock=386804 nvcsw=673808 nivcsw=6457
>    Wed Jan 18 14:09:13 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 14:09:13 2017: relp: origin=core.queue size=0
>    enqueued=412439 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 14:09:13 2017: main Q: origin=core.queue size=0
>    enqueued=311 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>    Wed Jan 18 14:10:53 2017: global: origin=dynstats
>    Wed Jan 18 14:10:53 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:10:53 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 14:10:53 2017: json: origin=core.action processed=533887
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:10:53 2017: norm: origin=core.action processed=533887
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:10:53 2017: tofile: origin=core.action
>    processed=533887 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:10:53 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:10:53 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:10:53 2017: imrelp[20514]: origin=imrelp submitted=533887
>    Wed Jan 18 14:10:53 2017: resource-usage: origin=impstats
>    utime=52020000 stime=39048000 maxrss=160108 minflt=39604 majflt=0
>    inblock=10 oublock=501018 nvcsw=838060 nivcsw=8119
>    Wed Jan 18 14:10:53 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 14:10:53 2017: relp: origin=core.queue size=0
>    enqueued=533887 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 14:10:53 2017: main Q: origin=core.queue size=0
>    enqueued=324 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>    Wed Jan 18 14:12:33 2017: global: origin=dynstats
>    Wed Jan 18 14:12:33 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:12:33 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 14:12:33 2017: json: origin=core.action processed=654124
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:12:33 2017: norm: origin=core.action processed=654124
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:12:33 2017: tofile: origin=core.action
>    processed=654124 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:12:33 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:12:33 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:12:33 2017: imrelp[20514]: origin=imrelp submitted=654124
>    Wed Jan 18 14:12:33 2017: resource-usage: origin=impstats
>    utime=62692000 stime=46500000 maxrss=194788 minflt=48284 majflt=0
>    inblock=10 oublock=614093 nvcsw=996589 nivcsw=9741
>    Wed Jan 18 14:12:33 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 14:12:33 2017: relp: origin=core.queue size=0
>    enqueued=654124 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 14:12:33 2017: main Q: origin=core.queue size=1
>    enqueued=337 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>    Wed Jan 18 14:14:13 2017: global: origin=dynstats
>    Wed Jan 18 14:14:13 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:14:13 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 14:14:13 2017: json: origin=core.action processed=758667
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:14:13 2017: norm: origin=core.action processed=758667
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:14:13 2017: tofile: origin=core.action
>    processed=758667 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:14:13 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:14:13 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:14:13 2017: imrelp[20514]: origin=imrelp submitted=758669
>    Wed Jan 18 14:14:13 2017: resource-usage: origin=impstats
>    utime=72720000 stime=53436000 maxrss=225016 minflt=55854 majflt=0
>    inblock=10 oublock=712410 nvcsw=1124894 nivcsw=11014
>    Wed Jan 18 14:14:13 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 14:14:13 2017: relp: origin=core.queue size=0
>    enqueued=758670 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 14:14:13 2017: main Q: origin=core.queue size=0
>    enqueued=350 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>    Wed Jan 18 14:15:53 2017: global: origin=dynstats
>    Wed Jan 18 14:15:53 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:15:53 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 14:15:53 2017: json: origin=core.action processed=860750
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:15:53 2017: norm: origin=core.action processed=860750
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:15:53 2017: tofile: origin=core.action
>    processed=860750 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:15:53 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:15:53 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:15:53 2017: imrelp[20514]: origin=imrelp submitted=860750
>    Wed Jan 18 14:15:53 2017: resource-usage: origin=impstats
>    utime=81732000 stime=59492000 maxrss=254704 minflt=63240 majflt=0
>    inblock=10 oublock=808415 nvcsw=1249978 nivcsw=12209
>    Wed Jan 18 14:15:53 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 14:15:53 2017: relp: origin=core.queue size=0
>    enqueued=860750 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 14:15:53 2017: main Q: origin=core.queue size=0
>    enqueued=363 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>    Wed Jan 18 14:17:33 2017: global: origin=dynstats
>    Wed Jan 18 14:17:33 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:17:33 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 14:17:33 2017: json: origin=core.action processed=922024
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:17:33 2017: norm: origin=core.action processed=922024
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:17:33 2017: tofile: origin=core.action
>    processed=922024 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:17:33 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:17:33 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:17:33 2017: imrelp[20514]: origin=imrelp submitted=922024
>    Wed Jan 18 14:17:33 2017: resource-usage: origin=impstats
>    utime=87368000 stime=63144000 maxrss=272284 minflt=67663 majflt=0
>    inblock=10 oublock=866042 nvcsw=1328558 nivcsw=12754
>    Wed Jan 18 14:17:33 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 14:17:33 2017: relp: origin=core.queue size=0
>    enqueued=922024 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 14:17:33 2017: main Q: origin=core.queue size=0
>    enqueued=376 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>    Wed Jan 18 14:19:13 2017: global: origin=dynstats
>    Wed Jan 18 14:19:13 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:19:13 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 14:19:13 2017: json: origin=core.action processed=922072
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:19:13 2017: norm: origin=core.action processed=922072
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:19:13 2017: tofile: origin=core.action
>    processed=922072 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:19:13 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:19:13 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:19:13 2017: imrelp[20514]: origin=imrelp submitted=922072
>    Wed Jan 18 14:19:13 2017: resource-usage: origin=impstats
>    utime=87376000 stime=63148000 maxrss=272284 minflt=67665 majflt=0
>    inblock=10 oublock=866091 nvcsw=1328630 nivcsw=12754
>    Wed Jan 18 14:19:13 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 14:19:13 2017: relp: origin=core.queue size=0
>    enqueued=922072 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 14:19:13 2017: main Q: origin=core.queue size=0
>    enqueued=389 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>    Wed Jan 18 14:19:25 2017: global: origin=dynstats
>    Wed Jan 18 14:19:25 2017: action 0: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:19:25 2017: omelasticsearch: origin=omelasticsearch
>    submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
>    failed.es=0
>    Wed Jan 18 14:19:25 2017: json: origin=core.action processed=922073
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:19:25 2017: norm: origin=core.action processed=922073
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:19:25 2017: tofile: origin=core.action
>    processed=922073 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:19:25 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:19:25 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:19:25 2017: imrelp[20514]: origin=imrelp submitted=922073
>    Wed Jan 18 14:19:25 2017: resource-usage: origin=impstats
>    utime=87376000 stime=63148000 maxrss=272284 minflt=67665 majflt=0
>    inblock=10 oublock=866094 nvcsw=1328649 nivcsw=12754
>    Wed Jan 18 14:19:25 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 14:19:25 2017: relp: origin=core.queue size=0
>    enqueued=922073 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
>    Wed Jan 18 14:19:25 2017: main Q: origin=core.queue size=0
>    enqueued=402 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>
>
> *Enabling omelasticsearch with 100m/s, takes much more RAM, but RGROW
> increasing at the same ratio. Lokk like a leak to me. *
> Queue size, on the other hand, increases constantly (ie: omelasticsearch
> consumer isn't as fast at it could).
>
>    Wed Jan 18 14:39:46 2017: global: origin=dynstats
>    Wed Jan 18 14:39:46 2017: omelasticsearch: origin=omelasticsearch
>    submitted=23878 failed.http=0 failed.httprequests=0
>    failed.checkConn=0 failed.es=0
>    Wed Jan 18 14:39:46 2017: json: origin=core.action processed=24022
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:39:46 2017: norm: origin=core.action processed=24022
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:39:46 2017: elastic: origin=core.action
>    processed=24022 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:39:46 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:39:46 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:39:46 2017: imrelp[20514]: origin=imrelp submitted=27030
>    Wed Jan 18 14:39:46 2017: resource-usage: origin=impstats
>    utime=5392000 stime=3828000 maxrss=25372 minflt=5800 majflt=0
>    inblock=10 oublock=16 nvcsw=93927 nivcsw=2722
>    Wed Jan 18 14:39:46 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 14:39:46 2017: relp: origin=core.queue size=4032
>    enqueued=27030 full=0 discarded.full=0 discarded.nf=0 maxqsize=4032
>    Wed Jan 18 14:39:46 2017: main Q: origin=core.queue size=0
>    enqueued=11 full=0 discarded.full=0 discarded.nf=0 maxqsize=1
>    Wed Jan 18 14:41:34 2017: global: origin=dynstats
>    Wed Jan 18 14:41:34 2017: omelasticsearch: origin=omelasticsearch
>    submitted=24902 failed.http=0 failed.httprequests=0
>    failed.checkConn=0 failed.es=0
>    Wed Jan 18 14:41:34 2017: json: origin=core.action processed=24982
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:41:34 2017: norm: origin=core.action processed=24982
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:41:34 2017: elastic: origin=core.action
>    processed=24982 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:41:34 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:41:34 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:41:34 2017: imrelp[20514]: origin=imrelp submitted=38857
>    Wed Jan 18 14:41:34 2017: resource-usage: origin=impstats
>    utime=5672000 stime=4440000 maxrss=37752 minflt=8820 majflt=0
>    inblock=10 oublock=16 nvcsw=98378 nivcsw=2528
>    Wed Jan 18 14:41:34 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 14:41:34 2017: relp: origin=core.queue size=14899
>    enqueued=38857 full=0 discarded.full=0 discarded.nf=0 maxqsize=14996
>    Wed Jan 18 14:41:34 2017: main Q: origin=core.queue size=5
>    enqueued=11 full=0 discarded.full=0 discarded.nf=0 maxqsize=5
>    Wed Jan 18 14:43:15 2017: global: origin=dynstats
>    Wed Jan 18 14:43:15 2017: omelasticsearch: origin=omelasticsearch
>    submitted=53292 failed.http=0 failed.httprequests=0
>    failed.checkConn=0 failed.es=0
>    Wed Jan 18 14:43:15 2017: json: origin=core.action processed=53654
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:43:15 2017: norm: origin=core.action processed=53654
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:43:15 2017: elastic: origin=core.action
>    processed=53654 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:43:15 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:43:15 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:43:15 2017: imrelp[20514]: origin=imrelp submitted=75862
>    Wed Jan 18 14:43:15 2017: resource-usage: origin=impstats
>    utime=13348000 stime=9720000 maxrss=55176 minflt=13236 majflt=0
>    inblock=10 oublock=18 nvcsw=213285 nivcsw=5412
>    Wed Jan 18 14:43:15 2017: relp[DA]: origin=core.queue size=0
>    enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>    Wed Jan 18 14:43:15 2017: relp: origin=core.queue size=23232
>    enqueued=75862 full=0 discarded.full=0 discarded.nf=0 maxqsize=23375
>    Wed Jan 18 14:43:15 2017: main Q: origin=core.queue size=0
>    enqueued=23 full=0 discarded.full=0 discarded.nf=0 maxqsize=6
>    Wed Jan 18 14:44:55 2017: global: origin=dynstats
>    Wed Jan 18 14:44:55 2017: omelasticsearch: origin=omelasticsearch
>    submitted=80727 failed.http=0 failed.httprequests=0
>    failed.checkConn=0 failed.es=0
>    Wed Jan 18 14:44:55 2017: json: origin=core.action processed=81302
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:44:55 2017: norm: origin=core.action processed=81302
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:44:55 2017: elastic: origin=core.action
>    processed=81302 failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:44:55 2017: error: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:44:55 2017: unk: origin=core.action processed=0
>    failed=0 suspended=0 suspended.duration=0 resumed=0
>    Wed Jan 18 14:44:55 2017: imrelp[20514]: origin=imrelp submitted=109044


More information about the rsyslog mailing list