[rsyslog-notify] Forum Thread: messages lost due to rate-limiting - (Mode 'edit_topic')
noreply at adiscon.com
noreply at adiscon.com
Wed Sep 7 16:30:07 CEST 2016
User: uppsalanet
Forumlink: http://kb.monitorware.com/viewtopic.php?p=26927#p26927
Message:
----------
I've problem to sort this out, where is the problem?
[i]rsyslogd-2177: rsyslogd[internal_messages]: 179 messages lost due to
rate-limiting[/i]
This i our version:
[code]rsyslogd 8.21.0, compiled with:
PLATFORM: x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64[/code]
We have a forward:
[code]ruleset(name="Inbound-udp") {
action(type="omfile"
DynaFile="TmplMsgUdp"
template="RSYSLOG_TraditionalFileFormat"
)
action (type="omfwd"
template="fwdRSYS"
protocol="tcp"
target="syslog.home.com"
port="2514"
StreamDriver="gtls"
StreamDriverMode="1"
StreamDriverAuthMode="x509/name"
StreamDriverPermittedPeers="syslog.home.com"
queue.spoolDirectory="/var/spool/rsyslog"
queue.filename="forwarding"
queue.size="1000000"
queue.type="LinkedList"
)
}[/code]
impstat
[code]Wed Sep 7 16:22:52 2016: global: origin=dynstats
Wed Sep 7 16:22:52 2016: imuxsock: origin=imuxsock submitted=0
ratelimit.discarded=0 ratelimit.numratelimiters=0
Wed Sep 7 16:22:52 2016: action 0: origin=core.action processed=912
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: action 1: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: action 2: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: action 3: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: action 4: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: action 5: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: action 6: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: dynafile cache TmplMsg: origin=omfile
requests=25613 level0=20304 missed=316 evicted=316 maxused=0
closetimeouts=0
Wed Sep 7 16:22:52 2016: action 7: origin=core.action processed=25613
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: action 8: origin=core.action processed=25613
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: dynafile cache TmplMsgUdp: origin=omfile
requests=15197 level0=8080 missed=1474 evicted=1474 maxused=0
closetimeouts=0
Wed Sep 7 16:22:52 2016: action 9: origin=core.action processed=15197
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: action 10: origin=core.action processed=15197
failed=0 suspended=0 suspended.duration=720 resumed=0
Wed Sep 7 16:22:52 2016: dynafile cache TmplMsgTsl: origin=omfile
requests=0 level0=0 missed=0 evicted=0 maxused=0 closetimeouts=0
Wed Sep 7 16:22:52 2016: action 11: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: dynafile cache TmplMsgRelp: origin=omfile
requests=0 level0=0 missed=0 evicted=0 maxused=0 closetimeouts=0
Wed Sep 7 16:22:52 2016: action 12: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: imudp(*:514): origin=imudp submitted=14777
Wed Sep 7 16:22:52 2016: imudp(*:514): origin=imudp submitted=436
Wed Sep 7 16:22:52 2016: imptcp(*/11514/IPv4): origin=imptcp
submitted=15611 bytes.received=9435982 bytes.decompressed=0
Wed Sep 7 16:22:52 2016: imptcp(*/11514/IPv6): origin=imptcp
submitted=10012 bytes.received=1717990 bytes.decompressed=0
Wed Sep 7 16:22:52 2016: imrelp[2514]: origin=imrelp submitted=0
Wed Sep 7 16:22:52 2016: imtcp(10514): origin=imtcp submitted=0
Wed Sep 7 16:22:52 2016: resource-usage: origin=impstats utime=293100607
stime=32012553 maxrss=41904 minflt=86742 majflt=0 inblock=0 oublock=1639496
nvcsw=2700512 nivcsw=348
Wed Sep 7 16:22:52 2016: action 10 queue[DA]: origin=core.queue size=0
enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Sep 7 16:22:52 2016: action 10 queue: origin=core.queue size=0
enqueued=15197 full=0 discarded.full=0 discarded.nf=0 maxqsize=920
Wed Sep 7 16:22:52 2016: main Q[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Sep 7 16:22:52 2016: main Q: origin=core.queue size=40 enqueued=41747
full=0 discarded.full=0 discarded.nf=0 maxqsize=24532
Wed Sep 7 16:22:52 2016: io-work-q: origin=imptcp enqueued=253 maxqsize=3
Wed Sep 7 16:22:52 2016: imudp(w0): origin=imudp called.recvmmsg=24377
called.recvmsg=0 msgs.received=15213
Wed Sep 7 16:23:02 2016: global: origin=dynstats
Wed Sep 7 16:23:02 2016: imuxsock: origin=imuxsock submitted=0
ratelimit.discarded=0 ratelimit.numratelimiters=0
Wed Sep 7 16:23:02 2016: action 0: origin=core.action processed=890
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: action 1: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: action 2: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: action 3: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: action 4: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: action 5: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: action 6: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: dynafile cache TmplMsg: origin=omfile
requests=19679 level0=15390 missed=358 evicted=358 maxused=0
closetimeouts=0
Wed Sep 7 16:23:02 2016: action 7: origin=core.action processed=19679
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: action 8: origin=core.action processed=19679
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: dynafile cache TmplMsgUdp: origin=omfile
requests=16037 level0=7837 missed=1696 evicted=1696 maxused=0
closetimeouts=0
Wed Sep 7 16:23:02 2016: action 9: origin=core.action processed=16037
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: action 10: origin=core.action processed=16037
failed=0 suspended=0 suspended.duration=720 resumed=0
Wed Sep 7 16:23:02 2016: dynafile cache TmplMsgTsl: origin=omfile
requests=0 level0=0 missed=0 evicted=0 maxused=0 closetimeouts=0
Wed Sep 7 16:23:02 2016: action 11: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: dynafile cache TmplMsgRelp: origin=omfile
requests=0 level0=0 missed=0 evicted=0 maxused=0 closetimeouts=0
Wed Sep 7 16:23:02 2016: action 12: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: imudp(*:514): origin=imudp submitted=15739
Wed Sep 7 16:23:02 2016: imudp(*:514): origin=imudp submitted=277
Wed Sep 7 16:23:02 2016: imptcp(*/11514/IPv4): origin=imptcp
submitted=9560 bytes.received=7499229 bytes.decompressed=0
Wed Sep 7 16:23:02 2016: imptcp(*/11514/IPv6): origin=imptcp
submitted=10108 bytes.received=1713842 bytes.decompressed=0
Wed Sep 7 16:23:02 2016: imrelp[2514]: origin=imrelp submitted=0
Wed Sep 7 16:23:02 2016: imtcp(10514): origin=imtcp submitted=0
Wed Sep 7 16:23:02 2016: resource-usage: origin=impstats utime=298870252
stime=32462689 maxrss=41904 minflt=87335 majflt=0 inblock=0 oublock=1660968
nvcsw=2741261 nivcsw=356
Wed Sep 7 16:23:02 2016: action 10 queue[DA]: origin=core.queue size=0
enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Sep 7 16:23:02 2016: action 10 queue: origin=core.queue size=0
enqueued=16037 full=0 discarded.full=0 discarded.nf=0 maxqsize=920
Wed Sep 7 16:23:02 2016: main Q[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Sep 7 16:23:02 2016: main Q: origin=core.queue size=9 enqueued=36575
full=0 discarded.full=0 discarded.nf=0 maxqsize=24532
Wed Sep 7 16:23:02 2016: io-work-q: origin=imptcp enqueued=201 maxqsize=3
Wed Sep 7 16:23:02 2016: imudp(w0): origin=imudp called.recvmmsg=25591
called.recvmsg=0 msgs.received=16016
[/code]
[b]Here is the problem: /var/log/messages[/b]
[code]
2016-09-07T16:15:05.582770+02:00 its-srv015 rsyslogd-2359: action 'action
8' resumed (module 'builtin:omfwd') [v8.21.0 try
http://www.rsyslog.com/e/2359 ]
2016-09-07T16:15:05.582804+02:00 its-srv015 rsyslogd-2359: action 'action
8' resumed (module 'builtin:omfwd') [v8.21.0 try
http://www.rsyslog.com/e/2359 ]
2016-09-07T16:15:05.584951+02:00 its-srv015 rsyslogd-2078: unexpected
GnuTLS error -53 - this could be caused by a broken connection. GnuTLS
reports: Error in the push function. [v8.21.0 try
http://www.rsyslog.com/e/2078 ]
2016-09-07T16:15:05.602471+02:00 its-srv015 rsyslogd-2359: action 'action
8' resumed (module 'builtin:omfwd') [v8.21.0 try
http://www.rsyslog.com/e/2359 ]
2016-09-07T16:15:05.602496+02:00 its-srv015 rsyslogd-2359: action 'action
8' resumed (module 'builtin:omfwd') [v8.21.0 try
http://www.rsyslog.com/e/2359 ]
2016-09-07T16:15:05.603702+02:00 its-srv015 rsyslogd-2078: unexpected
GnuTLS error -53 - this could be caused by a broken connection. GnuTLS
reports: Error in the push function. [v8.21.0 try
http://www.rsyslog.com/e/2078 ]
2016-09-07T16:15:05.621486+02:00 its-srv015 rsyslogd-2359: action 'action
8' resumed (module 'builtin:omfwd') [v8.21.0 try
http://www.rsyslog.com/e/2359 ]
2016-09-07T16:15:05.621521+02:00 its-srv015 rsyslogd-2359: action 'action
8' resumed (module 'builtin:omfwd') [v8.21.0 try
http://www.rsyslog.com/e/2359 ]
2016-09-07T16:15:05.623165+02:00 its-srv015 rsyslogd-2078: unexpected
GnuTLS error -53 - this could be caused by a broken connection. GnuTLS
reports: Error in the push function.
[/code]
And hide in above logs
[b]tail -100f messages| grep rate[/b]
[code]2016-09-07T16:15:30.012119+02:00 its-srv015 rsyslogd-2177:
rsyslogd[internal_messages]: 163 messages lost due to rate-limiting
2016-09-07T16:16:00.005651+02:00 its-srv015 rsyslogd-2177:
rsyslogd[internal_messages]: 4 messages lost due to rate-limiting
2016-09-07T16:16:06.003981+02:00 its-srv015 rsyslogd-2177:
rsyslogd[internal_messages]: 135 messages lost due to rate-limiting
2016-09-07T16:16:12.146655+02:00 its-srv015 rsyslogd-2177:
rsyslogd[internal_messages]: 4 messages lost due to rate-limiting
2016-09-07T16:16:18.011937+02:00 its-srv015 rsyslogd-2177:
rsyslogd[internal_messages]: 208 messages lost due to rate-limiting
2016-09-07T16:16:42.008406+02:00 its-srv015 rsyslogd-2177:
rsyslogd[internal_messages]: 197 messages lost due to rate-limiting
2016-09-07T16:17:00.009581+02:00 its-srv015 rsyslogd-2177:
rsyslogd[internal_messages]: 179 messages lost due to rate-limiting
2016-09-07T16:17:42.009426+02:00 its-srv015 rsyslogd-2177:
rsyslogd[internal_messages]: 16 messages lost due to rate-limiting[/code]
More information about the rsyslog-notify
mailing list