[rsyslog-notify] Forum Thread: messages lost due to rate-limiting - (Mode 'edit_topic')
noreply at adiscon.com
noreply at adiscon.com
Wed Sep 7 16:31:05 CEST 2016
User: uppsalanet
Forumlink: http://kb.monitorware.com/viewtopic.php?p=26927#p26927
Message:
----------
I've problem to sort this out, where is the problem?
[i:3tj6yvg0]rsyslogd-2177: rsyslogd[internal_messages]: 179 messages lost
due to rate-limiting[/i:3tj6yvg0]
This i our version:
[code:3tj6yvg0]rsyslogd 8.21.0, compiled with:
PLATFORM: x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64[/code:3tj6yvg0]
We have a forward and inbound:
[code:3tj6yvg0]ruleset(name="Inbound-udp") {
action(type="omfile"
DynaFile="TmplMsgUdp"
template="RSYSLOG_TraditionalFileFormat"
)
action (type="omfwd"
template="fwdRSYS"
protocol="tcp"
target="syslog.home.com"
port="2514"
StreamDriver="gtls"
StreamDriverMode="1"
StreamDriverAuthMode="x509/name"
StreamDriverPermittedPeers="syslog.home.com"
queue.spoolDirectory="/var/spool/rsyslog"
queue.filename="forwarding"
queue.size="1000000"
queue.type="LinkedList"
)
}[/code:3tj6yvg0]
impstat
[code:3tj6yvg0]Wed Sep 7 16:22:52 2016: global:
origin=dynstats
Wed Sep 7 16:22:52 2016: imuxsock: origin=imuxsock
submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
Wed Sep 7 16:22:52 2016: action 0: origin=core.action
processed=912 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: action 1: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: action 2: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: action 3: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: action 4: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: action 5: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: action 6: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: dynafile cache TmplMsg:
origin=omfile requests=25613 level0=20304 missed=316 evicted=316 maxused=0
closetimeouts=0
Wed Sep 7 16:22:52 2016: action 7: origin=core.action
processed=25613 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: action 8: origin=core.action
processed=25613 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: dynafile cache TmplMsgUdp:
origin=omfile requests=15197 level0=8080 missed=1474 evicted=1474 maxused=0
closetimeouts=0
Wed Sep 7 16:22:52 2016: action 9: origin=core.action
processed=15197 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: action 10: origin=core.action
processed=15197 failed=0 suspended=0 suspended.duration=720 resumed=0
Wed Sep 7 16:22:52 2016: dynafile cache TmplMsgTsl:
origin=omfile requests=0 level0=0 missed=0 evicted=0 maxused=0
closetimeouts=0
Wed Sep 7 16:22:52 2016: action 11: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: dynafile cache TmplMsgRelp:
origin=omfile requests=0 level0=0 missed=0 evicted=0 maxused=0
closetimeouts=0
Wed Sep 7 16:22:52 2016: action 12: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:22:52 2016: imudp(*:514): origin=imudp
submitted=14777
Wed Sep 7 16:22:52 2016: imudp(*:514): origin=imudp
submitted=436
Wed Sep 7 16:22:52 2016: imptcp(*/11514/IPv4):
origin=imptcp submitted=15611 bytes.received=9435982
bytes.decompressed=0
Wed Sep 7 16:22:52 2016: imptcp(*/11514/IPv6):
origin=imptcp submitted=10012 bytes.received=1717990
bytes.decompressed=0
Wed Sep 7 16:22:52 2016: imrelp[2514]:
origin=imrelp submitted=0
Wed Sep 7 16:22:52 2016: imtcp(10514): origin=imtcp
submitted=0
Wed Sep 7 16:22:52 2016: resource-usage: origin=impstats
utime=293100607 stime=32012553 maxrss=41904 minflt=86742 majflt=0 inblock=0
oublock=1639496 nvcsw=2700512 nivcsw=348
Wed Sep 7 16:22:52 2016: action 10 queue[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0
discarded.nf=0 maxqsize=0
Wed Sep 7 16:22:52 2016: action 10 queue:
origin=core.queue size=0 enqueued=15197 full=0 discarded.full=0
discarded.nf=0 maxqsize=920
Wed Sep 7 16:22:52 2016: main Q[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0
discarded.nf=0 maxqsize=0
Wed Sep 7 16:22:52 2016: main Q: origin=core.queue
size=40 enqueued=41747 full=0 discarded.full=0 discarded.nf=0
maxqsize=24532
Wed Sep 7 16:22:52 2016: io-work-q: origin=imptcp
enqueued=253 maxqsize=3
Wed Sep 7 16:22:52 2016: imudp(w0): origin=imudp
called.recvmmsg=24377 called.recvmsg=0 msgs.received=15213
Wed Sep 7 16:23:02 2016: global: origin=dynstats
Wed Sep 7 16:23:02 2016: imuxsock: origin=imuxsock
submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
Wed Sep 7 16:23:02 2016: action 0: origin=core.action
processed=890 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: action 1: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: action 2: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: action 3: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: action 4: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: action 5: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: action 6: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: dynafile cache TmplMsg:
origin=omfile requests=19679 level0=15390 missed=358 evicted=358 maxused=0
closetimeouts=0
Wed Sep 7 16:23:02 2016: action 7: origin=core.action
processed=19679 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: action 8: origin=core.action
processed=19679 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: dynafile cache TmplMsgUdp:
origin=omfile requests=16037 level0=7837 missed=1696 evicted=1696 maxused=0
closetimeouts=0
Wed Sep 7 16:23:02 2016: action 9: origin=core.action
processed=16037 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: action 10: origin=core.action
processed=16037 failed=0 suspended=0 suspended.duration=720 resumed=0
Wed Sep 7 16:23:02 2016: dynafile cache TmplMsgTsl:
origin=omfile requests=0 level0=0 missed=0 evicted=0 maxused=0
closetimeouts=0
Wed Sep 7 16:23:02 2016: action 11: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: dynafile cache TmplMsgRelp:
origin=omfile requests=0 level0=0 missed=0 evicted=0 maxused=0
closetimeouts=0
Wed Sep 7 16:23:02 2016: action 12: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Sep 7 16:23:02 2016: imudp(*:514): origin=imudp
submitted=15739
Wed Sep 7 16:23:02 2016: imudp(*:514): origin=imudp
submitted=277
Wed Sep 7 16:23:02 2016: imptcp(*/11514/IPv4):
origin=imptcp submitted=9560 bytes.received=7499229
bytes.decompressed=0
Wed Sep 7 16:23:02 2016: imptcp(*/11514/IPv6):
origin=imptcp submitted=10108 bytes.received=1713842
bytes.decompressed=0
Wed Sep 7 16:23:02 2016: imrelp[2514]:
origin=imrelp submitted=0
Wed Sep 7 16:23:02 2016: imtcp(10514): origin=imtcp
submitted=0
Wed Sep 7 16:23:02 2016: resource-usage: origin=impstats
utime=298870252 stime=32462689 maxrss=41904 minflt=87335 majflt=0 inblock=0
oublock=1660968 nvcsw=2741261 nivcsw=356
Wed Sep 7 16:23:02 2016: action 10 queue[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0
discarded.nf=0 maxqsize=0
Wed Sep 7 16:23:02 2016: action 10 queue:
origin=core.queue size=0 enqueued=16037 full=0 discarded.full=0
discarded.nf=0 maxqsize=920
Wed Sep 7 16:23:02 2016: main Q[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0
discarded.nf=0 maxqsize=0
Wed Sep 7 16:23:02 2016: main Q: origin=core.queue
size=9 enqueued=36575 full=0 discarded.full=0 discarded.nf=0
maxqsize=24532
Wed Sep 7 16:23:02 2016: io-work-q: origin=imptcp
enqueued=201 maxqsize=3
Wed Sep 7 16:23:02 2016: imudp(w0): origin=imudp
called.recvmmsg=25591 called.recvmsg=0 msgs.received=16016
[/code:3tj6yvg0]
[b:3tj6yvg0]Here is the problem: /var/log/messages[/b:3tj6yvg0]
[code:3tj6yvg0]
2016-09-07T16:15:05.582770+02:00 its-srv015
rsyslogd-2359: action 'action 8' resumed (module 'builtin:omfwd')
[v8.21.0 try http://www.rsyslog.com/e/2359 ]
2016-09-07T16:15:05.582804+02:00 its-srv015
rsyslogd-2359: action 'action 8' resumed (module 'builtin:omfwd')
[v8.21.0 try http://www.rsyslog.com/e/2359 ]
2016-09-07T16:15:05.584951+02:00 its-srv015
rsyslogd-2078: unexpected GnuTLS error -53 - this could be caused by a
broken connection. GnuTLS reports: Error in the push function.
[v8.21.0 try http://www.rsyslog.com/e/2078 ]
2016-09-07T16:15:05.602471+02:00 its-srv015
rsyslogd-2359: action 'action 8' resumed (module 'builtin:omfwd')
[v8.21.0 try http://www.rsyslog.com/e/2359 ]
2016-09-07T16:15:05.602496+02:00 its-srv015
rsyslogd-2359: action 'action 8' resumed (module 'builtin:omfwd')
[v8.21.0 try http://www.rsyslog.com/e/2359 ]
2016-09-07T16:15:05.603702+02:00 its-srv015
rsyslogd-2078: unexpected GnuTLS error -53 - this could be caused by a
broken connection. GnuTLS reports: Error in the push function.
[v8.21.0 try http://www.rsyslog.com/e/2078 ]
2016-09-07T16:15:05.621486+02:00 its-srv015
rsyslogd-2359: action 'action 8' resumed (module 'builtin:omfwd')
[v8.21.0 try http://www.rsyslog.com/e/2359 ]
2016-09-07T16:15:05.621521+02:00 its-srv015
rsyslogd-2359: action 'action 8' resumed (module 'builtin:omfwd')
[v8.21.0 try http://www.rsyslog.com/e/2359 ]
2016-09-07T16:15:05.623165+02:00 its-srv015
rsyslogd-2078: unexpected GnuTLS error -53 - this could be caused by a
broken connection. GnuTLS reports: Error in the push function.
[/code:3tj6yvg0]
And hide in above logs
[b:3tj6yvg0]tail -100f messages| grep rate[/b:3tj6yvg0]
[code:3tj6yvg0]2016-09-07T16:15:30.012119+02:00 its-srv015
rsyslogd-2177: rsyslogd[internal_messages]: 163 messages
lost due to rate-limiting
2016-09-07T16:16:00.005651+02:00 its-srv015
rsyslogd-2177: rsyslogd[internal_messages]: 4 messages lost
due to rate-limiting
2016-09-07T16:16:06.003981+02:00 its-srv015
rsyslogd-2177: rsyslogd[internal_messages]: 135 messages
lost due to rate-limiting
2016-09-07T16:16:12.146655+02:00 its-srv015
rsyslogd-2177: rsyslogd[internal_messages]: 4 messages lost
due to rate-limiting
2016-09-07T16:16:18.011937+02:00 its-srv015
rsyslogd-2177: rsyslogd[internal_messages]: 208 messages
lost due to rate-limiting
2016-09-07T16:16:42.008406+02:00 its-srv015
rsyslogd-2177: rsyslogd[internal_messages]: 197 messages
lost due to rate-limiting
2016-09-07T16:17:00.009581+02:00 its-srv015
rsyslogd-2177: rsyslogd[internal_messages]: 179 messages
lost due to rate-limiting
2016-09-07T16:17:42.009426+02:00 its-srv015
rsyslogd-2177: rsyslogd[internal_messages]: 16 messages
lost due to rate-limiting[/code:3tj6yvg0]
More information about the rsyslog-notify
mailing list