[rsyslog-notify] Forum Thread: Re: RSyslog not sending messages - (Mode 'reply')

[noreply at adiscon.com]

User: lethalduck 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=25186#p25186

Message: 
----------
Config attached. Using TCP over TLS.

[quote:3vqq2xsw]you could be loosing messages when a firewall between the
sender and receiver cuts the connection an forces rsyslog to re-establish
it[/quote:3vqq2xsw]

Any idea why a router would cut the connection? I haven't seen anything
like this in my logs. Normally once a TCP connection is established there
is no reason for a router to block it. Sometimes it goes OK for a day,
sometimes half a day, sometimes 2 days. It looks like the
[code:3vqq2xsw]action 'action 18' resumed (module
'builtin:omfwd')[/code:3vqq2xsw]  happens immediately after the failed
messages in most (not all) cases.

If a TCP packet is sent and no acknowledgement is received by the sender,
the same packet will be re-transmitted right? This doesn't appear to be
happening. This looks like a fault with rsyslog's implementation?

[quote:3vqq2xsw]If you are using UDP to send messags (@IP), then messages
will be lost if the network, router/firewall, or receiver run out of
capacity.[/quote:3vqq2xsw]

TCP not UDP in this case. Pretty sure there are no capacity problems. Each
day produces about 260 lines of logging. That's a little over 1 message per
10 minutes. Of course this will change when I get DDoSed and that's a
concern.

[quote:3vqq2xsw]If the queues are getting overloaded and you have rsyslog
configured with watermark settings, that tells rsyslog to throw away
messages when too many are waiting to be sent.[/quote:3vqq2xsw]

I can't see how a couple of messages would overload a queue unless I've
really misunderstood the configuration? The volume of messages in the
images are typical. So very low volume. Surely this wouldn't overload
anything?

[quote:3vqq2xsw]you would need to be using RELP to avoid loosing messages
due to network hiccups, because the RELP protocol adds application level
acknowlegements so that the sender knows for sure that the receiver has the
message.[/quote:3vqq2xsw]

I've heard this quite a bit, but don't really understand it. TCP is
supposed to be reliable. If a message is sent and an ACK isn't received,
then the sender will just keep sending. This doesn't appear to be
happening. Why do we need application layer acknowlegements when we have
transport layer acknowlegements? It seems that the application may not be
communicating effectively with the transport. Does that sound right?

[quote:3vqq2xsw]If you aren't running impstats in your config, you should
look into adding it. The stats it produces show when you run into errors
with an output, and also the state of the queues.[/quote:3vqq2xsw]

Cool. Thanks for that advice. I'll have a look into impstats and configure
it.

What I'm also noticing is that the majority of the missing messages are
occurring before the [code:3vqq2xsw]action 18[/code:3vqq2xsw] which occures
before the nn:17 cron.hourly, but strangely there is a gap of at least 10
minutes. We get the nn:55 then no nn:05

Also see the collection (attached) of [code:3vqq2xsw]action
18[/code:3vqq2xsw] messages that only seem to be occurring right before the
cron.hourly.

Thanks for your help again and hopefully we get to the bottom of it.