[rsyslog-notify] Forum Thread: Re: Freezes the server when rsyslog can't send logs over TCP - (Mode 'reply')

[noreply at adiscon.com]

User: dlang 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=25890#p25890

Message: 
----------
short version, rsyslog is doing exactly what you told it to do.

What you told it to do is to forward all logs to the remote system, and if
it can't reach the remote system, hold on to all logs until it comes back.

When the queue then fills up, it can no longer accept new log messages, and
so the thing trying to write the log message does what it's supposed to do,
it waits for rsyslog to accept the message.

As a result, the server stops.

you can make it last longer before it stops by making the memory queue
larger, or configuring it to use a disk assisted queue (storing the logs it
can't send on disk), but eventually you will run out of memory/disk space.

you can also make the receiving system highly available to reduce the
amount of downtime, but that doesn't help if there are network problems,
and even HA systems go down sometime.

So you need to decide which you would rather have happen when things go
wrong badly enough:

1. lose some logs

2. have the server stop

99%+ of people would rather loose some logs :-)

The simplist thing to do is to forward logs via UDP instead of TCP and the
sending system won't care that nothing is receiving the logs. If you have a
pair of HA boxes receiving the logs, you will loose very few.

You can also configure monitoring to alert you when the queues get large
(look at the impstats module and what it reports, it does a lot more on
newer versions, 5.8.10 is ancient, current is 8.12)

And (at least on newer versions), you can configure rsyslog to start
dropping some messages when the queues get too full.