[rsyslog] computer hang-up and WorkDirectory

[david at lang.hm]

On Fri, 6 Nov 2009, Miguel Angel Nieto wrote:

>> in other cases you are willing to loose logs rather than freezing the
>> machine and can configure rsyslog to accept messages, even when it can't
>> do anything with them to avoid this sort of lockup.
>
> How can I do to tell rsyslog to accept all messages and not use any queue?

you cannot tell rsyslog to not use a queue.

the fundamental architecture of rsyslog is that the input modules receive 
messages, parse minimal info out of them, and put them in a queue.worker 
threads then pull messages from this queue and send them to their 
destination. this queue can be in ram, ram + disk file, or disk-only

classic syslog doesn't use any queue and must fully process each message 
before receiving the next message. syslog-ng has the ability to delay 
writing the logs to disk a bit to do them in batches, but otherwise is the 
same.

in rsyslog, significantly more time is spent in the output side of things 
than in the input side.


historicly, syslog was intended to be a reliable logging mechanism, so 
applications do not complete their write until the syslog daemon has the 
log safe on disk. the performance of doing this is so horrible that 
everybody has some way of relaxing this requirement (the - option in 
sysklogd, the batch option in syslog-ng, the memory queue in rsyslog), but 
all of these options only allow a finite number of items to be buffered 
before the syslog daemon will stop to wait for them to really get to the 
destination.

this is why your systems locked up, they were trying to write to rsyslog 
on the client, rsyslog on the client was configured to not consider the 
logs processed until they were acknowledged by the TCP stack of the 
server. so when the server is not available the clients keep accepting 
messages and queue them up, but when the queue filled up they stopped 
accepting new messages.

the reason to use TCP for syslog instead of UDP is so that you can have 
the log server push back on the sender and tell the sender that it can't 
process the log message right now, the sender should hang on to it and 
tray again to send it later. if you are really willing to loose logs if 
the log server is down or backed up, why not just use UDP for the logs?

look at the MainMsgQueue* options for setting high watermark, low 
watermark, discard options, etc if you want to change how rsyslog acts 
when the queue fills up. I haven't used these, so all I can do is to point 
you in the right direction.

David Lang