[rsyslog] rsyslogd 'clients' take out machines if rsyslogd 'server' is down

[Philip M. Gollucci]

rsyslog.conf [client version]:
-------------------------------

....

$WorkDirectory /var/spool/rsyslog # location for work (spool) file
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

....
## Local
....

## Remote
*.* :omrelp:cl.tld:2514


If anything on the 'centralized server' fails like mysql is down, or 
rsyslogd is down, or a table is wrong, or insert permissions are wrong, 
or diskio is *really* *really* bad, or the box is 'off' --- about 45 
minutes later. All 50 or so of the clients talking to 'block' and 
suddenly spin in a tight loop and are not killable without -9.

I'm sure its something filling up b/c the timing is nearly to the second 
of the 45 minute mark [given a relatively constant amount of load].

I assume that changing
$ActionResumeRetryCount -1
to
$ActionResumeRetryCount 5

or some other arbirtarily low number in all clients would fix this.  As 
whatever ram buffers are filling up will stop doing that.  If not, maybe 
falling back from RELP to TCP would do the trick [I'd be okay with 
message loss to prevent downtime].

I'm rather more concerned though that it doesn't start queueing to disk 
the spool dir when the ram buffers are full instead of 'crashing' and 
taking the box or all process on it with it.



2542.938673679:800b021c0: rsyslogd 5.3.7 startup, compatibility mode 5, 
module path ''

on FreeBSD 8.0-RELENG-p2








-- 
------------------------------------------------------------------------
1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70  3F8C 75B8 8FFB DB9B 8C1C
Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354
VP Apache Infrastructure; Member, Apache Software Foundation
Committer,                        FreeBSD Foundation
Consultant,                       P6M7G8 Inc.
Sr. System Admin,                 Ridecharge Inc.

Work like you don't need the money,
love like you'll never get hurt,
and dance like nobody's watching.