[rsyslog] spool files not clearing

David Gloe dgloe at cray.com
Thu May 4 17:53:47 CEST 2017


Hello,

On one of our servers, rsyslog began spooling log messages that should have been written to a local file.
After restarting rsyslog, new messages go to the file as we expect, but the spooled messages never get cleared out.
We now have 11,189 spool files that we’re not sure how to save.

Here’s the configuration for that rule:
$RuleSet rule.controllers.syslog
$MainMsgQueueFileName controllers_syslog
$MainMsgQueueDiscardMark 880000
$MainMsgQueueHighWatermark 660000
$RulesetCreateMainQueue on
$IncludeConfig /var/spool/rsyslog/local-rules/always.conf
$IncludeConfig /var/spool/rsyslog/local-rules/rule.controllers.syslog.conf
$IncludeConfig /var/spool/rsyslog/rsyslog-forward.conf
$IncludeConfig /var/spool/rsyslog/rsyslog-options.conf
if $syslogseverity <= '3' and strlen($msgid) > 1 then -?file-urgent-sessionid;format-rfc5424
if $syslogseverity <= '3' and strlen($msgid) < 2 then -?file-urgent;format-rfc5424
if $app-name startswith 'hss_' then -?file-controller-alt;format-rfc5424_ctrlr_emu
& stop
if $structured-data startswith '[hss_' then -?file-controller-llm;format-rfc5424
& stop
*.* -?file-controller-syslog;format-rfc5424
$InputPTCPServerBindRuleset rule.controllers.syslog
$InputPTCPServerRun 5171
$InputUDPServerBindRuleset rule.controllers.syslog
$UDPServerRun 5171

I see this in the logs:
<46>1 2017-05-04T10:04:24.928391-05:00 cst79-1 rsyslogd-pstats - - - dynafile cache file-controller-syslog: requests=3281489 lev
el0=2004734 missed=26682 evicted=26672 maxused=10 closetimeouts=0
<46>1 2017-05-04T10:04:24.928394-05:00 cst79-1 rsyslogd-pstats - - - action 67: processed=3281489 failed=0 suspended=0 suspended.duration=0 resumed=0

The processed number just keeps increasing. I don’t see any errors in the rsyslog log.

Seems like it thinks that it’s processing the spooled logs but they don’t go away?
Any idea on how to debug the issue? I tried running rsyslog -dn but didn’t discover anything from that.

Thanks,
David


More information about the rsyslog mailing list