[rsyslog-notify] Forum Thread: rsyslog has stopped logging after midnight after moving logs - (Mode 'post')

[noreply at adiscon.com]

User: QuietLeni 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=26896#p26896

Message: 
----------
Dear All,

We ran out of space on the disk that hosts /var/logs folder the other day
on our remote rsyslog server, so I moved the log files to the /home/hosts
directory (500GB) and set the permissions on the directry to 770. rsyslog
was logging nicely until midnight, but then, after midnight, it stopped
logging at all. I have tried various methods to work out what is going
wrong, but I cannot see the problem. We need to have a long retention time
on the logs at the moment and so, to help with rotation, I am splitting
them up into days and then delete all of the old days that are over 120
days old. The system is a Centos 7 box (3.10.0-229.el7.x86_64) with
rsyslogd  7.4.7.

The rsyslog.conf is:

[code:2yqml3cp]# Modules

$ModLoad imtcp
$ModLoad imudp
$ModLoad imuxsock
$ModLoad imklog

# Templates

$umask 0000
$FileCreateMode 0740
$DirCreateMode 0750
$fileOwner root
$fileGroup syslogs
$dirOwner root
$dirGroup syslogs

# log every host in its own directory
$template
RemoteHost,"/home/hosts/%$YEAR%%$MONTH%%$DAY%/%FROMHOST-IP%/%FROMHOST%/host.log"

### Rulesets

# Local Logging
$RuleSet local
kern.*                                                
/var/log/messages
*.info;mail.none;authpriv.none;cron.none               
/var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                 
-/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 *
uucp,news.crit                                         
/var/log/spooler
local7.*                                               
/var/log/boot.log
# use the local RuleSet as default if not specified otherwise
$DefaultRuleset local

# Remote Logging
$RuleSet remote
*.* ?RemoteHost

### Listeners

# bind ruleset to tcp listener
$InputTCPServerBindRuleset remote
# and activate it:
$InputTCPServerRun 514

$InputUDPServerBindRuleset remote
$UDPServerRun 514
[/code:2yqml3cp]

I enabled debugging with:

[code:2yqml3cp]export RSYSLOG_DEBUGLOG="/tmp/debuglog"
export RSYSLOG_DEBUG="Debug"
service rsyslog stop
rsyslogd -d | head -10[/code:2yqml3cp]

And got this:
[code:2yqml3cp]Stack now 0
Entering state 1
Next token is token PRIFILT ()
Shifting token PRIFILT ()
Entering state 14
Reading a token: Next token is token LEGACY_ACTION ()
Shifting token LEGACY_ACTION ()
Entering state 12
Reducing stack by rule 35 (line 169):
   $1 = token LEGACY_ACTION ()
-> $$ = nterm s_act ()
Stack now 0 1 14
Entering state 22
Reducing stack by rule 32 (line 165):
   $1 = nterm s_act ()
-> $$ = nterm actlst ()
Stack now 0 1 14
Entering state 21
Reading a token: Next token is token LEGACY_RULESET ()
Reducing stack by rule 23 (line 150):
   $1 = nterm actlst ()
-> $$ = nterm stmt ()
Stack now 0 1 14
Entering state 31
Reducing stack by rule 30 (line 163):
   $1 = nterm stmt ()
-> $$ = nterm block ()
Stack now 0 1 14
Entering state 32
Reducing stack by rule 28 (line 161):
   $1 = token PRIFILT ()
   $2 = nterm block ()
-> $$ = nterm stmt ()
Stack now 0 1
Entering state 20
Reducing stack by rule 3 (line 124):
   $1 = nterm conf ()
   $2 = nterm stmt ()
-> $$ = nterm conf ()
Stack now 0
Entering state 1
Next token is token LEGACY_RULESET ()
Shifting token LEGACY_RULESET ()
Entering state 13
Reducing stack by rule 4 (line 125):
   $1 = nterm conf ()
   $2 = token LEGACY_RULESET ()
-> $$ = nterm conf ()
Stack now 0
Entering state 1
Reading a token: Next token is token PRIFILT ()
Shifting token PRIFILT ()
Entering state 14
Reading a token: Next token is token LEGACY_ACTION ()
Shifting token LEGACY_ACTION ()
Entering state 12
Reducing stack by rule 35 (line 169):
   $1 = token LEGACY_ACTION ()
-> $$ = nterm s_act ()
Stack now 0 1 14
Entering state 22
Reducing stack by rule 32 (line 165):
   $1 = nterm s_act ()
-> $$ = nterm actlst ()
Stack now 0 1 14
Entering state 21
Reading a token:
Now at end of input.
7988.941375910:7fc05c5ab780: rsyslogd 7.4.7 startup,
module path '', cwd:/home/hosts
Reducing stack by rule 23 (line 150):
7988.941493297:7fc05c5ab780: caller requested object 'net', not
found (iRet -3003)
   $1 = 7988.941502940:7fc05c5ab780: Requested to load module
'lmnet'
nterm actlst (7988.941507791:7fc05c5ab780: loading module
'/usr/lib64/rsyslog/lmnet.so'
)7988.941582808:7fc05c5ab780: module lmnet of type 2 being
loaded (keepType=0).

7988.941587128:7fc05c5ab780: entry point
'isCompatibleWithFeature' not present in module
-> $$ = 7988.941589454:7fc05c5ab780: entry point 'setModCnf'
not present in module
nterm stmt (7988.941591808:7fc05c5ab780: entry point
'getModCnfName' not present in module
)7988.941593851:7fc05c5ab780: entry point 'beginCnfLoad' not
present in module

Stack now 0 1 14
Entering state 31
Reducing stack by rule 30 (line 163):
   $1 = nterm stmt ()
-> $$ = nterm block ()
Stack now 0 1 14
Entering state 32
Reducing stack by rule 28 (line 161):
   $1 = token PRIFILT ()
   $2 = nterm block ()[/code:2yqml3cp]

I can't understand, though, why I am not getting any logs created for today
or later? The permissions on the /home/hosts folder are 770.

What is going on what can I do to fix this and let it work in future?