[rsyslog-notify] Forum Thread: Re: imfile seems to just stop reading files - (Mode 'reply')

[noreply at adiscon.com]

User: vizette 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=26845#p26845

Message: 
----------
So I have a follow up on this with more testing...

It is definitely related to logs that are being written by syslog, and yet
is also related to logrotate.

1)  I created an empty log (manually) in /var/log/ and used imfile to
monitor it and send to kafka - this was just added to the existing rsyslog
config.  I was able to echo to it and see it send out (via tcpdump).  After
logrotate ran last night I am still able to echo to it and see this send
out to kafka, but logs being written by rsyslog are no longer sending. 
Note:  this test log was not rotated last night but I have it set to rotate
tonight to see if it triggers the issue.  I do not expect it to based on
what we've seen so far (non rsyslog logs continue to ship to kafka).

2) Confirmed this happens during log rotation - I have an strace of rsyslog
while this happens but I'm not able to determine what the issue might be
based on that.  However if it's of interest to you I can send it.

3) I created a separate rsyslog process on one machine, whose entire
purpose is to just watch (via imfile) a log being written by the system
rsyslog and send to kafka.  This also stopped sending data after
logrotation - the new rsyslog process got a HUP during logrotation.  HUPing
the new rsyslog monitor manually does not fix it, but killing and
restarting it does (just like the original issue)

4) I tried to force the issue by running "logrotate --force" against the
applications logrotate configuration file - this does
[i:2tzpqrib]not[/i:2tzpqrib] seem to trigger the issue.  Data still sends
to kafka after this, and logs rotate and continue as expected.

5) If I manually move a log file that is having issues and HUP rsyslog, a
new log is created and data populates as expected.  Data continues to be
sent to kafka as expected.

6) Logrotate is current for this OS.  Also tested with rsyslog 8.20 - same
issues.

I'm running out of testing scenarios to narrow this down...  We had this
issue in the past apparently with some different applications and different
servers - the solution then was to change HUP to restart for rsyslog during
log rotation.  I'm really trying to avoid this if possible.

I'm going to set a cron to do a force logrotate and see what happens. 
Maybe it's an environment issue with cron/logrotate?

-Rich