[rsyslog] how to best rotate files

[Rainer Gerhards]

On Mon, 2008-09-15 at 00:10 -0700, david at lang.hm wrote:
> > This very much smells like a bug. Can you elaborate on what you did?
> 
> rsyslogd 3.18.3 on one machine with the config file
> 
> $ModLoad immark
> $MarkMessagePeriod 1200
> $ModLoad imuxsock
> $ModLoad imudp
> $ModLoad imklog
> $UDPServerRun 514
> $AllowedSender UDP, 127.0.0.1, 192.0.0.0/8
> $umask 0000
> $FileGroup root
> $FileOwner root
> $template TraditionalFormat,"%timegenerated% %HOSTNAME% 
> %syslogtag%%msg:::drop-last-lf%\n"
> *.*     /var/log/messages;TraditionalFormat      # log to a file in the 
> traditional format
> 
> tcpreplay on a second machine spewing log entries at it at a rate of 
> ~30,000 logs/sec
> 
> crontab running the following script every 5 min
> 
> #!/bin/sh
> #
> 
> PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
> 
> umask 022
> year=`date +%Y`
> month=`date +%m`
> day=`date +%d`
> fdate=`date +%Y%m%d.%H%M`
> logroot=/var/log
> logroll=$logroot/oldlogs
> 
> cd $logroot
> mkdir -p $logroll/$year/$month/$day >/dev/null 2>/dev/null
> mv messages $logroll/$year/$month/$day/messages.$fdate
> pkill -HUP rsyslogd
> 2>/dev/null &
> gzip -9 $logroll/$year/$month/$day/messages.$fdate
> 
> 
> I get a couple of rotations sucessfully (with ~22,000 logs/sec in the 
> files), but then rsyslog seems to hang.
> lsof show it with the file open that was moved (and later deleted by 
> gzip).
> I need to do a kill -9 of rsyslogd to get it to recover.

Definitely a bug... I now "just" need to find out where. I guess there
is some mutex issue in the queue engine during shutdown. Don't expect a
fix too soon, but I'll do my best. I hope I can find a simpler error
condition (if my conclusion on the problem source is right...)

Rainer