[rsyslog] Development of failsafe disk based queue

[Rainer Gerhards]

Hi David,

On Thu, 2008-10-09 at 10:53 +0200, David Ecker wrote:
> Hi Rainer,
> 
> the assert error seemed to be fixed in HEAD.

It is good to hear this. I begun to have some doubts when I reviewed the
code. I would really appreciate if you could download and test this
version here:

http://download.rsyslog.com/rsyslog/rsyslog-3.21.6-Test2.tar.gz

I will probably release that tomorrow and so some indication if the
problem is actually gone would be very good. The final 3.21.6 will see
one more patch, but nothing that affects the assert in question.

> 
> Mounting the ext3 filesystem with (noatime,sync,dirsync,rw) seemed to
> work a lot better.
> 
> /etc/rsyslog.conf
> ----------------
> $WorkDirectory /rsyslog/
> $ActionQueueFileName buffer
> $ActionQueueMaxDiskSpace 1g
> $ActionQueueSaveOnShutdown on
> $ActionQueueType Disk
> $ActionQueueMaxFileSize 100m
> $ActionQueueSize 10000000
> $ActionQueueCheckpointInterval 1
> $ActionResumeRetryCount -1
> *.* @@10.8.0.1:514
> -----------------
> 
> I found out that the queue was limited to 1000 elements if you do not
> define ActionQueueSize inside the config.

Yes, that's the default for *action* queues (the main message queue
default is different, I think 10,000).

> One difference was that version 3.18.3 actually created one file per msg
> if I did set ActionQueueCheckpointInterval  to 1. Right now only one
> file in addition to the .qi file is created containing all messages. I
> haven't waited to see what will happen if I reach 100MB on the data file.

That's probably a result of a bug fixed in the repo but not yet
released. But I don't check if it is gone now. 3.18.x will only see one
more release, then we move on to 3.20.x.

> If I shutdown rsyslog normally some status information seems to be
> written to the .qi file (508 bytes -> 1024 bytes). After restarting the
> qi file actually shrinks to 508 bytes again.

That's right.

> 
> If I kill rsyslogd (SIGKILL) the qi file is not updated with this
> information (as expected). But it looks like the queue is still working
> correctly after restarting rsyslogd at least it doesn't invalidate the
> queue or loosing messages.

I'd say it depends on when exactly it is doing when it is being killed.

> 
> Turning the system off immediatly seems to be working most of the times.
> After 11 tries the last one failed. It actually looked like that I
> turned the system off during a write.

This is also within what I expect. If you hit it during a write, things
are really bad. 

> 
> bye
> David Ecker
> 
> Rainer Gerhards schrieb:
> > Please use the version from git. I didn't realize that the bug affects
> > normal operations, but obviously it does. This is fixed and I'll see
> > that I release 3.21.6 ASAP, but I am not sure if I manage to do this
> > today.
> >
> > Gitweb available at http://git.adiscon.com
> >
> > Rainer
> >   
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com