[rsyslog] disk-assisted queue - feedback requested
rgerhards at hq.adiscon.com
Thu May 28 09:59:21 CEST 2009
as you probably know, I am currently implementing ultra-reliable queue
processing mode. This is a re-design of important parts of the queue engine.
The bulk of work went rather well, but I am now having some serious trouble
with terminating the queue in disk-assisted cases. Also, there are some
issues with disk-assisted modes at all.
Just as a reminder: disk-assisted (DA) mode is a mode where the queue usually
runs in memory but is configured to go to disk if we hit the configured upper
bound of messages to be kept in core. DA mode is implemented as two queues
running concurrently. Whenever the high water mark is reached, the regular
queue consumer is stopped, and a special queue-to-disk-queue consumer is
started which then shuffles messages from the main queue to the disk queue.
The regular consumer is than run off the disk queue (and only the disk
queue). This is reversed as soon as the in-memory queue consumption hits a
low water mark.
Experience has shown that DA mode is of limited use when it comes to bursts -
that is because it effectively slows down the engine greatly, as all messages
need to go to the disk first.
This was done to preserve the order of messages. However, with potentially
multiple workers and large batches, we do not have any decent order of
messages at all.
I think DA mode would greatly benefit if we give up the approach to try
preserve message sequence. In that case, I can run both the regular and the
disk-based consumer at the same time. This solves at least a couple of the
issues I have with termination (at least I think so), and it also makes this
mode more efficient in practical use. This is because we now can continue to
process data from the in-memory queue in parallel to shuffeling it to disk
(this is also the source of out-of-order processing).
I would appreciate feedback on this issue.
More information about the rsyslog