[rsyslog] disk-assisted queue - feedback requested
rgerhards at hq.adiscon.com
Thu May 28 14:27:38 CEST 2009
An update: I have implemented this change to DA mode, and it looks very good.
The performance is obviously much better and the mis-order is not so bad in
usual cases, as only few messages make it to the DA queue. So it is not much
worse than with batching and multiple workers.
Anyhow, I've started a new v5-devel branch, as I think the changes to the
queue engine are so large that this code will probably become v5, so v4 will
end up to be just an interim release with only some additions.
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Thursday, May 28, 2009 9:59 AM
> To: rsyslog-users
> Subject: [rsyslog] disk-assisted queue - feedback requested
> Hi all,
> as you probably know, I am currently implementing ultra-reliable queue
> processing mode. This is a re-design of important parts of the queue
> The bulk of work went rather well, but I am now having some serious trouble
> with terminating the queue in disk-assisted cases. Also, there are some
> issues with disk-assisted modes at all.
> Just as a reminder: disk-assisted (DA) mode is a mode where the queue
> runs in memory but is configured to go to disk if we hit the configured
> bound of messages to be kept in core. DA mode is implemented as two queues
> running concurrently. Whenever the high water mark is reached, the regular
> queue consumer is stopped, and a special queue-to-disk-queue consumer is
> started which then shuffles messages from the main queue to the disk queue.
> The regular consumer is than run off the disk queue (and only the disk
> queue). This is reversed as soon as the in-memory queue consumption hits a
> low water mark.
> Experience has shown that DA mode is of limited use when it comes to bursts
> that is because it effectively slows down the engine greatly, as all
> need to go to the disk first.
> This was done to preserve the order of messages. However, with potentially
> multiple workers and large batches, we do not have any decent order of
> messages at all.
> I think DA mode would greatly benefit if we give up the approach to try
> preserve message sequence. In that case, I can run both the regular and the
> disk-based consumer at the same time. This solves at least a couple of the
> issues I have with termination (at least I think so), and it also makes
> mode more efficient in practical use. This is because we now can continue
> process data from the in-memory queue in parallel to shuffeling it to disk
> (this is also the source of out-of-order processing).
> I would appreciate feedback on this issue.
> rsyslog mailing list
More information about the rsyslog