[rsyslog] untra-reliable speed test

[Rainer Gerhards]

> > A small queue may build up depending on the OS scheduler, 
> but I think
> > most often, input and output will just wait for the queue 
> to complete.
> > In that sense, this mode is similar to DIRECT mode, except 
> that a queue
> > can build up when the action needs to be retried.
> 
> interestingly, except for ext2 the queue fills up to it's limit.

I overlooked that fact this morning. I have to admit that *this* puzzles me.
Some queue buildup makes sense but filling up to the limit... That really
sound strange.

> >>> With disk queues, there is always only a single queue 
> worker (the disk queue is purely sequential).
> >>
> >> interesting, the OS can do enough in parallel that it may 
> be worth looking
> >> into this if we ever go in the direction of optimizing this mode.
> >
> > If we optimize it, the best thing to do is a totally new 
> queue storage
> > driver for such cases. Sequential files do not really work 
> well if we
> > have multiple producers running.
> >
> > This is a major effort and even then we need to think about the
> > implications I raised in regard to processing cost above.
> >
> 
> <snip large discussion on optimization and history of disk queue>
> 
> the current speed is high enough to be useful, even without 
> batching (and 
> as noted before, batching should improve things significantly)

That's good news.

> so at the moment I would like to avoid discussion of how to 
> optimize the 
> disk queue (as much as I am itching to do so). for stability 
> reasons I 
> agree the best approach is probably to create a new queue 
> type rather than 
> doing major changes to the existing queue.

Just one final word for this quarter: it's not just about stability, a new
driver also can be designed for optimal performance from the ground up.

> 
> instead I would like to see how close we are to having it be 
> audit grade, 
> and how difficult it is to close that gap.

That's what I am currently looking at (and thus be a bit silent). I need to
review some of the queue code, thus it makes limited sense to do writeup now.
The queue is rather complex and what I do right now is make some
micro-changes while I do the review.

The ultra-reliable mode adds complexity to the dequeue operation. However,
the current mode has complexity during the queue shutdown, especially when it
needs to cancel worker threads after their timeout expired. The queue
shutdown needs to be modified for batch operation, especially the areas that
are most complex. So my idea is to do both together. That takes a bit longer
(mostly thinking), but I can envision that there is a good chance the
resulting code will be less complex. In the long term, that could also result
in easier maintenance.

This is also the reason why I think about the queue storage drivers: even by
thinking on a non-detail level, I hope to be able to see the hooks that we
need so that I can influence design decision into a direction that
facilitates the later addition (at least if the choices are otherwise more or
less equal). If things continue to work out like they did today, I hope to
have workable experimental code (and sufficient enough understanding) by
early next week. It depends a bit if I see code inside the queue that - after
15 month of practical experience with it - shold be
optimized/shuffled/whatever. I think this is a opportunity to apply knowledge
gained.

Rainer