[rsyslog] RFC: On rsyslog output modules and support for batchoperations
david at lang.hm
david at lang.hm
Fri Apr 3 06:36:03 CEST 2009
On Thu, 2 Apr 2009, Rainer Gerhards wrote:
> Just a partial response (but quick ;))
>> I confess to being a bit confused as to why the existing output module
>> interface wasn't readily extending to batching,
> That would be the real solution (and David Lang suggested it long ago). The
> only problem is it takes quite some effort, as we need to make sure we do not
> lose messages along that way.
for those who are interested, what I proposed was to shift completely away
from the idea that the output module processes a fixed number of records,
and instead have a loop something like the following.
if (# events > N)
grab first N events
grab all events
create sql string
insert to database
mark the events grabbed as written
with the create sql string being something like the following perlish code
$sql=$header.join($mid, at events).$footer;
so you could say
$header='insert into table logs values ('
$mid = '),('
and if you pass it three events you get
insert into table logs values (msg1),(msg2),(msg3);
five values you would get
insert into table logs values (msg1),(msg2),(msg3),(msg4),(msg5);
I was not concerned about the command parsing time, due to the fact that
if it takes a little longer, it just means that there are more events in
the queue for the next pass to handle. there could reach a point where you
have so many events that it matters, but since this process could easily
insert hundreds or thousands of messages in one statement the overhead is
> It is still on my agenda, but without a sponsor I fear it'll stay there
> for quite a while.
> In most cases, rsyslog is simply too fast to see a bottleneck.
More information about the rsyslog