[rsyslog] output plugin calling interface

[david at lang.hm]

On Sun, 3 May 2009, Tom Metro wrote:

> david at lang.hm wrote:
>> Rainer Gerhards wrote:
>>> After a lot of thinking today, we can have a "kind of" transactional queue,
>>> but we need to accept potential message *duplication* in the event of
>>> failures (but no loss).
>>
>> this is the approach that you have taken for other things (relp for
>> example), and when we were discussing reliability for direct mode vs disk
>> queues you mentioned that rsyslog could duplicate messages in case of
>> failures, but would not loose messages.
>
> I also noticed this side effect mentioned in the RELP documentation and
> wondered why message duplication couldn't be avoided by something as
> simple as assigning a serial number to each log record. A 32-bit
> monotonically increasing counter that rolls over periodically.
>
> The receiving side would cache the serial numbers for the last N records
> (something that could be done quite memory efficiently if the records
> show up mostly in order) and discard records it had seen.
>
> A hash might work well too, providing you're using high-res time stamps
> so you don't get false positive duplications.
>
> With a strictly in-memory cache of seen records, you could still get
> duplication after the receiver gets restarted, but at least you'd have
> greatly narrowed the potential. And the receiver could always pre-seed
> its cache from the last N stored records on startup.

note that this would have to be a per-sender list of records, what if you 
are getting messages from lots of systems?

since you can have two or more threads sending you messages you can't 
assume that you will get them in order (if you could assume this you could 
just store the last sucessful message processed)

I agree that this is the basic approach that would need to be taken, but 
before we worry about filtering out duplicates for cases like this, we 
need to make sure we aren't loosing any messages ;-)

David Lang