[rsyslog] output plugin calling interface

[Tom Metro]

david at lang.hm wrote:
> Rainer Gerhards wrote:
>> After a lot of thinking today, we can have a "kind of" transactional queue,
>> but we need to accept potential message *duplication* in the event of
>> failures (but no loss).
> 
> this is the approach that you have taken for other things (relp for 
> example), and when we were discussing reliability for direct mode vs disk 
> queues you mentioned that rsyslog could duplicate messages in case of 
> failures, but would not loose messages.

I also noticed this side effect mentioned in the RELP documentation and 
wondered why message duplication couldn't be avoided by something as 
simple as assigning a serial number to each log record. A 32-bit 
monotonically increasing counter that rolls over periodically.

The receiving side would cache the serial numbers for the last N records 
(something that could be done quite memory efficiently if the records 
show up mostly in order) and discard records it had seen.

A hash might work well too, providing you're using high-res time stamps 
so you don't get false positive duplications.

With a strictly in-memory cache of seen records, you could still get 
duplication after the receiver gets restarted, but at least you'd have 
greatly narrowed the potential. And the receiver could always pre-seed 
its cache from the last N stored records on startup.

  -Tom