[rsyslog] output plugin calling interface

[david at lang.hm]

On Mon, 4 May 2009, Tom Metro wrote:

> david at lang.hm wrote:
>> if you make your app talk RELP you should be able to get complete
>> end-to-end reliability.
>
> Makes sense. Is there a RELP client library that is a drop-in
> replacement for the equivalent syslog API?

not that would be a drop-in replacement.

> So presumably this RELP client library would incorporate a queue. Of
> course unless you want the absence of logging to block your application,
> most of these reliability enhancements only buy you some time until the
> buffer fills up.

not nessasarily, with classic syslog, the application blocks until the 
syslog daemon completes the write of the log message to disk (including 
doing an fsync). if the syslog daemon can't handle the request, the 
application waits.

the only syslogd that I know of that lets you avoid this blockage is the 
linux sysklogd (syslog-ng and rsyslog are replacements that add a _lot_ of 
other functionality as well as memory buffering). if you are using syslogd 
on any *nix platform other than linux, your apps have to wait for the disk 
for each log message.

yes, this can cripple application performance (and frequently does)

with rsyslog's modular structure it's also possible to write an input 
module that does application-level acknowledgements like relp does, that 
then feeds the message into the normal rsyslog mechansim. as long as the 
input module doesn't acknowledge recipt of the message before it's in the 
queue, this should result in a reliable end-to-end message delivery 
mechansim.

I define this as:

If the application sucessfully writes the log message, the message will 
not be lost, even in the face of infrastructure failures

If the application doesn't sucessfully write the log message (either can't 
contact rsyslog, or doesn't get an acknowledgement), the application 
should assume that the log message did not get through and respond 
accordingly.

this could be to pause and retry, it could be to abort the action it was 
about to make, or something else (if the application needs to be _really_ 
paranoid, it will log that it's about to do the critical action, do the 
action, then log that it completed the action. that way if things go down 
in the middle you have a record that you were in that state and can do 
futher investigation to see if that event actually ovccured or not).

In general I believe that duplicated messages are a far better situation 
than lost messages. Applicaitons can put unique IDs in a message if they 
are worried about duplicates (and filter them on the back-end), they can't 
do anything about messages that disappear on them.

David Lang