[rsyslog] Development of failsafe disk based queue

[david at lang.hm]

On Wed, 1 Oct 2008, Rainer Gerhards wrote:

> On Wed, 2008-10-01 at 14:45 +0200, David Ecker wrote:
>
> [snip]
>
>> as long as you do sector based writes (512 byte per sector, usual) you
>> can be sure that the write wasn"t partial.. Writing more than one sector
>> or not starting at a correct offset (n*512,n=0,1,2,...x) might result in
>> a partial write. I'll already tested that with my devel client here. So
>> fencing each sector with a crc32 value would help detecting errors
>> during a write operation. This is actually only a problem if you are
>> writing directly to a block device like any filesystem does and yes,
>> reordering is definitly a problem. So validating the content written to
>> the disk afterwards is important.
>>
>> If writing through a filesystem reserving space in the destination file
>> beforehand actually minimizes errors since the file system table doesn't
>> have to be updated (you should also use the Flag O_NOATIME for that
>> case). See for example VMWare ESX VMDK file handling.
>
> Well, first of all let me re-iterate that I do not intend to do a block
> device driver for rsyslog (but I definitely do not object getting one
> contributed ;)).
>
> Still thinking about the case and thinking about non-solid-state,
> non-internal-battery-backed-up disk, I can't see how you can be sure the
> data will be written. David just told me there are no capacitors. So if
> power fails, it fails rather quickly. So how can you be sure the disk
> will be able to finish writing that sector? Let's say the drive has
> begun to write the sector and been able to write the first 5 bytes. Now
> power fails. No capacitors, no battery-backup, so why should there be
> enough power to drive the disk write head for another 507 bytes? It the
> drives assures it can do that, it needs capacitors - doesn't it?
>
> Am I overlooking something obvious?

one possible thing is tht if the write has not completed then the system 
sending you the logs has not received confirmation that you have the log 
yet, so they are the ones responsible for it.

it's only after you acknowledge the message (via relp or equivalent) that 
you are required to not loose the log message.

with ext3 and data=journaled you can do this. the writes to the journal 
are done by the filesystem in such a way that they can be considered 
atomic (either they happen and are reliable, or they can be treated as if 
they never happened), once that write (or rather the sync on that write) 
has completed the data is safe. the filesystem will later modify the 
actual sector on disk, but it jumps through hoops to make that safe from 
power outages.

David Lang