[rsyslog] Development of failsafe disk based queue

[David Ecker]

Hi,

[quote]
Depending on the media and the block device driver design, individual
sector writes may
not be atomic. A physical sector in typical devices is 512 bytes. In
most cases, physical
sector writes are atomic (either completely written, or not modified at
all). A truly
reliable file system, however, cannot count on this.
[/quote]

In most cases it works but some way of validating the data is needed if
you want ultra reliability, which I don't need. If the last few messages
a few seconds before an immediate shutdown are lost but all other
messages are send correctly afterwards then that would be OK in my case.

I'll just test version 2.21.5 with the altered open behauvior. The disk
based queue-array developed by myself is just a fallback solution if the
disk-based queue doesn't work with an immediate shutdown.

David

Rainer Gerhards schrieb:
> On Wed, 2008-10-01 at 14:45 +0200, David Ecker wrote:
>
> [snip]
>
>   
>> as long as you do sector based writes (512 byte per sector, usual) you
>> can be sure that the write wasn"t partial.. Writing more than one sector
>> or not starting at a correct offset (n*512,n=0,1,2,...x) might result in
>> a partial write. I'll already tested that with my devel client here. So
>> fencing each sector with a crc32 value would help detecting errors
>> during a write operation. This is actually only a problem if you are
>> writing directly to a block device like any filesystem does and yes,
>> reordering is definitly a problem. So validating the content written to
>> the disk afterwards is important.
>>
>> If writing through a filesystem reserving space in the destination file
>> beforehand actually minimizes errors since the file system table doesn't
>> have to be updated (you should also use the Flag O_NOATIME for that
>> case). See for example VMWare ESX VMDK file handling.
>>     
>
> Well, first of all let me re-iterate that I do not intend to do a block
> device driver for rsyslog (but I definitely do not object getting one
> contributed ;)).
>
> Still thinking about the case and thinking about non-solid-state,
> non-internal-battery-backed-up disk, I can't see how you can be sure the
> data will be written. David just told me there are no capacitors. So if
> power fails, it fails rather quickly. So how can you be sure the disk
> will be able to finish writing that sector? Let's say the drive has
> begun to write the sector and been able to write the first 5 bytes. Now
> power fails. No capacitors, no battery-backup, so why should there be
> enough power to drive the disk write head for another 507 bytes? It the
> drives assures it can do that, it needs capacitors - doesn't it?
>
> Am I overlooking something obvious?
>
> Rainer
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>