[rsyslog] Development of failsafe disk based queue

[david at lang.hm]

On Wed, 1 Oct 2008, Rainer Gerhards wrote:

> On Wed, 2008-10-01 at 05:07 -0700, david at lang.hm wrote:
>> On Wed, 1 Oct 2008, Rainer Gerhards wrote:
>>
>>> One thing I forgot to mention: a pure disk queue (not a disk-assisted
>>> one) gets you as close to your goal as possible (well, mostly - we
>>> could, at a considerable performance expense, require synced writing).
>>> With that case, all data is immediately stored on disk. You can
>>> configure it to also write the meta data out immediately (and again with
>>> sync, not yet supported). However, you still have a window of exposure,
>>> for example if the power loss happens right in the middle of when the
>>> disk actually writes data to the disk sector.
>>>
>>> I still wonder why this scenario would be useful to address...
>>
>> not all uses of rsyslog are for simple system logs. it's a good general
>> purpose log tool, and there are some cases where you want to be as sure as
>> you possibly can be that once a message has been acknowledged it has no
>> chance of being lost.
>
> I designed the engine for audit-class reliability. However, I assumed
> that the rest of the system is also playing in that class. Doing
> everything with a potential power failure in mind creates a lot of extra
> demands. And I have never heard of anybody doing serious datacenter work
> without a proper UPS. Is this *really* an issue?

Yes.

UPSs fail.
generators fail
power cords come loose.
power cords get unplugged by someone who thinks they are unplugging a 
different system
people bump power switches on power strips.
power supplies are defective

I had one production outage where a visiting tech pulled a power cord from 
an overhead plug and dropped it on the ground, where it happened to hit 
the power switch on a power strip.

I've had high-end systems with redundant power supplies go down becouse of 
faulty hardware that decided to disble both power supplies at once (it 
turned out that there was a defect in the whole batch of servers, but it 
took IBM several weeks to figure out what was going on)

I've had UPS systems blow up (literally)

I've had a datacenter go down becouse the it was running on generator 
power (due to other issues), and the refueling guy filled the tank 
incorrectly and got air bubbles into the fuel system, a few min later the 
500Kw diesel generator couldn't maintain constant speed and the safety 
triggers kicked in and disabled it.

it's amazing the things that happen in real-life

David Lang

> Rainer
>>
>> useing some form of solid-state reliable storage (battery backed ram on a
>> raid controller, a battery backed ram disk, a flash disk) it is possible
>> (but not nessasarily cheap) to get the ability to do tens to hundreds of
>> thousands of writes + syncs per second
>>
>> David Lang
>>
>>> Rainer
>>>
>>> On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote:
>>>> Hi,
>>>>
>>>> I am looking for a failsafe solution to store syslog messages localy
>>>> until they could be send later. I already looked at the disk based
>>>> memory queue and the disk based queue. Both queue's don't work if you
>>>> just power down the system immediatly actually loosing the whole queue.
>>>> I already looked at queue.c and it seemed to me that both queues were
>>>> not designed for that kind of failure, but I could be wrong there. Since
>>>> an immediate power down of the system is the major failure which will
>>>> occure pretty often I need to create a soltution there.
>>>>
>>>> Did you already start to develop something addressing that problem?
>>>> Could you help me extend rsyslog (3.18.4) so that I can develop a new
>>>> queue myself? I would contribute the code to the rsyslog project if you
>>>> would like afterwards.
>>>>
>>>> bye
>>>> David Ecker
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>