[rsyslog] Development of failsafe disk based queue

[david at lang.hm]

On Wed, 1 Oct 2008, Rainer Gerhards wrote:

> Sorry, I overlooked this mail in the big bunch of messages. That's good
> reasoning.

I'm replying out of order as I see things anyway

> To cover these scenarios, we need to do everything with syncing. This
> also means that you can not use any of the disk-assisted modes, because
> in these modes we always try to keep things in memory in order to save
> writes.

I think you are saying that we must use the disk-only mode which is 
correct.

> So while you have convinced me things can go wrong, I'd still say that
> is is very unusual (at least very costly) to care for all these things.

absolutly!!

> But, of course, there are situations where it is needed. I'll probably
> see that I provide a facility to open files in "always sync" mode, but
> that for sure will not be the default setting ;)

thanks.

> But even with the fast solid state disks (and similar methods) you
> mention, I think there will be a severe impact on performance because
> everything now needs to go through two write (data+metadata) and two
> read (again, data+metadata) OS call where we currently simply update an
> in-memory structure.

given the performance gains that we have seen by eliminating syscalls, it 
will hurt to add these back in, even with solid-state disks. that being 
said, it looks like the output module is nowhere close to being the limit 
(when I could get a good, stable reading on it, it looked like it was 
eating ~15% cpu compared to the input module at 100%) so it may not make 
much of a difference.

> Just out of curiosity: do you expect the majority of you rollouts to be
> using such methods?

absolutly not.

I have one case I am considering (the one I am talking to you about more 
efficiant database writes) that would be this paranoid, but the rest of it 
will be optimized for speed (battery-backed disk caches on the final 
server, but everything else can just use ram)

David Lang

> Rainer
>
> On Wed, 2008-10-01 at 05:35 -0700, david at lang.hm wrote:
>>> ... And I have never heard of anybody doing serious datacenter work
>>> without a proper UPS. Is this *really* an issue?
>>
>> Yes.
>>
>> UPSs fail.
>> generators fail
>> power cords come loose.
>> power cords get unplugged by someone who thinks they are unplugging a
>> different system
>> people bump power switches on power strips.
>> power supplies are defective
>>
>> I had one production outage where a visiting tech pulled a power cord from
>> an overhead plug and dropped it on the ground, where it happened to hit
>> the power switch on a power strip.
>>
>> I've had high-end systems with redundant power supplies go down becouse of
>> faulty hardware that decided to disble both power supplies at once (it
>> turned out that there was a defect in the whole batch of servers, but it
>> took IBM several weeks to figure out what was going on)
>>
>> I've had UPS systems blow up (literally)
>>
>> I've had a datacenter go down becouse the it was running on generator
>> power (due to other issues), and the refueling guy filled the tank
>> incorrectly and got air bubbles into the fuel system, a few min later the
>> 500Kw diesel generator couldn't maintain constant speed and the safety
>> triggers kicked in and disabled it.
>>
>> it's amazing the things that happen in real-life
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>