[rsyslog] untra-reliable speed test
david at lang.hm
david at lang.hm
Fri May 8 19:38:55 CEST 2009
On Fri, 8 May 2009, Kenneth Marshall wrote:
> On Fri, May 08, 2009 at 01:18:37AM -0700, david at lang.hm wrote:
>> On Fri, 8 May 2009, Rainer Gerhards wrote:
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>> I have a box put togeaterh for a first cut at a speed test of rsyslog in
>>>> untra-reliable mode. the outline below is intended to minimize the number
>>>> of variables.
>>>> the box is a dual quad-core opteron with 8G of ram, one SATA drive and a
>>>> fusionIO SSD PCIE drive, currently running RHEL 5.3 kernel 2.6.18-53
>>>> (redhat stock kernel) I intend to format the SSD with ext2 (as the
>>>> application is providing data integrity, and to avoid the known
>>>> performance problems with ext3 and fsync)
>>> Just a question, because I do not know enough about ext2: does ext2 guarantee
>>> that when an application does fsync, all data, INCLUDING related file system
>>> control structures are written to disk? Or, to phrase it the other way
>>> around, can ext2 guarantee that fsync'ed data can always be read after a
>>> power failure. I think along the lines of some control structures not being
>>> written, thus the fsynced app data may be present on the disk, but cannot be
>>> accessed any longer. In the worst case, would it be possible that a whole
>>> file be lost during a file system check after reboot?
>>> My *uneducated* understanding is that ext3 does guard against this (thus the
>>> performance problems) but ext2 does not.
>> the performance problem with ext3 is that it forces ALL pending writes to
>> disk when anything does a fsync
>> now that you mention it, I think that with all filesystems other than ext2
>> you need to do a fsync on the directory as well as on the file
>>> If my understanding would be correct (and I don't say so), we would need to
>>> use ext3.
> I think if you use ext3 with data=writeback, you will not have the
> flush everything problem. Of course, you will need to precreate the
with an application that does fsync, at that point you have no reliability
gain compared to ext2, but you still have the overhead of the journal
(including the need to write the data twice, once to the journal, once to
it's final location)
>> I'll try both (and later on, when I use by own kernel rather than the
>> redhat one I'll also test XFS)
>> I think that if no other disk activity is taking place ext3 maynot be too
>> bad (one other advantage that ext2 would have over ext3 and XFS is that
>> journaling filesystems have to write whatever they journal twice (once to
>> the journal and once to the final location)
>>>> for the rsyslog test I am thinking the following
>>>> useing rsyslog 4.1.7
>>>> enable input file
>>> Not sure if I got this bullet point right. Do you mean you intend to use
>>> imfile for input generation?
>> yes, that was my intent. just to simplify things by making the test
>> completely self contained to the one box.
>>> In any case, I would suggest to do a test with UDP and one with TCP senders,
>>> both sending at maximum rate. With UDP, we would see a message loss rate,
>>> while with TCP we would see the actual number of messages that the system can
>>> process. So TCP is probably the more meaningful number, but packet loss rate
>>> for UDP - a common use case - would also be interesting, at least I think so.
>> will do.
>> I will be interested in seeing the UDP loss rate, I suspect that with
>> appropriate OS tuning I can get it down to zero loss rate at the data
>> rates that the rest of the system maintains (the OS has a buffer prior to
>> rsyslog's input process that can cover delays on the input threads)
>>>> set the main queue mode to disk
>>>> enable fsyncs everwhere
>>> Just as a reminder: this includes $MainMsgQueueCheckpointInterval 1 (which is
>>> a *real* performance eater and puts a lot of burden on the consistency of the
>>> file system's control structures, thus my question on ext2 vs. ext3 above).
>> does this do a fsync on the directory.
>>>> set the output to log *.* to a file
>>>> run a cron job that rolls the log file once a min and sends a HUP to
>>>> create a large file of log information
>>>> run this for a while and then count the number of logs in each rolled log
>>>> file. hopefully the number will be reasonably consistant.
>>>> does this sound like a reasonable approach? or is this going to not be
>>>> representitive for some reason?
>>> With the few comments above, I think this is a very reasonable approach and
>>> should provide very good insight.
>>> Actually, I hope that it can prove my point that this setup is too slow
>> there will definantly be a performance issue at some point here, the
>> question is if it's fast enough to be useable.
>> the drive claims to be able to do >100,000 I/O ops/sec. if we can manage
>> to get a few thousand logs/sec written on this, it will be extremely
>> David Lang
>> rsyslog mailing list
> rsyslog mailing list
More information about the rsyslog