[rsyslog] untra-reliable speed test

[Kenneth Marshall]

On Fri, May 08, 2009 at 01:18:37AM -0700, david at lang.hm wrote:
> On Fri, 8 May 2009, Rainer Gerhards wrote:
> 
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >>
> >> I have a box put togeaterh for a first cut at a speed test of rsyslog in
> >> untra-reliable mode. the outline below is intended to minimize the number
> >> of variables.
> >>
> >> the box is a dual quad-core opteron with 8G of ram, one SATA drive and a
> >> fusionIO SSD PCIE drive, currently running RHEL 5.3 kernel 2.6.18-53
> >> (redhat stock kernel) I intend to format the SSD with ext2 (as the
> >> application is providing data integrity, and to avoid the known
> >> performance problems with ext3 and fsync)
> >
> > Just a question, because I do not know enough about ext2: does ext2 guarantee
> > that when an application does fsync, all data, INCLUDING related file system
> > control structures are written to disk? Or, to phrase it the other way
> > around, can ext2 guarantee that fsync'ed data can always be read after a
> > power failure. I think along the lines of some control structures not being
> > written, thus the fsynced app data may be present on the disk, but cannot be
> > accessed any longer. In the worst case, would it be possible that a whole
> > file be lost during a file system check after reboot?
> >
> > My *uneducated* understanding is that ext3 does guard against this (thus the
> > performance problems) but ext2 does not.
> 
> the performance problem with ext3 is that it forces ALL pending writes to 
> disk when anything does a fsync
> 
> now that you mention it, I think that with all filesystems other than ext2 
> you need to do a fsync on the directory as well as on the file
> 
> > If my understanding would be correct (and I don't say so), we would need to
> > use ext3.
> 
FYI,

I think if you use ext3 with data=writeback, you will not have the
flush everything problem. Of course, you will need to precreate the
files.

Regards,
Ken

> I'll try both (and later on, when I use by own kernel rather than the 
> redhat one I'll also test XFS)
> 
> I think that if no other disk activity is taking place ext3 maynot be too 
> bad (one other advantage that ext2 would have over ext3 and XFS is that 
> journaling filesystems have to write whatever they journal twice (once to 
> the journal and once to the final location)
> 
> >> for the rsyslog test I am thinking the following
> >>
> >> useing rsyslog 4.1.7
> >> enable input file
> >
> > Not sure if I got this bullet point right. Do you mean you intend to use
> > imfile for input generation?
> 
> yes, that was my intent. just to simplify things by making the test 
> completely self contained to the one box.
> 
> > In any case, I would suggest to do a test with UDP and one with TCP senders,
> > both sending at maximum rate. With UDP, we would see a message loss rate,
> > while with TCP we would see the actual number of messages that the system can
> > process. So TCP is probably the more meaningful number, but packet loss rate
> > for UDP - a common use case - would also be interesting, at least I think so.
> 
> will do.
> 
> I will be interested in seeing the UDP loss rate, I suspect that with 
> appropriate OS tuning I can get it down to zero loss rate at the data 
> rates that the rest of the system maintains (the OS has a buffer prior to 
> rsyslog's input process that can cover delays on the input threads)
> 
> >> set the main queue mode to disk
> >> enable fsyncs everwhere
> >
> >
> > Just as a reminder: this includes $MainMsgQueueCheckpointInterval 1 (which is
> > a *real* performance eater and puts a lot of burden on the consistency of the
> > file system's control structures, thus my question on ext2 vs. ext3 above).
> 
> does this do a fsync on the directory.
> 
> >> set the output to log *.* to a file
> >>
> >> run a cron job that rolls the log file once a min and sends a HUP to
> >> rsyslog
> >>
> >> create a large file of log information
> >>
> >> run this for a while and then count the number of logs in each rolled log
> >> file. hopefully the number will be reasonably consistant.
> >>
> >> does this sound like a reasonable approach? or is this going to not be
> >> representitive for some reason?
> >
> > With the few comments above, I think this is a very reasonable approach and
> > should provide very good insight.
> >
> > Actually, I hope that it can prove my point that this setup is too slow
> > wrong...
> 
> there will definantly be a performance issue at some point here, the 
> question is if it's fast enough to be useable.
> 
> the drive claims to be able to do >100,000 I/O ops/sec. if we can manage 
> to get a few thousand logs/sec written on this, it will be extremely 
> usable.
> 
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>