Sync writes very slow. Ubuntu 10.10, 32 bits, ext4

Question

I am running ActiveMQ on my Macbook Pro that runs Ubuntu 10.10, 32 bits with an ext4 partition.

Linux iker-laptop 2.6.35-23-generic-pae #40-Ubuntu SMP Wed Nov 17 22:32:51 UTC 2010 i686 GNU/Linux

If I enable persistence in ActiveMQ the performance drops terribly. I've tested the same thing on other machines and the difference is of 2 orders of magnitude.

There is a tool with activeMQ to test the HD, here are the results:

iker@iker-laptop:~/apps/apache-activemq-5.4.1$ java -classpath lib/kahadb-5.4.1.jar org.apache.kahadb.util.DiskBenchmark 

Benchmarking: /home/iker/apps/apache-activemq-5.4.1/disk-benchmark.dat
Writes: 
  146171 writes of size 4096 written in 11.074 seconds.
  13199.477 writes/second.
  51.560455 megs/second.

Sync Writes: 
  197 writes of size 4096 written in 10.006 seconds.
  19.688187 writes/second.
  0.07690698 megs/second.

Reads: 
  5589861 reads of size 4096 read in 10.001 seconds.
  558930.2 writes/second.
  2183.321 megs/second.

Performance for Sync Writes is s**t. There must be something misconfigured, but this is the only app where I've noticed a HD performace issue.

hdparm throws expected values:

iker@iker-laptop:~$ sudo hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   6282 MB in  2.00 seconds = 3141.73 MB/sec
 Timing buffered disk reads:  240 MB in  3.00 seconds =  79.88 MB/sec

Answer 1

The major limiting factor for synchronous IO isn't throughput of your harddrive, but rather the time it takes from when the write is issued and it being committed to disk. The most relevant performance metric for a harddrive in this regard would be the seek time of the harddrive and not it's throughput under ideal circumstances.

In addition to the hardware working against you, so are the kernel, I'm guessing you might see a small improvement (although, probably nowhere near what you'll get from doing async IO) if you can ionice the benchmark (application) to run under the realtime IO scheduling class. By default, applications will be scheduled under the best effort class, which probably also will add to the wait time of your writes. Use the realtime scheduling class at your own risk, as it'll have adverse effects on other applications performance when accessing disk.

In general, I don't really think there's anything horribly wrong with the synchronous write performance you're seeing. Synchronous IO will in general perform horribly compared to asynchronous IO.

As a side note a quick google of activemq and synchronous io gave the following:

For performance reasons you may wish to stream messages to the broker as fast as possible even if you are using persistent messages

Answer 2

The cfq I/O scheduler tends to perform horribly in these sorts of tests. In addition to the ionice suggestion earlier, you might want to try switching to the deadline I/O scheduler (either by booting with elevator=deadline or by doing for n in /sys/block/sd*/queue/scheduler ; do echo deadline > $n ; done as root).

Answer 3

A synchronous write must receive back an acknowlegement that the write has been committed (whether the commit was a success or error) before it can return itself. This is by design, and inherently makes synchronous writes much slower due to the high latency times involved with a spinning metal disk (on disk ram cache doesn't count).

Asynchronous writes are usually written to RAM and the OS deals with committing it to disk later (later usually being mere seconds or less (I believe ZFS is 5x/second, or every 5 seconds)). Disk seek times are measured in ms while RAM seek times are measured in ns. That's a difference of 1000x.

Use synchronous writes when it's absolutely critical that the data is permanently stored before continuing and a 1 second delay where power loss may occur is unacceptable.

Use asynchronous writes at all other times.

Answer 4

The most probable reason you are seeing this performance breakdown is because you are using "-o sync" with a journaled filesystem and barriers turned on (which is the default for ext4).

This is where the decision on what to do to improve it becomes very difficult.

It mostly boils down to how much you trust your hardware though.

From mount(8): "Write barriers enforce proper on-disk ordering of journal commits, making volatile disk write caches safe to use, at some performance penalty. The ext3 filesystem does not enable write barriers by default. Be sure to enable barriers unless your disks are battery-backed one way or another. Otherwise you risk filesystem corruption in case of power failure."

So either accept the fact "-o sync" performance is dismal, or buy battery-backed cache for your controller and really good SAS disks, then turn barriers off using "-o sync,nobarrier".

If what you're currently using is a proper enterprise-class FC or iSCSI storage backend, then I guess you're safe to do the latter, too.

All in all, ActiveMQ 5.4 uses the KahaDB storage backend by default, and that one has its own transaction log, too, so perhaps even turning journaling off on the filesystem level could work for you, but then make absolutely certain you use "enableJournalDiskSyncs" option for the backend, and you probably want to put it on a separate block device as well if you haven't yet.

(see http://activemq.apache.org/kahadb.html for more)

Answer 5

Synchronous writes are slow, its why we buffer everything. Take a look at IOPS on Wikipedia and you will see a typical 7,200 rpm HDD has 75-100 IOPS. Now take a look at the technical specs of a Macbook Pro, and it has a 5,400 rpm HDD. That's 75% performance at best so you are looking at 50-75 IOPS at best for the laptop.

A MQ maybe writing a data ledger and an accounting ledger, which gets you to the 20 IOPS you are seeing in the ActiveMQ benchmark.

You have two options, test on tmpfs, i.e. in-memory file system, or use a SSD. Normally servers using synchronous writes will have a significant SAS/SCSI RAID array with 15,000 rpm disks. Extra disks are added to the array to improve performance, not to increase capacity.

Answer 6

On a hosted VM (in VirtualBox) running Ubuntu 11.10 server 64 bit also using ext4 we obtained the following results:

Sync Writes:
 288 writes of size 4096 written in 10.034 seconds.
 28.702412 writes/second.
 0.112118796 megs/second.

On a physical server running Redhat 5.7 64 bit using ext3 the following results were obtained:

Sync Writes:
  54987 writes of size 4096 written in 10.001 seconds.
  5498.1504 writes/second.
  21.47715 megs/second.

I wonder if the OP was running this in a VM as well or if there is a problem between ext3 and ext4. I appreciate that there could be a difference between hosted and non-hosted environments, however did not expect such a great difference.

Answer 7

Use a bigger block size. Are you maybe confusing sync/async IO with direct/buffered IO?