0

I have recently done some performance tests on a new server (using dd) and wonder why read performance is so much worse than write performance? Shouldn't it be another way around?

file size was in both tests 550GB, read: in seconds: 3704 in MB/s: 148

and write: in seconds: 1539 in MB/s: 357

write command:

time sh -c "dd if=/dev/zero of=/local/postgresql/bigfile 
bs=8k count=67108864 && sync"

read command:

time dd if=/local/postgresql/bigfile of=/dev/null bs=8k

bash time command output:

real: 61m44.335s
user: 0m12.721s
sys: 10m35.884s

Bonnie++ results command:

bonnie++ -f -D -n 0 -u root -d /local/postgresql/

results are for file that was twice as big as RAM size.

write:

419 918 K/sec

read:

~ 187 000 K/sec

  • which kind of test did you run? can you post the whole command line you used? How much ram do you have in your environment? – pqnet Aug 17 '14 at 17:34
4

You need to use the write sync flags to test performance, to ensure you are actually writing on disk and not on cache. Use conv=fdatasync to force a sync of buffers after writing has ended. See here for details.

time dd .... conv=fdatasync

for read test, discard caches before testing:

flush
echo 3 | sudo tee /proc/sys/vm/drop_caches
time dd ....
  • Also, note that it may be very possible for the write speed to be faster than read speed: for example, if your write are sequential and read are not (and you have a spinning disk). This is particularly true in RAID systems – pqnet Aug 17 '14 at 17:44
  • I did something similar: "dd if=/dev/zero of=/local/postgresql/bigfile bs=8k count=67108864 && sync" so sync was performed at the end of operation. drop_caches was used between all tests. Is it pretty the same or not? – Marcin Bykowski Aug 17 '14 at 17:45
  • @MarcinBykowski using sync at the end does not affect the time reported by dd, which has already written your results before even evaluating sync command... – pqnet Aug 17 '14 at 17:47
  • ok, but it should affect bash time command? post edited with bash output. – Marcin Bykowski Aug 17 '14 at 17:49
  • @MarcinBykowski I guess so. Probably your system is really faster at writing than reading. Are you sure the reading is sequential? – pqnet Aug 17 '14 at 17:51
1

What was the command you used? dd does very different things related to percormance depending on the options.

But from what you write,

I think you were reading small blocks, which will get read from the disk as you ask for them, roughly.

And you where writing small blocks, which will be written to the disk when the kernel feels it has time to do it - not when dd writes them out.

That would already explain the difference, right?

  • I have just edited my post with the exact command I have used. Yes, blocks were 8k. – Marcin Bykowski Aug 17 '14 at 17:47
1

I'm very doubtful that you can get meaningful benchmarks from dd. dd just shows you how large sequential reads or large sequential asynchronous writes perform between various devices. As long as your workload consists mainly of copying large files between these filesystems you're alright. I doubt that's your workload, though.

Your best bet is to profile disk usage and use a real I/O benchmarking suite (link bonnie++ or something) to test how much effect changing various tunables has. For a database, I would expect a lot of random reads. Setting noatime and doing data=writeback on your main data files (with regular backups being made) is probably the best you can do with what information we have so far.

To answer what seems to be your bigger question, it's because asynchronous writes (like those made by dd) can get buffered in memory and committed to disk. They're kind of I/O bound insofar as queues and buffers can fill and you would have to wait for them to become available again (by committing to disk) before you can stack more on.

Reads, on the other hand, are definitionally I/O bound so you don't usually get the same asynchronous action going on. You can play around with read_ahead_kb and the like so that more sequential data is read into memory in anticipation of being asked for by the workload in the near future.

That's about all I can think to answer with what we know so far. Let me know if you have any questions.

  • after dd test I did test with bonnie++, I have edited my post with results. – Marcin Bykowski Aug 17 '14 at 18:27
  • using conv=fdatasync option or using sync command afterwards should solve the cache issue. – pqnet Aug 17 '14 at 18:33
  • @Joel Davis as I have red in newer kernels default option is relatime (aliases with atime), so setting the parameter to noatime will reduce write operations not that much as before. in my tests haven't seen any differences as well. – Marcin Bykowski Aug 17 '14 at 18:34
  • @pqnet - yes thank you I'll for sure use this option with next test. – Marcin Bykowski Aug 17 '14 at 18:35
  • @pqnet that's still not a good test as that just does asynchronous writes followed by an fsync. It gets things closer to the mark but I'd wager the vast majority of writes to database files are synchronous. The cache thing isn't that much of an issue. The real issue is that dd isn't good at benchmarking, it's basically just a really beefed up cp command. – Bratchley Aug 17 '14 at 18:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.