How can I benchmark my HDD?

Question

I've seen commands to benchmark one's HDD such as this using dd:

$ time sh -c "dd if=/dev/zero of=ddfile bs=8k count=250000 && sync"

Are there better methods to do so than this?

Answer 1

I usually use hdparm to benchmark my HDD's. You can benchmark both the direct reads and the cached reads. You'll want to run the commands a couple of times to establish an average value.

Examples

Here's a direct read.

$ sudo hdparm -t /dev/sda2

/dev/sda2:
 Timing buffered disk reads: 302 MB in  3.00 seconds = 100.58 MB/sec

And here's a cached read.

$ sudo hdparm -T /dev/sda2

/dev/sda2:
 Timing cached reads:   4636 MB in  2.00 seconds = 2318.89 MB/sec

Details

-t     Perform  timings  of  device reads for benchmark and comparison 
       purposes.  For meaningful results, this operation should be repeated
       2-3 times on an otherwise inactive system (no other active processes) 
       with at least a couple of megabytes of free memory.  This displays  
       the  speed of reading through the buffer cache to the disk without 
       any prior caching of data.  This measurement is an indication of how 
       fast the drive can sustain sequential data reads under Linux, without 
       any filesystem overhead.  To ensure accurate  measurements, the 
       buffer cache is flushed during the processing of -t using the 
       BLKFLSBUF ioctl.

-T     Perform timings of cache reads for benchmark and comparison purposes.
       For meaningful results, this operation should be repeated 2-3
       times on an otherwise inactive system (no other active processes) 
       with at least a couple of megabytes of free memory.  This displays
       the speed of reading directly from the Linux buffer cache without 
       disk access.  This measurement is essentially an indication of the
       throughput of the processor, cache, and memory of the system under 
       test.

Using dd

I too have used dd for this type of testing as well. One modification I would make to the above command is to add this bit to the end of your command, ; rm ddfile.

$ time sh -c "dd if=/dev/zero of=ddfile bs=8k count=250000 && sync"; rm ddfile

This will remove the ddfile after the command has completed. NOTE: ddfile is a transient file that you don't need to keep, it's the file that dd is writing to (of=ddfile), when it's putting your HDD under load.

Going beyond

If you need more rigorous testing of your HDD's you can use Bonnie++.

References

• How to use 'dd' to benchmark your disk or CPU?
• Benchmark disk IO with DD and Bonnie++

Answer 2

(This is a very popular question - you can see variations of it on https://stackoverflow.com/q/1198691 , https://serverfault.com/q/219739/203726 and https://askubuntu.com/q/87035/740413 )

Are there better methods [than dd] to [benchmark disks]?

Yes but they will take longer to run and require knowledge of how to interpret the results - there's no single number that will tell you everything in one go because the following influence the type of test you should run:

• Are you interested in the performance of I/O that is random, sequential or some mix of the two?
• Are you reading from or writing to the disk (or some mixture of the two)?
• Are you concerned about latency, throughput or both?
• Are you trying to understand how different parts of the same hard disk perform (generally speeds a faster closer to the centre of spinning disks)?
• Are you interested in how a given filesystem will perform when using your disk or do you want results closer to the disk's raw performance by doing I/O straight to a block device?
• Are you interested in how a particular size of I/O performs?
• Are you submitting the I/O synchronously or asynchronously?
• How much I/O are you submitting (submit too little the wrong way and all the I/O could be cached so you wind up testing the speed of your RAM rather than the speed of the disk)?
• How compressible is the content of the data you are writing (e.g. zero only data is highly compressible and some filesystems/disks even have a special fast-path for zero only data leading to numbers that are unobtainable with other content)?

And so on.

Here's a short list of tools with easiest to run at the top and difficult/more thorough/better nearer the bottom:

1. dd (sequential reads or writes, only shows throughput, can be configured to use a filesystem or block device, can be configured to bypass the block cache/wait for I/O to be really completed)
2. hdparm (sequential reads only, only shows throughput, never uses a filesystem, can be configured to bypass the block cache, cache test only re-reads the starting 2 MBytes)
3. GNOME Disk Utility's benchmark (easy to run, never uses a filesystem, graphical but requires a full GNOME install, gives latency and throughput numbers for different types of I/O but write workload is actually doing read/write/fsync per sample size).
4. fio (can do nearly anything and gives detailed results but requires configuration and an understanding of how to interpret said results). Here's what Linus says about it:

   Greg - get Jens' FIO code. It does things right, including writing actual pseudo-random contents, which shows if the disk does some "de-duplication" (aka "optimize for benchmarks):

   [ https://github.com/axboe/fio/ ]

   Anything else is suspect - forget about bonnie or other traditional tools.

Source: comment left on Google Plus to Greg Kroah-Hartman by Linus Torvalds.

Answer 3

with the IOPS tool

If you can't be bothered to read all this I'd just recommend the IOPS tool. It will tell you real-world speed depending on block size.

---

Otherwise - when doing an IO benchmark I would look at the following things:

• blocksize/cache/IOPS/direct vs buffered/async vs sync
• read/write
• threads
• latency

• CPU utilization

• Which blocksize will you use: If you want to read/write 1 GB from/to disk this will be quick if you do one I/O operation. But if your application needs to write in 512 byte chunks all over the harddisk in non-sequential pieces (called random I/O although it is not random) this will look differently. Now, databases will do random I/O for the data volume and sequential I/O for the log volume due to their nature. So, first you need to become clear what you want to measure. If you want to copy large video files that's different than if you want to install Linux.

  This blocksize is effecting the count of I/O operations you do. If you do e.g. 8 sequential read (or write, just not mixed) operations the I/O scheduler of the OS will merge them. If it does not, the controller's cache will do the merge. There is practically no difference if you read 8 sequential blocks of 512 bytes or one 4096 bytes chunk. One exception - if you manage to do direct sync IO and wait for the 512 bytes before you request the next 512 bytes. In this case, increasing the block size is like adding cache.

  Also you should be aware that there is sync and async IO: With sync IO you will not issue the next IO request before the current one returns. With async IO you can request e.g. 10 chunks of data and then wait as they arrive. Disctinct database threads will typically use sync IO for log and async IO for data. The IOPS tool takes care of that by measuring all relevant block sizes starting from 512 bytes.

• Will you read or write: Usually reading is faster than writing. But note that caching works quite a different way for reads and writes:

  • For writes, the data will be handed over to the controller and if it caches, it will acknowledge before the data is on disk unless the cache is full. Using the tool iozone you can draw beautiful graphs of plateaus of cache effects (CPU cache effect and buffer cache effect). The caches becomes less efficient the more has been written.

  • For reads, read data is held in cache after the first read. The first reads take longest and caching becomes more and more effective during uptime. Noteable caches are the CPU cache, the OS' file system cache, the IO controller's cache and the storage's cache. The IOPS tool only measures reads. This allows it to "read all over the place" and you do not want it to write instead of read.

• How many threads will you use: If you use one thread (using dd for disk benchmarks) you will probably get a much worse performance than with several threads. The IOPS tool takes this into account and reads on several threads.

• How important is latency for you: Looking at databases, IO latency becomes enormously important. Any insert/update/delete SQL command will be written into the database journal ("log" in database lingo) on commit before it is acknowledged. This means the complete database may be waiting for this IO operation to be completed. I show here how to measure the average wait time (await) using the iostat tool.

• How important is CPU utilization for you: Your CPU may easily become the bottleneck for your application's performance. In this case you must know how much CPU cycles get burned per byte read/written and optimize into that direction. This can mean to decide for/against PCIe flash memory depending on your measurement results. Again the iostat tool can give you a rough estimation on CPU utilization by your IO operations.

Answer 4

You can use fio - the Multithreaded IO generation tool. It is packaged by several distributions, e.g. Fedora 25, Debian and OpenCSW.

The fio tool is very flexible, it can be easily used to benchmark various IO scenarios - including concurrent ones. The package comes with some example configuration files (cf. e.g. /usr/share/doc/fio/examples). It properly measures things, i.e. it also prints the standard deviation and quantitative statistics for some figures. Things some other popular benchmarking tools don't care about.

A simple example (a sequence of simple scenarios: sequential/random X read/write):

$ cat fio.cfg
[global]
size=1g
filename=/dev/sdz

[randwrite]
rw=randwrite

[randread]
wait_for=randwrite
rw=randread
size=256m

[seqread]
wait_for=randread
rw=read

[seqwrite]
wait_for=seqread
rw=write

The call:

# fio -o fio-seagate-usb-xyz.log fio.cfg
$ cat fio-seagate-usb-xyz.log
[..]
randwrite: (groupid=0, jobs=1): err= 0: pid=11858: Sun Apr  2 21:23:30 2017
  write: io=1024.0MB, bw=16499KB/s, iops=4124, runt= 63552msec
clat (usec): min=1, max=148280, avg=240.21, stdev=2216.91
 lat (usec): min=1, max=148280, avg=240.49, stdev=2216.91
clat percentiles (usec):
 |  1.00th=[    2],  5.00th=[    2], 10.00th=[    2], 20.00th=[    7],
 | 30.00th=[   10], 40.00th=[   11], 50.00th=[   11], 60.00th=[   12],
 | 70.00th=[   14], 80.00th=[   16], 90.00th=[   19], 95.00th=[   25],
 | 99.00th=[ 9408], 99.50th=[10432], 99.90th=[21888], 99.95th=[38144],
 | 99.99th=[92672]
bw (KB  /s): min= 7143, max=371874, per=45.77%, avg=15104.53, stdev=32105.17
lat (usec) : 2=0.20%, 4=15.36%, 10=6.58%, 20=69.35%, 50=6.07%
lat (usec) : 100=0.49%, 250=0.07%, 500=0.01%, 750=0.01%
lat (msec) : 4=0.01%, 10=1.20%, 20=0.54%, 50=0.08%, 100=0.03%
lat (msec) : 250=0.01%
  cpu          : usr=1.04%, sys=4.79%, ctx=4977, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued    : total=r=0/w=262144/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
 latency   : target=0, window=0, percentile=100.00%, depth=1
randread: (groupid=0, jobs=1): err= 0: pid=11876: Sun Apr  2 21:23:30 2017
  read : io=262144KB, bw=797863B/s, iops=194, runt=336443msec
[..]
bw (KB  /s): min=  312, max= 4513, per=15.19%, avg=591.51, stdev=222.35
[..]

Note that the [global] section has global defaults that can be overriden by other sections. Each section describes a job, the section name is the job name and can be freely choosen. By default, different jobs are started in parallel, thus the above example explicitly serializes the job execution with the wait_for key. Also, fio uses a block size of 4 KiB - which can be changed, as well. The example directly uses the raw device for read and write jobs, thus, make sure that you use the right device. The tool also supports using a file/directory on existing filesystems.

Other Tools

The hdparm utility provides a very simple read benchmark, e.g.:

# hdparm -t -T /dev/sdz

It's not a replacement for a state-of-the-art benchmarking tool like fio, it just should be used for a first plausibility check. For example, to check if the external USB 3 drive is wrongly recognized as USB 2 device (you would see ~ 100 MiB/s vs. ~ 30 MiB/s rates then).

Answer 5

If you have installed PostgreSQL, you can use their excellent pg_test_fsync benchmark. It basically test your write sync performance.

On Ubuntu you find it here: /usr/lib/postgresql/9.5/bin/pg_test_fsync

The great thing about it, is that this tool will show you why enterprise SSD's are worth the extra $.

Answer 6

As pointed out here here, you can use gnome-disks (if you use Gnome).

Click to the the drive that you want to test and the click on "Additional partition options" (the wheels). Then Benchmark Partition. You'll get average read/write in MB/s and average access times in milliseconds. I found that very comfortable.

Answer 7

It's a little crude, but this works in a pinch:

find <path> -type f -print0 | cpio -0o >/dev/null

You can do some interesting things with this technique, including caching all the /lib and /usr/bin files. You can also use this as part of a benchmarking effort:

find / -xdev -type f -print0 | 
sort -R --from0-file=- | 
timeout "5m" cpio -0o >/dev/null

All filenames on root are found, sorted randomly, and copy them into cache for up to 1 minute. The output from cpio tells you how many blocks were copied. Repeat 3 times to get an average of blocks-per-minute. (Note, the find/sort operation may take a long time -- much longer than the copy. It would be better to cache the find / sort and use split to get a sample of files.)