I've seen commands to benchmark one's HDD such as this using
$ time sh -c "dd if=/dev/zero of=ddfile bs=8k count=250000 && sync"
Are there better methods to do so than this?
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.Sign up to join this community
I usually use
hdparm to benchmark my HDD's. You can benchmark both the direct reads and the cached reads. You'll want to run the commands a couple of times to establish an average value.
Here's a direct read.
$ sudo hdparm -t /dev/sda2 /dev/sda2: Timing buffered disk reads: 302 MB in 3.00 seconds = 100.58 MB/sec
And here's a cached read.
$ sudo hdparm -T /dev/sda2 /dev/sda2: Timing cached reads: 4636 MB in 2.00 seconds = 2318.89 MB/sec
-t Perform timings of device reads for benchmark and comparison purposes. For meaningful results, this operation should be repeated 2-3 times on an otherwise inactive system (no other active processes) with at least a couple of megabytes of free memory. This displays the speed of reading through the buffer cache to the disk without any prior caching of data. This measurement is an indication of how fast the drive can sustain sequential data reads under Linux, without any filesystem overhead. To ensure accurate measurements, the buffer cache is flushed during the processing of -t using the BLKFLSBUF ioctl. -T Perform timings of cache reads for benchmark and comparison purposes. For meaningful results, this operation should be repeated 2-3 times on an otherwise inactive system (no other active processes) with at least a couple of megabytes of free memory. This displays the speed of reading directly from the Linux buffer cache without disk access. This measurement is essentially an indication of the throughput of the processor, cache, and memory of the system under test.
I too have used
dd for this type of testing as well. One modification I would make to the above command is to add this bit to the end of your command,
; rm ddfile.
$ time sh -c "dd if=/dev/zero of=ddfile bs=8k count=250000 && sync"; rm ddfile
This will remove the
ddfile after the command has completed. NOTE:
ddfile is a transient file that you don't need to keep, it's the file that
dd is writing to (
of=ddfile), when it's putting your HDD under load.
If you need more rigorous testing of your HDD's you can use Bonnie++.
Are there better methods [than dd] to [benchmark disks]?
Yes but they will take longer to run and require knowledge of how to interpret the results - there's no single number that will tell you everything in one go because the following influence the type of test you should run:
And so on.
Here's a short list of tools with easiest to run at the top and difficult/more thorough/better nearer the bottom:
Greg - get Jens' FIO code. It does things right, including writing actual pseudo-random contents, which shows if the disk does some "de-duplication" (aka "optimize for benchmarks):
Anything else is suspect - forget about bonnie or other traditional tools.
If you can't be bothered to read all this I'd just recommend the IOPS tool. It will tell you real-world speed depending on block size.
Otherwise - when doing an IO benchmark I would look at the following things:
Which blocksize will you use: If you want to read/write 1 GB from/to disk this will be quick if you do one I/O operation. But if your application needs to write in 512 byte chunks all over the harddisk in non-sequential pieces (called random I/O although it is not random) this will look differently. Now, databases will do random I/O for the data volume and sequential I/O for the log volume due to their nature. So, first you need to become clear what you want to measure. If you want to copy large video files that's different than if you want to install Linux.
This blocksize is effecting the count of I/O operations you do. If you do e.g. 8 sequential read (or write, just not mixed) operations the I/O scheduler of the OS will merge them. If it does not, the controller's cache will do the merge. There is practically no difference if you read 8 sequential blocks of 512 bytes or one 4096 bytes chunk. One exception - if you manage to do direct sync IO and wait for the 512 bytes before you request the next 512 bytes. In this case, increasing the block size is like adding cache.
Also you should be aware that there is sync and async IO: With sync IO you will not issue the next IO request before the current one returns. With async IO you can request e.g. 10 chunks of data and then wait as they arrive. Disctinct database threads will typically use sync IO for log and async IO for data. The IOPS tool takes care of that by measuring all relevant block sizes starting from 512 bytes.
Will you read or write: Usually reading is faster than writing. But note that caching works quite a different way for reads and writes:
For writes, the data will be handed over to the controller and if it caches, it will acknowledge before the data is on disk unless the cache is full. Using the tool iozone you can draw beautiful graphs of plateaus of cache effects (CPU cache effect and buffer cache effect). The caches becomes less efficient the more has been written.
For reads, read data is held in cache after the first read. The first reads take longest and caching becomes more and more effective during uptime. Noteable caches are the CPU cache, the OS' file system cache, the IO controller's cache and the storage's cache. The IOPS tool only measures reads. This allows it to "read all over the place" and you do not want it to write instead of read.
How many threads will you use: If you use one thread (using dd for disk benchmarks) you will probably get a much worse performance than with several threads. The IOPS tool takes this into account and reads on several threads.
How important is latency for you: Looking at databases, IO latency becomes enormously important. Any insert/update/delete SQL command will be written into the database journal ("log" in database lingo) on commit before it is acknowledged. This means the complete database may be waiting for this IO operation to be completed. I show here how to measure the average wait time (await) using the iostat tool.
How important is CPU utilization for you: Your CPU may easily become the bottleneck for your application's performance. In this case you must know how much CPU cycles get burned per byte read/written and optimize into that direction. This can mean to decide for/against PCIe flash memory depending on your measurement results. Again the iostat tool can give you a rough estimation on CPU utilization by your IO operations.
You can use
fio - the Multithreaded IO generation tool. It is packaged by several distributions, e.g. Fedora 25, Debian and OpenCSW.
The fio tool is very flexible, it can be easily used to benchmark various IO
scenarios - including concurrent ones. The package comes with some example
configuration files (cf. e.g.
/usr/share/doc/fio/examples). It properly measures things, i.e. it also prints the
standard deviation and quantitative statistics for some figures. Things some
other popular benchmarking tools don't care about.
A simple example (a sequence of simple scenarios: sequential/random X read/write):
$ cat fio.cfg [global] size=1g filename=/dev/sdz [randwrite] rw=randwrite [randread] wait_for=randwrite rw=randread size=256m [seqread] wait_for=randread rw=read [seqwrite] wait_for=seqread rw=write
# fio -o fio-seagate-usb-xyz.log fio.cfg $ cat fio-seagate-usb-xyz.log [..] randwrite: (groupid=0, jobs=1): err= 0: pid=11858: Sun Apr 2 21:23:30 2017 write: io=1024.0MB, bw=16499KB/s, iops=4124, runt= 63552msec clat (usec): min=1, max=148280, avg=240.21, stdev=2216.91 lat (usec): min=1, max=148280, avg=240.49, stdev=2216.91 clat percentiles (usec): | 1.00th=[ 2], 5.00th=[ 2], 10.00th=[ 2], 20.00th=[ 7], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 11], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 16], 90.00th=[ 19], 95.00th=[ 25], | 99.00th=[ 9408], 99.50th=, 99.90th=, 99.95th=, | 99.99th= bw (KB /s): min= 7143, max=371874, per=45.77%, avg=15104.53, stdev=32105.17 lat (usec) : 2=0.20%, 4=15.36%, 10=6.58%, 20=69.35%, 50=6.07% lat (usec) : 100=0.49%, 250=0.07%, 500=0.01%, 750=0.01% lat (msec) : 4=0.01%, 10=1.20%, 20=0.54%, 50=0.08%, 100=0.03% lat (msec) : 250=0.01% cpu : usr=1.04%, sys=4.79%, ctx=4977, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=1 randread: (groupid=0, jobs=1): err= 0: pid=11876: Sun Apr 2 21:23:30 2017 read : io=262144KB, bw=797863B/s, iops=194, runt=336443msec [..] bw (KB /s): min= 312, max= 4513, per=15.19%, avg=591.51, stdev=222.35 [..]
Note that the
[global] section has global defaults that can be overriden by
other sections. Each section describes a job, the section name is the job name
and can be freely choosen. By default, different jobs are started in parallel,
thus the above example explicitly serializes the job execution with the
wait_for key. Also, fio uses a block size of 4 KiB - which can be changed, as
well. The example directly uses the raw device for read and write jobs, thus,
make sure that you use the right device. The tool also supports using a
file/directory on existing filesystems.
hdparm utility provides a very simple read benchmark, e.g.:
# hdparm -t -T /dev/sdz
It's not a replacement for a state-of-the-art benchmarking tool like fio, it just should be used for a first plausibility check. For example, to check if the external USB 3 drive is wrongly recognized as USB 2 device (you would see ~ 100 MiB/s vs. ~ 30 MiB/s rates then).
If you have installed PostgreSQL, you can use their excellent pg_test_fsync benchmark. It basically test your write sync performance.
On Ubuntu you find it here:
The great thing about it, is that this tool will show you why enterprise SSD's are worth the extra $.
As pointed out here here, you can use
gnome-disks (if you use Gnome).
Click to the the drive that you want to test and the click on "Additional partition options" (the wheels). Then
Benchmark Partition. You'll get average read/write in MB/s and average access times in milliseconds. I found that very comfortable.
It's a little crude, but this works in a pinch:
find <path> -type f -print0 | cpio -0o >/dev/null
You can do some interesting things with this technique, including caching all the
/usr/bin files. You can also use this as part of a benchmarking effort:
find / -xdev -type f -print0 | sort -R --from0-file=- | timeout "5m" cpio -0o >/dev/null
All filenames on root are found, sorted randomly, and copy them into cache for up to 1 minute. The output from cpio tells you how many blocks were copied. Repeat 3 times to get an average of blocks-per-minute. (Note, the find/sort operation may take a long time -- much longer than the copy. It would be better to cache the find / sort and use
split to get a sample of files.)