8

I have a USB key (PQI U822V-Speedy 32G) that I am trying to benchmark quick'n'dirty on Linux. I'm testing write bandwith.

dd on raw partition

I created a partition starting at sector 2048, then did a 4 GB sequential write: 

dd if=/dev/zero of=/dev/sdb1 bs=1M count=4096

I get ~22 MB/s.

I also tried several (4) dd running in parallel like the one above but using count=1024 and the seek= option to write to different areas of the drive. Same results.

dd on filesystem

However, when I format the sdb1 partition with either ext4 or NTFS, and copy big files to it (either real or /dev/zero), like this: 

time dd if=/dev/zero of=/media/USBKEY/file.bin bs=1M count=4096 ; time sync

I do achieve > 66 MB/s as advertised by the manufacturer. Of course, I considered the sync duration just after the copy.

Why is there such a big performance difference?

Improve this question
4
  • 1
    Your file system is probably caching that - what is the difference with o/iflag=nocache? There are also direct and sync options which are relevant - i always use sync at least with usb just so i can be sure not to pull the thumbdrive too soon. And if you are syncing then maybe see if you can get the same speed raw with nonblock
    – mikeserv
    Apr 7, 2014 at 17:58
  • The filesystem will be writing the file somewhere other than the start of the partition. Perhaps there is something wrong with the area near the start? Try a single dd but with higher seek arguments to write to different parts of the disk.
    – psusi
    Apr 8, 2014 at 0:17
  • Interesting. Could you repeat the experiment with various block-sizes (various powers of two, say 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K) but identical total data? I'm very curious if that will make a difference.
    – marcelm
    Feb 25, 2016 at 11:07
  • I've found Linux slow at writing to USB keys even with a FAT32 filesystem. I've found that using rsync --bwlimit when copying files can actually help. It seems to be something like a traffic jam: if Linux tries to ram too much data down the USB device's throat, it actually goes slower, and stays slow until I suspend the copy while the buffer flushes. (this is with cheap USB sticks).
    – Peter Cordes
    Feb 25, 2016 at 11:43

4 Answers 4

Reset to default
1

Now that I look again, I realized you said this was a usb key ( flash drive ) not a hard drive. Flash memory can only be erased in large blocks, and individual sectors can not be written without erasing them ( and the whole block they are in ) first. Since software expects to be able to write wherever it wants on the disk at any time, the disk has translation logic in it to transparently handle the erasing. How this is done has a dramatic affect on write performance. Many devices use an algorithm for most of the disk that handles sequential writes very well, but sucks at random writes. The area near the start of the disk is normally used by the FAT in the FAT filesystem they come preformatted with, and this area is randomly written to frequently, so they use a different algorithm in this area that is slower at sequential writes, but not terrible at random writes.

Thus, I am now pretty sure that my initial guess I added as a comment was right. What you are seeing when you write to the filesystem is the performance of the rest of the disk, and when you dd at offset zero, you are writing to the fat area. If you seek the dd destination a few hundred mb in, it should speed up considerably.

Improve this answer
7
  • 3
    It's a bold assertion to claim any level of certainty despite no update from the asker and, 0 cited doumentation and no evidence of any of your own tests. Too bold, I think.
    – mikeserv
    Apr 9, 2014 at 0:13
  • 1
    @mikeserv, see ccs.neu.edu/home/pjd/papers/fast10-flash.pdf, and idke.ruc.edu.cn/people/dazhou/Papers/AsurveyFlash-JSA.pdf
    – psusi
    Apr 9, 2014 at 1:09
  • 1
    Thank you. You might enjoy this: bunniestudios.com/blog/?p=3554 They did an hour long talk here: youtube.com/watch?feature=player_embedded&v=r3GDPwIuRKI It won't let me reverse my vote. It's been too long. And you should put that in your answer - but I owe you one. No, I don't. We're even.
    – mikeserv
    Apr 9, 2014 at 1:20
  • @psusi: so it does get better than dd to the raw block device and in the performed benchmark, using a file system was the enabler for this, wasn't it?
    – Bananguin
    Apr 9, 2014 at 11:36
  • 1
    @Bananguin, your argument is equivalent to getting in a car accident on the way to work, and claiming that working causes car accidents. It may be true that had you not gotten in the car at that time to go to work, you would not have been in that particular accident, but you can not conclude that working causes car accidents. It just so happens that is where you were going when you got in one. Here, it just so happens that the filesystem put the data in part of the disk that is faster than where the OP dd'd to. That doesn't mean that the filesystem magically is faster than dd in general.
    – psusi
    Apr 10, 2014 at 14:39
0

Your measurement results can be explained with the kernel architecture. Using filesystem access will unlease the full potential of the kernel with all buffers and optimizations that it can do. Especially the buffers will speed up your benchmark (b/c the kernel is 100% a_A_syncronous). dd on a device file does not use any/much of this.

Improve this answer
9
  • 3
    Writing directly to a block device does use the cache.
    – psusi
    Apr 8, 2014 at 0:14
  • @psusi - maybe. But if there's no target file system, and dd is writing out sequential blocks of equal size, then which file system caches the write?
    – mikeserv
    Apr 8, 2014 at 2:19
  • 1
    @mikeserv, the block device interface caches direct access to block devices.
    – psusi
    Apr 8, 2014 at 3:44
  • @psusi so the physical device cache then or is there some in-kernel caching layer as well?
    – mikeserv
    Apr 8, 2014 at 3:47
  • 2
    @mikeserv The block device interface is part of the kernel.
    – Chris Down
    Apr 8, 2014 at 4:06
0

Try using hdparm instead to benchmark a drives performance with and without using any caching:

$ sudo hdparm -tT /dev/sda1

/dev/sda1:
 Timing cached reads:   6314 MB in  2.00 seconds = 3157.61 MB/sec
 Timing buffered disk reads: 244 MB in  3.04 seconds =  80.26 MB/sec
Improve this answer
2
  • I'm benchmarking writes, see question title.
    – Totor
    Apr 8, 2014 at 14:12
  • 1
    If I recall correctly hdparm can also do (destructive) writes.
    – Thorbjørn Ravn Andersen
    Jul 10, 2014 at 19:46
-2

This happens due to the optimization of writing Sparse files in filesystem.

When you do dd if=/dev/zero to raw device, the zero blocks are actually written to disk.

However when you write them to a file, the filesystem ignores writing the data and saves just the metadata. This results in very few blocks being written to disk. The file can be seen as a big hole, which contains nothing.

To test performance in this manner, use /dev/urandom as input file ( dd if=/dev/urandom ). That will force filesystem to write random data into disk.

Improve this answer
1
  • 4
    Wrong, for three reasons. 1/ I said I tested with real files: "either real or /dev/zero"; 2/ on many machines, /dev/urandom is much slower than storage devices; 3/ most often, filesystem optimisations like sparse files must be explicitly asked for, see cp --sparse and others...
    – Totor
    Apr 8, 2014 at 15:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .