Has 'sync' been proved to be necessary when writing to a block device with 'dd'?

Question

Every time I've written a file to an empty raw block device, e.g.

# dd if=image.iso of=/dev/sdb status=progress

I've never used any type of sync (i.e. sync; conv=fsync; conv=fdatasync; oflag=sync; oflag=dsync).

I've noticed that dd doesn't ever exit until all writing has finished.

I always verify this using Conky's I/O facility and grep Dirty /proc/meminfo. Also, the checksum of the device always matches that of the file that was written to it. So I am always 100% sure that the whole file was written to the device.

I've written files to an ext4 volume to compare. For example using:

$ dd if=/dev/urandom of=~/file bs=1M count=50 iflag=fullblock

When writing to an ext4 volume, after dd exits there is always a delay of about 20 seconds before the data is actually written to the disk.

Many people advocate using the sync command after the dd command, or including one of the several sync options in the dd command when writing to a block device. E.g. here and here. However, I've not known anyone actually prove that it is necessary.

One of the comments on this page is:

sync is pointless here [i.e. writing directly to /dev/sdX]. It only affects file system operations.

Five people have upvoted this comment and it is consistent with my experience.

So when writing to a block device, are the any circumstances in which dd will exit before all writing has completely finished? Has this actually ever happened to anyone?

What about other writing options, such as cp and cat? Can they exit before writing to a block device has finished?

Answer 1

It's about time to answer this.

For such a long time, I've been thinking that only writes to filesystems were cached, not block devices, only to discover that I was wrong : writes to block devices are cached.

Strictly speaking I don't have an official source on this, and will update my answer as I get one, but you can find some info here : linux - Block device cache v.s. a filesystem - Unix & Linux Stack Exchange. Sorry, I just noticed you saw that question too !

In fact, I had the problem of dd'ing an image to a USB stick, eject it after completion and then discover it was corrupted. It shouldn't have behaved like that since it seems a complete sync is done on the last close() on the device, hence I suppose either I got data corruption on my device, or I missed a process that still had the block device opened, but anyway I'm not taking chances any more.

So yes, adding sync options to dd when writing to a block device doesn't seem to be an urban legend. I think conv=fdatasync should be enough and best performing (no file metadata is added after all, and only syncing at the end of the process is really needed), but the extremists among us may prefer oflag=sync (complete data + metadata sync after each write). Check man 2 open and man 2 fdatasync for more insight.

According to what I've read (but nothing official as of now, unfortunately), cp and cat to a block device can indeed exit before sync if some other process still has the block device open, unless explicitly requested with some SYNC flags when the device is open()ed or with a sync system call such as fdatasync(), fsync() or more exaggeratingly sync(), which none of these commands does...

Answer 2

So I've been testing something similar and the tellall is the stacktrace on the dd process.

I did a dd directly to the block device (note WITHOUT oflag=direct which bypasses the page cache) including status=progress and what I get is: the status will say really quick speeds 187MB/s (as the amount of time to write to the page cache is small) but then dd calls blkdev_close on the block device, and then you wait, and when it finishes, you get the speed based on the total time to write including the flushing on the close call, which is therefore much slower.

echo;echo;echo;date; dd if=/dev/urandom bs=1M of=/dev/mockdevice status=progress; date

Thu 08 Oct 2020 10:47:36 AM EDT
3172990976 bytes (3.2 GB, 3.0 GiB) copied, 17 s, 187 MB/s
dd: error writing '/dev/mockdevice': No space left on device
3073+0 records in
3072+0 records out
3221225472 bytes (3.2 GB, 3.0 GiB) copied, 1681.94 s, 1.9 MB/s
Thu 08 Oct 2020 11:15:38 AM EDT

Here's the stack trace while it's flushing after the dd completes (in my case, it errored on disk full)

cat /proc/1130961/stack
[<0>] __lock_page+0x128/0x230
[<0>] write_cache_pages+0x354/0x4b0
[<0>] generic_writepages+0x57/0x90
[<0>] blkdev_writepages+0xe/0x10
[<0>] do_writepages+0x43/0xd0
[<0>] __filemap_fdatawrite_range+0xd5/0x110
[<0>] filemap_write_and_wait+0x44/0xa0
[<0>] __blkdev_put+0x72/0x1e0
[<0>] blkdev_put+0x4e/0xe0
[<0>] blkdev_close+0x26/0x30
[<0>] __fput+0xcc/0x260
[<0>] ____fput+0xe/0x10
[<0>] task_work_run+0x8f/0xb0
[<0>] exit_to_usermode_loop+0x131/0x160
[<0>] do_syscall_64+0x163/0x190
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

It's also worth noting that when you fill up the pagecache with dd, the page cache flushing to the block device does so one page at a time. Filesystems will send bios with lots of pages to flush in one bio, but the page cache flusher in the stacktrace above, only seems to push out one page per bio.

So from this, my experience is that you do not have to call sync, because the blkdev_close actually flushes all dirty cache pages to the block device before it returns and lets dd complete.