dd command oflag direct and sync flags

Question

I have the below shell script and I wonder whether oflag's direct does the syncing automatically or if it is explicitly required:

dd bs=10M oflag=direct,sync of=ofile.bin

Also what is the difference in saying oflag=sync and conv=sync and conv=fsync?

What impact does it have if I change the code to the line below?

dd bs=10M conv=fsync oflag=direct of=ofile.bin

Answer 1

We can probably rule out conv=sync to start with. It does something rather different, which I expect you do not want :-).

pad every input block with NULs to ibs-size; when used with block or unblock, pad with spaces rather than NULs

---

oflag=direct does not sync automatically on its own.[*]

conv=fsync differs from oflag=sync. oflag=sync effectively syncs after each output block. conv=fsync does one sync at the end.

The end result is the same, but the performance along the way is different :-).

1. oflag=sync could be significantly slower. You can mitigate this, by increasing the block size.

2. If device-specific caches are large[1], this will affect the progress reported e.g. by the status=progress option.

3. If you do not use oflag=direct, then large amounts of writes can build up in the system page cache. This build-up will affect the progress you see[2]. But also, Linux sometimes responds badly to the build-up, and degrades performance for all devices[3].

---

[1] "Apparently your hardware has hundreds of megabytes of cache... In my case, it is because the kernel is [actually running inside a virtual machine]". https://unix.stackexchange.com/a/420300/29483

[2] Why does a gunzip to dd pipeline slow down at the end?

[3] System lags when doing large R/W operations on external disks

[*] When you write directly to a block device node, Linux syncs the block device when it is closed (and is not open by any other program). See: Block device cache v.s. a filesystem. Sometimes I see people who do not use an explicit sync when writing to a block device. It will often seem to work OK... until it doesn't. So I recommend at least using conv=fsync.