0

I have the below shell script and I wonder whether oflag's direct does the sync automatically or it is explicitly required

dd bs=10M oflag=direct,sync of=ofile.bin

Also what is the difference in saying oflag=sync and conv=sync or conv=fsync?

What impact does it bring if I change the code as below?

dd bs=10M conv=fsync oflag=direct of=ofile.bin
  • direct uses direct i/o, without buffer cache (check your blocksize though), oflag=sync uses synchronous data/metadata processing. conv=fsync uses sync() call after processing, conv=sync pads input blocks with zeroes to the blocksize. – stoney Mar 26 at 11:04
1

We can probably rule out conv=sync to start with. It does something rather different, which I expect you do not want :-).

pad every input block with NULs to ibs-size; when used with block or unblock, pad with spaces rather than NULs


oflag=direct does not sync automatically on its own.[*]

conv=fsync differs from oflag=sync. oflag=sync effectively syncs after each output block. conv=fsync does one sync at the end.

The end result is the same, but the performance along the way is different :-).

  1. oflag=sync could be significantly slower. You can mitigate this, by increasing the block size.

  2. If device-specific caches are large[1], this will affect the progress reported e.g. by the status=progress option.

  3. If you do not use oflag=direct, then large amounts of writes can build up in the system page cache. This build-up will affect the progress you see[2]. But also, Linux sometimes responds badly to the build-up, and degrades performance for all devices[3].


[1] "Apparently your hardware has hundreds of megabytes of cache... In my case, it is because the kernel is [actually running inside a virtual machine]". https://unix.stackexchange.com/a/420300/29483

[2] Why does a gunzip to dd pipeline slow down at the end?

[3] System lags when doing large R/W operations on external disks

[*] When writing directly to a block device node, Linux syncs the block device when it is closed (and is not open by any other program). I really do not recommend relying on this. However it explained something for me. Sometimes I see people who do not explicitly sync when writing to a block device, and it seems to work OK for them :-). See: Block device cache v.s. a filesystem

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.