What's the difference between direct-IO ("losetup --direct-io") and "mount -o sync" for loop-devices?

Question

I understand that without use of any flags, there are two caches involved with loop-devices. Once a page cache when writing to the file-system within the loop-device, and then again to the page cache when writing to the file-system of the underlying file. Is that correct?

The documentation of direct-IO says

--direct-io[=on|off] Enable or disable direct I/O for the backing file. The optional argument can be either on or off. If the argument is omitted, it defaults to on.

Does this mean that with this option, there is only a single cache, namely the cache for the filesystem of the underlying file?

The documentation of -o sync says

-o sync All I/O to the filesystem should be done synchronously. In the case of media with a limited number of write cycles (e.g. some flash drives), sync may cause life-cycle shortening.

Since in the past I had serious performance problems with sync filesystems on loop-mounts, and based on the warning about life-cycle shortening, I take this to mean that there's no cache anymore, not even for the underlying file. Data is written directly to the hardware.

Is this correct, or are matters more complicate?

Answer 1

Disclaimer: this may not be 100% accurate. It is my understanding.

In general, when opening a file, you can have O_SYNC or O_DIRECT.

Sync means that data are synced to the disk after each write. That probably means: Copy from userspace to kernel space, then perform sync on the fs which ultimately perform some kind of sync on the disks after syncing all the way to the bottom.

Direct-io means that data are written directly to the disk, bypassing internal buffers when possible.

Differences:

• The cache is still used with sync and can speed up reads. It also requires additional memory copies
• Sync will almost certainly result in some kind of disk syncs, which means that data from other partitions will also be synced (because you may have mounted only one partition with "sync" but have others without "sync")
• I believe that direct-io doesn't auto-sync, which means that (a) data from other partitions are not synced and (b) that the drive's internal cache is not flushed
• Some other stuff like using multiples of 512 bytes

Note: In a more complicated setup, where you have layers over layers of things (e.g. hard disks -> software raid -> encryption -> lvm -> filesystem), the syncs travel all the way down to the hardware.

Mounting a loopback device with direct-io would mean that:

• Caches are not used for the loopback device (but are used for the rest of the stack)
• Data are not implicitly synced to the disk, so you'll get better performance
• You don't waste memory