4

I understand that without use of any flags, there are two caches involved with loop-devices. Once a page cache when writing to the file-system within the loop-device, and then again to the page cache when writing to the file-system of the underlying file. Is that correct?

The documentation of direct-IO says

--direct-io[=on|off] Enable or disable direct I/O for the backing file. The optional argument can be either on or off. If the argument is omitted, it defaults to on.

Does this mean that with this option, there is only a single cache, namely the cache for the filesystem of the underlying file?

The documentation of -o sync says

-o sync All I/O to the filesystem should be done synchronously. In the case of media with a limited number of write cycles (e.g. some flash drives), sync may cause life-cycle shortening.

Since in the past I had serious performance problems with sync filesystems on loop-mounts, and based on the warning about life-cycle shortening, I take this to mean that there's no cache anymore, not even for the underlying file. Data is written directly to the hardware.

Is this correct, or are matters more complicate?

Improve this question

1 Answer 1

Reset to default
2

Disclaimer: this may not be 100% accurate. It is my understanding.

In general, when opening a file, you can have O_SYNC or O_DIRECT.

Sync means that data are synced to the disk after each write. That probably means: Copy from userspace to kernel space, then perform sync on the fs which ultimately perform some kind of sync on the disks after syncing all the way to the bottom.

Direct-io means that data are written directly to the disk, bypassing internal buffers when possible.

Differences:

  • The cache is still used with sync and can speed up reads. It also requires additional memory copies
  • Sync will almost certainly result in some kind of disk syncs, which means that data from other partitions will also be synced (because you may have mounted only one partition with "sync" but have others without "sync")
  • I believe that direct-io doesn't auto-sync, which means that (a) data from other partitions are not synced and (b) that the drive's internal cache is not flushed
  • Some other stuff like using multiples of 512 bytes

Note: In a more complicated setup, where you have layers over layers of things (e.g. hard disks -> software raid -> encryption -> lvm -> filesystem), the syncs travel all the way down to the hardware.

Mounting a loopback device with direct-io would mean that:

  • Caches are not used for the loopback device (but are used for the rest of the stack)
  • Data are not implicitly synced to the disk, so you'll get better performance
  • You don't waste memory
Improve this answer
4
  • O_SYNC doesn’t cause full syncs, it’s equivalent to calling fsync after every write. Other dirty buffers on the same underlying device aren’t affected.
    – Stephen Kitt
    Oct 14, 2018 at 5:25
  • @StephenKitt but is O_SYNC the same thing as the "sync" option of mount? Anyway, I conducted a test with perf, hanging in on various trace points. It appears that even if both sync and direct-IO is enabled, the write system-call only causes a write to the loop device. All block-device writes I did see are on major=7, which is the loop-device. None to the block-device of the underlying file.
    – Johannes Schaub - litb
    Oct 14, 2018 at 10:22
  • @StephenKitt I'm not sure how that's possible in a journaled filesystem or a shared disk device. The only way to ensure that the disk has written things is to flush its cache. For a journaled FS, you'll probably have to cause a barrier, so it will also sync other things from the buffers. Maybe the FS cache won't be affected as a whole (note: buffers vs cache), but things below that should be. I'm not 100% sure what happens with raid/md and lvm, but I guess that a sync expands considerably.
    – V13
    Oct 14, 2018 at 14:30
  • @V13 that’s true of ext3 in data=ordered mode, but it’s not a necessity on all journaling file systems; I think ext4 doesn’t flush all outstanding writes, and I’d have to check but I think XFS and Btrfs don’t either. You’re right about on-device write caches; they have to be fully flushed. (I was referring to dirty buffers in the kernel sense, but that wasn’t clear.)
    – Stephen Kitt
    Oct 14, 2018 at 15:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .