6

Usually, block device drivers report correct size of the device, and it is possible to actually use all the "available" blocks. So, the filesystem knows how much it can write to such device in prior.
But in some special cases, like it is with dm-thin or dm-vdo devices, this statement is false. This kind of block devices can return ENOSPC error at any moment, if their underlying storage (which the upper-level FS knows nothing about) gets full.

Therefore, my question is, what happens in such scenario: an EXT4 filesystem is mounted r/w, in async mode (which is the default), and it is doing a massive amount of writes. The disk cache (dirty memory) gets involved too, and at the moment there is a lot of data to be written if user runs sync command.

But suddenly, the underlying block device of that EXT4 filesystem starts to refuse any writes due to "no space left". What will be the behavior of the filesystem?
Will it print errors and go to r/o mode aborting all the writes and possibly causing data loss? If not, will it just wait for space, periodically retrying writes and refusing new ones? In that case, what will happen to the huge disk cache, if other processes try to allocate lots of RAM? (On Linux, dirty memory is considered Available, isn't it?).
Considering worst scenario, if the disk cache was taking up most of the RAM at the moment of ENOSPC error (because admin has set vm.dirty_ratio too high), can the kernel crash or lock up? Or it will just make all processes which want allocate memory wait/hang? Finally, does the behavior differ across filesystems?
Thanks in advance.

4
  • backend layer has a way to tell this to the FS: I/O error.
    – A.B
    Feb 20 at 13:06
  • @A.B and the behavior will be similar to any I/O error - r/o remount and abortion of writes? (If the fs is mounted with errors=remount-ro)
    – melonfsck
    Feb 20 at 13:19
  • The FS/mount-dependent behavior will trigger upon receiving such error.
    – A.B
    Feb 20 at 13:23
  • Of course this should never happen. thin provisionned space has to be monitored. Example for VDO: access.redhat.com/documentation/en-us/red_hat_enterprise_linux/…
    – A.B
    Feb 20 at 13:24

2 Answers 2

10

When the block device overcommits its available data capacity like when using thin provisioning or has other reasons to not be able to accept more writes, like having a snapshot full, it has to send a message to what is writing to it. ENOSPC would make no sense in this context, so the error chosen is usually EIO (Input/output error).

UPDATE: actually LVM has configurable behavior. For Thin provisioned LV:

  • --errorwhenfull n (default): blocks for up to (configurable) 60 seconds, just as OP considered, then errors. Unless an automatic action is performed during these 60s chances are the result will be the same as immediate error.

    Note also that if the timeout is completely disabled:

    Disabling timeouts can result in the system running out of resources, memory exhaustion, hung tasks, and deadlocks. (The timeout applies to all thin pools on the system.)

  • --errorwhenfull y: immediately returns an error

If the "user" is a filesystem, it will react to I/O error the same as if this was caused by an actual media error, possibly depending on mount options (eg, for ext4 possible options are errors={continue|remount-ro|panic}). I can't tell for sure what happens to dirty data still in cache when one of the non-panic options is chosen. One could imagine it's either left in cache or will be lost, but one should assume it will be lost anyway.

As this is a severe result, such disk space should be actively monitored and once a threshold is reached, there should be either data freed or more actual space added so the overcommitted space never gets full. Same for snapshots, especially the non-thin-provisioned kind which uses more space over time: it should be removed when not needed anymore. There are even options to auto-increase the thin-provisioned space for emergencies (when the layer providing space to the thin provisioning layer can still provide more).

further references:

5

It depends on the filesystem (and possibly mount options) and the underlying storage.

In most cases, a failure to write to a block device due to it being over-committed on space will immediately be propagated up to the filesystem driver as an IO error. LVM has an option to delay this (mostly so that it’s auto-extension functionality has time to kick in), but it is disabled by default. QEMU has an option controlling behavior for this with sparse disk images, but by default it will propagate the error to the guest OS (it may also be configured to ignore the error or pause the VM). Most other stuff though will just throw the error up to the filesystem driver immediately.

From there, what happens depends on the filesystem. In almost all cases, the error will be propagated to userspace, though it will almost always be EIO and not ENOSPC (ENOSPC means the filesystem is out of space, but that’s not technically what’s wrong here, and the filesystem also can’t usually determine what caused the IO error it got from the lower layer so it normally has no way to tell that it’s due to the lower layer running out of space). ext4 will by default do nothing other than that, though depending on mount options (and things set by tune2fs) it may instead remount read-only, or it might trigger a kernel panic. BTRFS will remount read-only, and in some multi-device setups may also switch to degraded mode. I’m not sure about other filesystems (though I would expect XFS to remount read-only in this case).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .