1

On my system df -h reports that the root file system is full (100% Available 0) and also btrfs fi usage / tells me that all of the space is allocated. After a restart it could not even bring up the necessary services, so I'm now in recovery mode.

I tried to balance the system as shown here btrfs ERROR: error during balancing - No space left on device. For btrfs balance start -dusage=27 / I get:

Done, had to relocate 0 out of 56 chunks`

For btrfs balance start -dusage=28 / I get:

ERROR: error during balancing '/': No space left on device

So I started deleting files in /var/log/ but even after removing several 100 MB df -h reports that no single bit is available.

I suspect, that there is some service filling up the space the moment it gets free. But since I'm in recovery mode, there should not be so many services.

How can I find out, what is filling the space immediately after removing the files?

How can I ensure to free space, that can be used by btrfs to rebalance?

My system is debian 10.3 (buster), filesystem btrfs and I have docker installed, which I suspect being part of the problem but which is not running in recovery mode, so …

4
  • 3
    Have you checked if snapshots are filling up your filesystem? That's a common btrfs issue.
    – eblock
    Apr 24 '20 at 12:51
  • Yes, there are tons of docker snapshots in /var/lib/docker/subvolumes . Can I just delete them, aren't they needed by docker, and will this solve the issue? Apr 24 '20 at 13:02
  • 1
    As eblock said, you need to look for snapshots and delete as many as you can. You can list subvolumes AND snapshots with this: btrfs subvol list / Don't bother using df for this; it won't help. If you have files which are heavy fragmented (virtual disk images, databases, systemd journal, etc) they can eat up your metadata chunks. I recommend adding the output of btrfs fi usage / to you question. Apr 24 '20 at 13:03
  • As for docker snapshots, you should consult it's documentation about how it handles images. Apr 24 '20 at 13:04
3

Historically, some out-of-space situations on btrfs required temporarily adding an additional disk to the volume, as the COW nature of btrfs may require some temporary space to save a modified version of blocks before being able to free the old version. Though, I understand that many of these situations have been fixed by the btrfs developers.

For emergency disk usage profiling on btrfs volumes, I would like to recommend btdu, a tool I've written:

https://github.com/CyberShadow/btdu

As it only needs to collect 100 samples to gain 1% resolution, it is able to identify large space hogs in seconds.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.