5

On my system df -h reports that the root file system is full (100% Available 0) and also btrfs fi usage / tells me that all of the space is allocated. After a restart it could not even bring up the necessary services, so I'm now in recovery mode.

I tried to balance the system as shown here btrfs ERROR: error during balancing - No space left on device. For btrfs balance start -dusage=27 / I get:

Done, had to relocate 0 out of 56 chunks`

For btrfs balance start -dusage=28 / I get:

ERROR: error during balancing '/': No space left on device

So I started deleting files in /var/log/ but even after removing several 100 MB df -h reports that no single bit is available.

I suspect, that there is some service filling up the space the moment it gets free. But since I'm in recovery mode, there should not be so many services.

How can I find out, what is filling the space immediately after removing the files?

How can I ensure to free space, that can be used by btrfs to rebalance?

My system is debian 10.3 (buster), filesystem btrfs and I have docker installed, which I suspect being part of the problem but which is not running in recovery mode, so …

5
  • 3
    Have you checked if snapshots are filling up your filesystem? That's a common btrfs issue.
    – eblock
    Commented Apr 24, 2020 at 12:51
  • Yes, there are tons of docker snapshots in /var/lib/docker/subvolumes . Can I just delete them, aren't they needed by docker, and will this solve the issue? Commented Apr 24, 2020 at 13:02
  • 1
    As eblock said, you need to look for snapshots and delete as many as you can. You can list subvolumes AND snapshots with this: btrfs subvol list / Don't bother using df for this; it won't help. If you have files which are heavy fragmented (virtual disk images, databases, systemd journal, etc) they can eat up your metadata chunks. I recommend adding the output of btrfs fi usage / to you question. Commented Apr 24, 2020 at 13:03
  • As for docker snapshots, you should consult it's documentation about how it handles images. Commented Apr 24, 2020 at 13:04
  • @white_gecko How did you end up resolving the problem? I'm facing the exact same problem. I'm in recovery mode and have seemingly no way to reclaim enough space to make my system usable again.
    – Mecha
    Commented Oct 12, 2022 at 12:16

1 Answer 1

7

Historically, some out-of-space situations on btrfs required temporarily adding an additional disk to the volume, as the COW nature of btrfs may require some temporary space to save a modified version of blocks before being able to free the old version. Though, I understand that many of these situations have been fixed by the btrfs developers.

For emergency disk usage profiling on btrfs volumes, I would like to recommend btdu, a tool I've written:

https://github.com/CyberShadow/btdu

As it only needs to collect 100 samples to gain 1% resolution, it is able to identify large space hogs in seconds.

1
  • Wow, I just have to say that this tool is awesome! Thanks a lot.
    – jpa
    Commented Nov 21, 2023 at 17:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .