2

This Debian server was running just fine until a week or so ago. Now it does not allow files to be allocated, despite there still being room.

  • The root volume is configured with LVM.
  • Kernel is Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.51-3 (2017-12-13) x86_64 GNU/Linux

A fsck and reboot did not help. Deleting some files did not help either.

df -i
        Filesystem                                         Inodes    IUsed       IFree IUse% Mounted on
    /dev/mapper/vg-root                       0        0           0     - /
    udev                                              2051270      380     2050890    1% /dev
    tmpfs                                             2053627      632     2052995    1% /run
    tmpfs                                             2053627        5     2053622    1% /dev/shm
    tmpfs                                             2053627        4     2053623    1% /run/lock
    tmpfs                                             2053627       13     2053614    1% /sys/fs/cgroup
    /dev/sda1                                           62248      328       61920    1% /boot
    tmpfs                                             2053627       13     2053614    1% /run/user/117
    tmpfs                                             2053627        4     2053623    1% /run/user/0
    tmpfs                                             2053627        4     2053623    1% /run/user/1000

Meanwhile, there is plenty room on the device

df -h
Filesystem                                     Size  Used Avail Use% Mounted on
/dev/mapper/vg-root              447G  293G  154G  66% /

The lvm2 setup is one volume group of 465G, which is fully used by / and swap (15G).

/ is formatted as btrfs:

btrfs filesystem df /
Data, single: total=444.63GiB, used=290.67GiB
System, DUP: total=8.00MiB, used=64.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=1.00GiB, used=764.58MiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=256.00MiB, used=0.00B

The btrfs volume is indeed full:

btrfs filesystem show
Label: none  uuid: 82ff2c95-6c6d-48c8-a0a0-3219e5cc2845
        Total devices 1 FS bytes used 212.50GiB
        devid    1 size 446.66GiB used 446.66GiB path /dev/mapper/vg-root

After deleting a huge logfile, the volume usage did not change; it is still full.

Any ideas on what happened and how to fix it?

  • 3
    Well, what filesystem is it? Can you still create new files in there? – ilkkachu Jun 18 at 13:37
  • 1
    Thanks, I had overlooked it is unexpectedly a btrfs. Investigating that now. – Posipiet Jun 18 at 14:21
2

As a modern filesystem, btrfs has no fixed inode limit at all, and that's why it reports inodes as all zeroes.

Check the status of btrfs subvolumes:

btrfs subvolume list -s /

If it turns out that you have snapshots hogging your disk space, you might need something like this to remove them:

btrfs subvolume delete -c /.snapshots/NNN/snapshot

See also this link for another user's adventure with btrfs and snapshots. The comments on that webpage include useful btrfs management commands among all the salt.

  • The subvolume list returns empty. But researching btrfs led to "btrfs fi balance start / -dusage=5" which freed up some space. This brings our server up and running, giving us time to figure out the subvolume problem properly. – Posipiet Jun 18 at 14:53
  • As you discovered, the subvolumes are not the only new and potentially surprising thing with btrfs. It just was the one I knew for sure about. – telcoM Jun 18 at 15:04
0

Based on the output of df -i it looks like you're out of inodes. When you run out of inodes you're unable to write new data even if there's space available. Your solutions are to either find out what's using all the inodes or add inodes to the existing filesystem. I don't know how to add inodes but it's possible that if you're using btrfs snapshots that they're consuming all the inodes. I'd start looking there and see if there's snapshots you can delete.

  • 1
    Generally so, but when a filesystem reports 0 inodes free and 0 used, out of 0 total, it might be the filesystem trying to tell the concept of a limited number of inodes is not applicable to it. – telcoM Jun 18 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.