1

I got a Raspberry PI Cluster running a Kubernetes setup and the master node recently stopped scheduling pods because of low disk space on the root filesystem (I removed the taint which prevented pod scheduling on the master node).

This is the output for df

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G   24G  4.3G  85% /
devtmpfs        3.6G     0  3.6G   0% /dev
tmpfs           3.7G     0  3.7G   0% /dev/shm
tmpfs           3.7G  1.3M  3.7G   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           3.7G     0  3.7G   0% /sys/fs/cgroup
/dev/sda1       229G   60M  217G   1% /mnt/nfs/storage-04-ssd-250gb
/dev/sdb1       229G   60M  217G   1% /mnt/nfs/storage-01-ssd-250gb
/dev/sdc1       7.3T  2.0T  5.3T  28% /mnt/nfs/storage-03-hdd-8tb
/dev/mmcblk0p1  253M   54M  199M  22% /boot
tmpfs           758M     0  758M   0% /run/user/1000

So I search online for ways to analyse the filesystem usage. I mainly find articles which suggest using du and ncdu.

Output from du

$ sudo du -cbsh --exclude /mnt /
du: cannot access '/proc/23440/task/23440/fd/3': No such file or directory
du: cannot access '/proc/23440/task/23440/fdinfo/3': No such file or directory
du: cannot access '/proc/23440/fd/4': No such file or directory
du: cannot access '/proc/23440/fdinfo/4': No such file or directory
7.3G    /
7.3G    total

Output from ncdu

sudo ncdu -er / --exclude /mnt
    4.0 GiB [##########] /var                                                                                                                                                                              
    2.5 GiB [######    ] /usr
  687.0 MiB [#         ] /lib
  104.2 MiB [          ] /opt
   53.6 MiB [          ] /boot.bak
   53.6 MiB [          ] /boot
   30.3 MiB [          ] /home
   11.5 MiB [          ] /sbin
   11.1 MiB [          ] /bin
    5.1 MiB [          ] /etc
    1.2 MiB [          ] /run
   44.0 KiB [          ] /root
   36.0 KiB [          ] /tmp
e  16.0 KiB [          ] /lost+found
e   4.0 KiB [          ] /srv
e   4.0 KiB [          ] /media
    0.0   B [          ] /sys
.   0.0   B [          ] /proc
    0.0   B [          ] /dev
<   0.0   B [          ]  mnt

Both ncdu and du show the same values, but these are different than what df outputs. I expected both of them showing disk usage amounting to 24G, as reported by df.

I already checked online to see what the possible reason is for such a difference and I mainly found mentions that deleted files still being used by processes still occupy disk space, but I don't seem to have any of that on my system.

$ sudo lsof | grep deleted
$ #outputs nothing

How can I find out whats occupying those 24G on the root mount?

NOTE: in the executions of ncdu and du I excluded the /mnt directory because it just has some external drives mounted there

1 Answer 1

2

It is possible that you wrote to /mnt when your disks were not mounted.

In that case you should unmount the external disks first and then check if all directories on /mnt are empty.

2
  • Nice catch. Seems like that was indeed the issue, which leads me to other questions like why the drive wasnt mounted, but thats another story. Thanks for the help! Sep 6, 2020 at 8:25
  • In my case it was a swapfile that should have been on the external disk of a Raspberry Pi but was in the mountpoint folder instead.
    – bomben
    Nov 27, 2020 at 15:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .