1

I have a fairly stock xubuntu install that is suddenly giving "disk full" errors, after operating apparently normally for about 1.5 years. It has a system SSD (~128GB), a "data" SSD (~500GB) containing a samba share, and a raid to back itself up (two ~4TB HDDs). It also runs an instance of subsonic to stream audio files. Here's the setup:

root@castor:/# lsblk
NAME                     MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                        8:0    0   3.7T  0 disk  
└─md0                      9:0    0   3.7T  0 raid1 
  └─md0p1                259:6    0   3.7T  0 md    /raid
sdb                        8:16   0   3.7T  0 disk  
└─md0                      9:0    0   3.7T  0 raid1 
  └─md0p1                259:6    0   3.7T  0 md    /raid
nvme0n1                  259:0    0   477G  0 disk  
└─nvme0n1p1              259:1    0   477G  0 part  /data
nvme1n1                  259:2    0 111.8G  0 disk  
├─nvme1n1p1              259:3    0   512M  0 part  /boot/efi
├─nvme1n1p2              259:4    0   732M  0 part  /boot
└─nvme1n1p3              259:5    0 110.6G  0 part  
  └─nvme0n1p3_crypt      253:0    0 110.6G  0 crypt 
    ├─xubuntu--vg-root   253:1    0 109.6G  0 lvm   /
    └─xubuntu--vg-swap_1 253:2    0   976M  0 lvm   [SWAP]

Above, notice that /data and /raid are on separate physical devices from /. The system thinks that / is full:

root@castor:/# df -h /
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/xubuntu--vg-root  108G  106G     0 100% /

But if I exclude /data and /raid I see only 9GB used, not 106GB used:

root@castor:/# du -hs --exclude={./data,./raid}
du: cannot access './run/user/110/gvfs': Permission denied
du: cannot access './proc/1747/task/1747/fd/4': No such file or directory
du: cannot access './proc/1747/task/1747/fdinfo/4': No such file or directory
du: cannot access './proc/1747/fd/3': No such file or directory
du: cannot access './proc/1747/fdinfo/3': No such file or directory
9.0G    .

What is going on here? Is this something to do with installing with encrypted filesystem, or using LVM? (I confess I don't really know how LVM works under the hood, I just know abstractly that it's supposed to allow multiple physical devices to constitute a single logical "drive").

In case it's useful, here's du without the exclude argument; it's clearly including the contents of /data and /raid in the total:

root@castor:/# du -hs /data
220G    /data
root@castor:/# du -hs /raid
220G    /raid
root@castor:/# du -hs .
du: cannot access './run/user/110/gvfs': Permission denied
du: cannot access './proc/1762/task/1762/fd/4': No such file or directory
du: cannot access './proc/1762/task/1762/fdinfo/4': No such file or directory
du: cannot access './proc/1762/fd/3': No such file or directory
du: cannot access './proc/1762/fdinfo/3': No such file or directory
448G    .

and here's the full output of df

root@castor:/# df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                          7.7G     0  7.7G   0% /dev
tmpfs                         1.6G  3.0M  1.6G   1% /run
/dev/mapper/xubuntu--vg-root  108G  106G     0 100% /
tmpfs                         7.7G     0  7.7G   0% /dev/shm
tmpfs                         5.0M  4.0K  5.0M   1% /run/lock
tmpfs                         7.7G     0  7.7G   0% /sys/fs/cgroup
/dev/nvme0n1p1                469G  220G  226G  50% /data
/dev/nvme1n1p2                705M   81M  573M  13% /boot
/dev/nvme1n1p1                511M  6.1M  505M   2% /boot/efi
tmpfs                         1.6G  4.0K  1.6G   1% /run/user/110
tmpfs                         1.6G     0  1.6G   0% /run/user/1000
/dev/md0p1                    3.6T  221G  3.2T   7% /raid

EDIT to answer "where's /proc?"

root@castor:/# du -hs /proc
du: cannot access '/proc/1797/task/1797/fd/4': No such file or directory
du: cannot access '/proc/1797/task/1797/fdinfo/4': No such file or directory
du: cannot access '/proc/1797/fd/3': No such file or directory
du: cannot access '/proc/1797/fdinfo/3': No such file or directory
0    /proc


root@castor:/# ll /proc -S | head
total 4
-r--------    1 root             root             140737477881856 Jun 11 13:53 kcore
drwxr-xr-x+  27 root             root                        4096 Jun 11 12:22 ../
lrwxrwxrwx    1 root             root                          11 Jun 11 13:53 mounts -> self/mounts
lrwxrwxrwx    1 root             root                           8 Jun 11 13:53 net -> self/net/
dr-xr-xr-x  185 root             root                           0 Jun 11 12:34 ./
dr-xr-xr-x    9 root             root                           0 Jun 11 12:34 1/
dr-xr-xr-x    9 root             root                           0 Jun 11 12:35 10/
dr-xr-xr-x    9 root             root                           0 Jun 11 12:35 100/
dr-xr-xr-x    9 root             root                           0 Jun 11 12:35 1016/
4
  • 1
    Where's /proc? Commented Jun 11, 2020 at 21:43
  • @roaima question edited
    – drammock
    Commented Jun 11, 2020 at 21:58
  • @roaima I think df on Linux hides a lot of pseudo-filesystems by default unless used with -a. eg on my current system: cat /proc/mounts|wc -l => 43. df|wc -l => 16 df -a|wc -l => 44.
    – A.B
    Commented Jun 11, 2020 at 22:23
  • @A.B I've removed my answer until I can investigate this further. Thank you Commented Jun 11, 2020 at 23:23

1 Answer 1

2

It turned out that /raid had not been properly re-mounted at some point, so that for a while the automatic daily backups were not writing to the actual raid /dev/md0 (mounted at /raid), but instead were getting written to a folder /raid on /dev/xubuntu-vg/root. I didn't notice this because at the time I started investigating, the actual raid was mounted at /raid and was effectively masking the contents of the underlying folder.

In sum, the problem was solved by

umount /raid
mdadm --stop /dev/md0
rm -rf /raid/*
mdadm --create --assume-clean --level=1 /dev/md0 --raid-devices=2 /dev/sda /dev/sdb 
mount /raid
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .