[nathanb /mnt/work] sudo du -hs . 23G . [nathanb /mnt/work] df -h . Filesystem Size Used Avail Use% Mounted on /dev/sdb1 40G 38G 6.4M 100% /mnt/work
Where is the other 15 GB?
/dev/sdb1 on /mnt/work type ext4 (rw,nosuid,nodev,relatime,data=ordered)
Updating to respond to comments
[nathanb /mnt/work] sudo tune2fs -l /dev/sdb1 tune2fs 1.42.5 (29-Jul-2012) Last mounted on: /mnt/work Inode count: 2621440 Block count: 10485752 Reserved block count: 524287 Free blocks: 3955615 Free inodes: 2522921 First block: 0 Block size: 4096 Fragment size: 4096
[nathanb /mnt/work] df -i . Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sdb1 2621440 29764 2591676 2% /mnt/work
[nathanb /mnt/work] sudo fsck -n /dev/sdb1 fsck from util-linux 2.20.1 e2fsck 1.42.5 (29-Jul-2012) Warning! /dev/sdb1 is mounted. Warning: skipping journal recovery because doing a read-only filesystem check. /dev/sdb1: clean, 98519/2621440 files, 6530137/10485752 blocks
[nathanb /mnt/work] sudo lsof | grep deleted [nathanb /mnt/work]
There are no mount points below /mnt/work
[nathanb /mnt/work] grep /mnt/work /proc/self/mountinfo 22 19 8:17 / /mnt/work rw,nosuid,nodev,relatime - ext4 /dev/sdb1 rw,data=ordered
Well, of all the things...seems to be working again. And just like I have no idea what caused the problem, I have no idea what fixed it.
[nathanb /mnt/work] df -h . Filesystem Size Used Avail Use% Mounted on /dev/sdb1 40G 22G 16G 59% /mnt/work
I had unmounted a couple of the NFS clients hitting up the volume in preparation for umounting and fscking it, but I hadn't unmounted all of them...and I checked right after the unmount and the space hadn't gone down. But then I got back from doing some other work and noticed it was unwedged.
Annoying and unfulfilling...wish I knew what the problem had been so I could award some points to some folks...thanks for all the help, though, and if it happens again I'll try to get more forensics.