1
[nathanb /mnt/work] sudo du -hs .
23G .
[nathanb /mnt/work] df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        40G   38G  6.4M 100% /mnt/work

Where is the other 15 GB?

/dev/sdb1 on /mnt/work type ext4 (rw,nosuid,nodev,relatime,data=ordered)

Updating to respond to comments

[nathanb /mnt/work] sudo tune2fs -l /dev/sdb1
tune2fs 1.42.5 (29-Jul-2012)
Last mounted on:          /mnt/work
Inode count:              2621440
Block count:              10485752
Reserved block count:     524287
Free blocks:              3955615
Free inodes:              2522921
First block:              0
Block size:               4096
Fragment size:            4096

And

[nathanb /mnt/work] df -i .
Filesystem      Inodes IUsed   IFree IUse% Mounted on
/dev/sdb1      2621440 29764 2591676    2% /mnt/work

And

[nathanb /mnt/work] sudo fsck -n /dev/sdb1
fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
Warning!  /dev/sdb1 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
/dev/sdb1: clean, 98519/2621440 files, 6530137/10485752 blocks

And

[nathanb /mnt/work] sudo lsof | grep deleted
[nathanb /mnt/work]

There are no mount points below /mnt/work

[nathanb /mnt/work] grep /mnt/work /proc/self/mountinfo
22 19 8:17 / /mnt/work rw,nosuid,nodev,relatime - ext4 /dev/sdb1 rw,data=ordered

Well, of all the things...seems to be working again. And just like I have no idea what caused the problem, I have no idea what fixed it.

[nathanb /mnt/work] df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        40G   22G   16G  59% /mnt/work

I had unmounted a couple of the NFS clients hitting up the volume in preparation for umounting and fscking it, but I hadn't unmounted all of them...and I checked right after the unmount and the space hadn't gone down. But then I got back from doing some other work and noticed it was unwedged.

Annoying and unfulfilling...wish I knew what the problem had been so I could award some points to some folks...thanks for all the help, though, and if it happens again I'll try to get more forensics.

1

Given that the filesystem was exported via NFS, there’s a fair chance that the discrepancy was due to deleted files... If files are deleted while open on NFS clients, lsof on the server won’t see them because there is no /proc/.../fd entry corresponding to them; but they will still occupy disk space as seen by df.

Diagnosing this requires running lsof with the -N option on every client.

(This doesn’t explain the delay you saw in recovering the space after unmounting the volume from the clients, but it’s the best explanation I can think of for the rest of the symptoms.)

  • Fair enough...I'll take it! Thanks for the help. – Nathan Feb 3 '15 at 17:43
0

fuser -k -M -m /mnt/work should heel. Warning! it just literally kills processes accessing /mnt/work. Include -TERM to request termination.

The culprit is deleted files that are held open, which du can't see but df could.

For e.g.

$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda6       209M   66M  128M  35% /boot
$ sudo du -sh .
64M     .
$  sudo fallocate -l 100M tmp_file
$ ls -lh tmp_file
-rw-r--r-- 1 root root 100M Feb  3 02:24 tmp_file
$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda6       209M  166M   28M  86% /boot
$ sudo du -sh .
164M    .
$ exec 20<tmp_file
$ sudo rm tmp_file
$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda6       209M  166M   28M  86% /boot
$ sudo du -sh .
64M     .

The tmp_file is still open. If it's closed 'df' can see free.

$ exec 20<&-
$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda6       209M   66M  128M  35% /boot
  • I don't see any deleted files with active file handles in lsof output, as mentioned in the comments above. – Nathan Feb 2 '15 at 21:07
  • You didn't sudo. Or are you root. No $/# ? – Nizam Mohamed Feb 2 '15 at 21:17
  • I re-ran with sudo just to verify and got the same results. Updated the post – Nathan Feb 2 '15 at 21:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.