1

My root filesystem on one of my servers is showing 87% full and I can't work out why. I have several servers that perform similar functions to this one that are running around 25-35% full. The 5GB free on this server is filling during month end when that 5GB freespace gets chewed up by sort tempfiles and their reporting stuff fails.

df -k reports:

Filesystem          1K-blocks     Used Available Use% Mounted on
/dev/mapper/ol-root  39265556 34049040   5216516  87% /

Yet: du -skx reports:

11776608        /

So there's 20-ish GB of space being taken up there by something I can't explain.

I thought I might have had some hidden files under several mount points from NFS mounted drives. I unmounted the mounts and rechecked and still the same - the directories under the mounts were empty.

From the root directory I did (cutting errors from /proc):

[root@xxx /]# for i in `ls -1` ; do  du -skx $i; done
0       appliance
0       bin
199876  boot
0       dev
66260   etc
0       files
74300   home
0       lib
0       lib64
0       media
0       mnt
3572720 opt
0       proc
75144   root
165212  run
0       sbin
0       srv
0       sys
263512  tmp
4872936 usr
4383596 var

I went and checked for open files:

lsof / | awk '{if($7 > 1048576) print $7/1048576 "MB" " " $9 }' | sort -n -u

There's nothing there larger than 180MB and most of them are single digit, certainly nothing in the 20GB range.

I've done google things looking for answers, most were a variation of the things I've done (and the lsof idea came exactly from a google search).

One of the other guys in my team has looked as well (neither of us are super experts), and we're both drawing blanks here.

Anyone got any bright ideas ?

Thanks in advance

Improve this question
6
  • Not sure how filesyste type applies here, but maybe including it in you question is a good idea, as some fs's allow snapshots etc, which might possibly be an answer.
    – user422875
    Feb 25, 2021 at 9:15
  • It's xfs. I'm not aware of any snapshotty things in this environment unless there's some process that does it? From fstab: /dev/mapper/ol-root / xfs defaults 0 0
    – SJWales
    Feb 25, 2021 at 9:30
  • Just a quick search result: askubuntu.com/questions/865141/… unix.stackexchange.com/questions/22939/…
    – user422875
    Feb 25, 2021 at 9:38
  • Could this be related to /var/log/lastlog huge size ? See askubuntu.com/a/618667/350004
    – solsTiCe
    Feb 25, 2021 at 10:21
  • Getting late, I'll dig more into these in the morning but du -skhx reports 10G and adding --apparent-size reports 9.6G. Tried the echo 3> drop_caches thing, no change. xfs_db reports next to no fragmentation. Like 2% Will read more tomorrow but not seeing much here so far ...
    – SJWales
    Feb 25, 2021 at 10:28

1 Answer 1

Reset to default
0

Well, ended up cloning a new server for production so I could have time to troubleshoot this problem child.

Unmounted all the NFS drives so I could play.

Ended up running: xfs_fsr /dev/mapper/ol-root

Ran for a few minutes and my disk usage went down from 95% to about 35%

Problem solved.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .