Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Possible Duplicate:
How can I diagnose and repair missing drive space?

root@database:/# treesize -x 2>/dev/null
1.1 GB .
757.1 MB ./usr
232.0 MB ./lib
97.2 MB ./var
29.0 MB ./boot
6.3 MB ./sbin
5.5 MB ./etc
5.4 MB ./bin
80.0 KB ./home
68.0 KB ./root
32.0 KB ./tmp
16.0 KB ./lost+found
12.0 KB ./mnt
12.0 KB ./media
4.0 KB ./srv
4.0 KB ./selinux
4.0 KB ./opt
0.0 KB ./sys
0.0 KB ./proc
0.0 KB ./dev
root@database:/# df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9,2G  9,0G     0 100% /
root@database:~$ df -i /
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             610800   57516  553284   10% /
root@database:/# du -sh /* 2>/dev/null
5,4M        /bin
29M /boot
0   /cdrom
144K        /dev
5,5M        /etc
84K /home
0   /initrd.img
0   /initrd.img.old
232M        /lib
0   /lib64
16K /lost+found
12K /media
12K /mnt
69G /opt
0   /proc
104K        /root
6,4M        /sbin
4,0K        /selinux
4,0K        /srv
0   /sys
32K /tmp
758M        /usr
98M /var
0   /vmlinuz
0   /vmlinuz.old

Contents of treesize

#/bin/sh
du -k --max-depth=1 "$@" | sort -nr | awk '
     BEGIN {
        split("KB,MB,GB,TB", Units, ",");
     }
     {
        u = 1;
        while ($1 >= 1024) {
           $1 = $1 / 1024;
           u += 1
        }
        $1 = sprintf("%.1f %s", $1, Units[u]);
        print $0;
     }
    '

After an upgrade from Ubuntu 9 to 10 LTS this suddenly happened a few days later.

Why is / out of space when I can't find a single file taking up the space? I've already tried fsck from gparted live. I've also booted into single user mode and saved a copy of lsof output. But since the machine has rebooted multiple times I doubt it's due to open files.

I have mysqld on /opt and it had some error.logs open in /var/log, I tried shutting down everything I could, vmware tools, cron, syslog, mysql, atd, nrpe, snmpd and again checked treesize but no difference in the output.

Edit: I guess my accept-rate will have to drop even further because it looks like this is yet another question on one can answer.

My solution was to install a new VM, mount the old VMDK in the new VM and mount /opt from the old VMDK. Interestingly enough when I mounted the old / on /mnt/root I saw that the FS was still full.

# df -h /mnt/root
Filsystem            Storlek Anvnt Tillg Anv% Monterat på
/dev/sdb1             9,2G  9,2G     0 100% /mnt/root
share|improve this question
5  
69G /opt on a 9,2G seems like something is wrong. fsck couldn't find any error and you checked the right filesystem? Also your accept rate is really low. –  scai Oct 24 '12 at 12:39
1  
An utility like ncdu could give you more insight on what is using space. –  Renan Oct 24 '12 at 12:46
    
I don't know what accept rate is. I meant to use -x on the last du command because /opt and /mnt are either other partitions or external shares. / is only 9.2G. –  Stefan Midjich Oct 24 '12 at 13:01
    
you asked X amount of questions and have only selected an answer for 25% of X. You've left 75% of your questions marked as "unanswered". meta.stackexchange.com/questions/5234/… –  h3rrmiller Oct 24 '12 at 13:29
1  
You could try bind-mounting / elsewhere - maybe some of your mount points are hiding things –  Mat Oct 24 '12 at 15:20
show 1 more comment

marked as duplicate by Gilles, Renan, warl0ck, jasonwryan, rahmu Oct 25 '12 at 8:51

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

1 Answer

Is the large utilization persistent across reboots? Is it possible that there was an open file descriptor writing to a file that was no longer there? I've seen this cause filesystems to think they are 100% when they're really not. In that case, a reboot will fix it.

Also, I don't see that your treesize script will by default consider hidden (.) files.

Try a command like

find . -x -type f -exec du -k {} \; | awk -F'  ' '{sum+=$1} {print $1"\t"$2} END{print "--------\n"sum"\tTotal K-Bytes"}'

That will provide similar output to your script, but will include a total line at the bottom that you can use to compare to what df says about the filesystem.

share|improve this answer
    
I mentioned that I have rebooted, and booted into single user mode to check lsof. And also run fsck from a live cd. I ended up just reinstalling a new VM and mounting the old VMDK /opt partition in the new VM, quick solution. –  Stefan Midjich Oct 25 '12 at 7:08
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.