4

Suddenly all the available disk space on / has disappeared.

If I make room in the disk (by deleting ~50GB of stuff, for example), after a few minutes I am back to 0 available disk space (according to df).

Clearly, some process is eating up disk space at a rapid rate, but I can't figure out what it is.

One thing is certain, though: whatever it is, it must be creating many small files, because there are no files bigger than 10GB on the disk, and all the ones bigger than 1GB are much older than today.

How can I find what's eating up disk space?


FWIW, only df sees the problem, not du.

For example, below I show several "snapshots" from du and df taken 60s. apart. (I did this after I had made some room in the disk.) Notice how du's output remains steady (at 495G), but df shows a steadily shrinking amount of available space. (I've followed the recommendation given here. IOW, /mnt/root is pointing to /.)

# while true; do du -sh /mnt/root && df -h /mnt/root; sleep 60; done
495G    /mnt/root
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       880G  824G   12G  99% /mnt/root
495G    /mnt/root
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       880G  825G   11G  99% /mnt/root
495G    /mnt/root
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       880G  827G  8.9G  99% /mnt/root
495G    /mnt/root
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       880G  827G  8.1G 100% /mnt/root
495G    /mnt/root
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       880G  828G  7.5G 100% /mnt/root
  • What happens if you restart syslog? – Rui F Ribeiro May 22 '17 at 19:15
  • 1
    @RuiFRibeiro: thanks for the idea... I've never restarted syslog, need to look into that... – kjo May 22 '17 at 19:16
  • @RuiFRibeiro: FWIW, /var is tiny (3.1G), and doesn't seem to be growing particularly fast. – kjo May 22 '17 at 19:19
  • You dealing with a deleted file, that is why du does not register it. The /var size, if syslog is the culprit, wont register it too. – Rui F Ribeiro May 22 '17 at 19:20
  • 1
    @RuiFRibeiro: a deleted file that is growing? – kjo May 22 '17 at 19:21
5

You are dealing with deleted files, that is why du does not register used space, but dfdoes.

Deleted files only disappear after the owner process is stopped; they remain in use while that does not happen.

So to find the culprit process, I recommend you doing:

sudo lsof -nP | grep '(deleted)'
1

You could use iotop to see which processes are performing the most disk write operations.

Example:

Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
    6 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
    7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
    8 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.