0

I have a Fedora server with 300GB on the /dev/xvda1 partition. I'm losing GB at a rapid rate. There is no way possible this is normal so I'm trying to figure out where this is coming from.

I'm not even using du, I'm just using df -h to see the drop of disk space and it is fast. Any help would be appreciated.

  • 1
    This is rather vague. Logs probably, gadd to the question the output of sudo du -h /var/log – Rui F Ribeiro Dec 7 '16 at 16:42
  • 1
    You should use du to identify large files on your filesystem. Chances are something is writing a lot to disk, and that's the cause of your issue. – Centimane Dec 7 '16 at 16:42
  • Start by installing ncdu and using it to find out which directory is getting filled up so quickly. It will help us in identifying a possible cause, giving us a place to start looking. – Mioriin Dec 7 '16 at 16:46
  • A quick command for spotting a growing file (assumption is that it is open) could be lsof -s | sort -nr -k 7 | head – Michael Felt Dec 7 '16 at 16:57
  • There is no simple way to do it. Use du to locate the file that is growing. Start from / (du -hs /*) then if you find that, say, /usr is growing, descend into subdirectories of /usr (du -hs /usr/*) etc. In the simple case you'll eventually locate a single file, and you can use lsof to find out which process writes to it. In more complicated scenarios you'll have a file that has been deleted on disk but it's still held open by a process. You'll need more advanved lsof-fu to deal with that. – Satō Katsura Dec 7 '16 at 16:59
2

This is what I do, when I face such issue. This is not a rule, but just sharing my insights.

1) Check output of df -kh, and confirm where is /dev/xvda1 mounted. (lets say /var directory)

2) If /dev/xvda1 is filling up fast, then a lot of I/O will be happening over it. Try and see output of iotop command (if you have), if not skip this step.

2.1) If you do get the output of iotop, see the process which is doing maximum I/O on the screen. It should be one of the top ones if space is filling up so fast. Note down the PID of the process. Thereafter, check following lsof -p <PID> <---- PID that you noted down At the end of output of lsof command, you see something like

rsyslogd 1171 root    0u  unix 0xffff8800379e8700      0t0      11328 /dev/log
rsyslogd 1171 root    1w   REG                8,1   628182    3540277 /var/log/messages
rsyslogd 1171 root    2w   REG                8,1    27890    3540278 /var/log/secure
rsyslogd 1171 root    3r   REG                0,3        0 4026532040 /proc/kmsg
rsyslogd 1171 root    4w   REG                8,1     6375    3540279 /var/log/maillog
rsyslogd 1171 root    5w   REG                8,1    15228    3539742 /var/log/cron

Make a note of values as 0u,1w,2w etc. These are file descriptors opened by that process. One of the file descriptors will be one where data would be written pretty fast.

3) Assuming that you do not have any other directory under /var directory mounted on a different block device, Do the following (if you do, ping in again, we might need some more information about your system)

cd /var <--- the directory which you saw where /dev/xvda1 was mounted.
find . -maxdepth 1 -type d -exec du -hs {} \;

This will present you the highest space consuming directory in /var directory. May be you get some lead from there.

Let me know if something works out of this or not.

Disclaimer: I am assuming that your /dev/xvda1 is mounted directory on some directory and it is not part of some LVM,SWRAID etc. If above commands do not help anything (preferably if Step 1 has no output about /dev/xvda1), let us know we share more steps to investigate.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.