0

So I check on my server using df -h and my centos-root is at 99% capacity.

Just a few days ago, the root was at 87%. For some reason, it increases so quick. Is there any suggestions on how to reduce the used space?

[root@host ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G   50G  713M  99% /
devtmpfs                 3.8G     0  3.8G   0% /dev
tmpfs                    3.8G     0  3.8G   0% /dev/shm
tmpfs                    3.8G  401M  3.4G  11% /run
tmpfs                    3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/mapper/centos-home  873G  136G  738G  16% /home
/dev/sda1                494M  172M  323M  35% /boot
[root@host ~]# du -csh /*
0       /bin
146M    /boot
0       /dev
23M     /etc
136G    /home
0       /lib
0       /lib64
0       /media
0       /mnt
24K     /nginxweblog
990M    /opt
du: cannot access `/proc/61778/task/61778/fd/4': No such file or directory
du: cannot access `/proc/61778/task/61778/fdinfo/4': No such file of directory
du: cannot access `/proc/61778/fd/4': No such file of directory
du: cannot access `/proc/61778/fdinfo/4': No such file of directory
0       /proc
41M     /root
401M    /run
0       /sbin
0       /srv
0       /sys
376K    /tmp
2.6G    /usr
14G     /var
153G    total
[root@host ~]# cd /; du -sh * 2> /dev/null
0       bin
146M    boot
0       dev
23M     etc
136G    home
0       lib
0       lib64
0       media
0       mnt
24K     nginxweblog
990M    opt
0       proc
41M     root
401M    run
0       sbin
0       srv
0       sys
376K    tmp
2.6G    usr
14G     var
[root@host ~]# vgs
  VG     #PV #LV #SN Attr   VSize   VFree
  centos   1   3   0 wz--n- 931.02g    0
[root@host ~]# cd /var/log
[root@host log]# ls
anaconda       cron           dmseg      lastlog           maillog-20160201   messages-20160124  ppp              secure-20160124   spooler-20160124  yum.log
audit          cron-20160111  dmesg.old  maillog           mariadb            messages-20160201  samba            secure-20160201   spooler-20160201  yum.log-20150101
boot.log       cron-20160117  firewalld  maillog-20160111  messages           nginx              secure           spooler           tallylog          yum.log-20160101
btmp           cron-20160124  grubby     maillog-20160117  messages-20160111  ntpstats           secure-20160111  spooler-20160111  tuned
btmp-20160201  cron-20160201  httpd      maillog-20160124  messages-20160117  php-fpm            secure-20160117  spooler-20160117  wtmp
[root@host ~]# du -csh /var/*
0       /var/adm
1.3M    /var/cache
0       /var/crash
36K     /var/db
0       /var/empty
0       /var/games
0       /var/gopher
0       /var/kerberos
13G     /var/lib
0       /var/local
0       /var/lock 
90M     /var/log
0       /var/mail
0       /var/nis
0       /var/opt
0       /var/preserve
0       /var/run
51M     /var/spool
4.0K    /var/tmp
0       /var/var
4.0K    /var/www
0       /var/yp
14G     total
[root@host andi]# du -csh /var/lib/*
64K     /var/lib/alternatives
8.0K    /var/lib/authconfig
0       /var/lib/cs
0       /var/lib/dav
0       /var/lib/dbus
0       /var/lib/dhclient
0       /var/lib/dnsmasq
0       /var/lib/games
0       /var/lib/initial-setup
0       /var/lib/initramfs
4.0K    /var/lib/logrotate.status
0       /var/lib/misc
13G     /var/lib/mysql
8.0K    /var/lib/NetworkManager
40K     /var/lib/nginx
0       /var/lib/ntp
0       /var/lib/os-prober
320K    /var/lib/pear
64K     /var/lib/php
0       /var/lib/phpMyAdmin
4.0K    /var/lib/plymouth
0       /var/lib/polkit-1
4.0K    /var/lib/postfix
4.0K    /var/lib/rhsm
58M     /var/lib/rpm
0       /var/lib/rpm-state
4.0K    /var/lib/rsyslog
0       /var/lib/samba
0       /var/lib/stateless
16K     /var/lib/systemd
0       /var/lib/tpm
0       /var/lib/udisks2
0       /var/lib/upower
0       /var/lib/virtuoso
13M     /var/lib/yum
13G     total
1
  • Log files should never show up in your root partition (unless you have only one). Check how to rotate/truncate logs automatically in your distribution.
    – vonbrand
    Feb 9, 2016 at 1:49

4 Answers 4

1

The outputs of du and df don't line up in terms of used disk; search for deleted files that are being written to. Stop (restart) the process writing to the file(s), or reboot the box.

 lsof | awk '/deleted/ && $5 ~ /w/'

Implement log rotation and find out why the log keeps growing at that tremendous rate (and rectify the root cause).

4
  • I did service nginx restart but it didn't reduce anything.
    – jj282890
    Feb 1, 2016 at 2:58
  • That's surprising. Can you check what the parent process of the php-process was that held the file? See if you can restart that instead.
    – tink
    Feb 1, 2016 at 3:01
  • actually after i did service php-fpm restart it went down from 99% to 34%. Thanks for this! But I'm guessing this is a temporary solution only though. What do you think?
    – jj282890
    Feb 1, 2016 at 3:21
  • Whoever set the machine up failed to implement log rotation. So yes, in this case it's temporary. But extending filesystems (shuffling disk about) won't make it go away, it will just postpone it. Implement log rotation. :}
    – tink
    Feb 1, 2016 at 18:09
0

Short answer; increase space either by removing data or increase the actual size of root filesystem.

Long answer;

When you have such a big disk at your expense and you are using lvm as well, it is encouraged that you crate separate file systems for /var, /tmp, /var/log, /opt etc.

First of all, find what is eating your disk space by running the following as root cd /; du -sh * 2> /dev/null

Looks like you have already run the command.

I can see that /var is taking too much space, probably it will be a good idea to put it on a separate filesystem. Can you share output of `vgs' command?

Here are the steps you need to take; Detailed steps here

  • Reboot the system, press any key at grub menu and append the following to the kernel line systemd.unit=rescue.target
  • It will ask for root password, type the root password in
  • lvchange --activate y
  • lvreduce -r -L 800G /dev/mapper/centos-home you can specify new size 700G, 800G
  • lvextend -r -L 100G /dev/mapper/centos-root say you want to make root as 100G file system Don't forget -r option, otherwise you will have to run resize2fs /dev/mapper/centos-root
9
  • All right, you dont have any free space on your volume group. Are you willing to shrink /home file-system? If yes, I can show you how to do that and then we can either use that free space to create another /var file-system or assign that extra space to / file-system. Feb 1, 2016 at 1:30
  • We can expand a LVM online but shrinking is another problem. I am guessing that you are using CENTOS, is it 6 or 7? Because the process is going to differ slightly. Also, you will need to have root user's password. Feb 1, 2016 at 1:36
  • Before you guys start resizing filesystems I'd like to point out that the space used in du and df is way out of whack. I'd suggest looking for deleted files that are still being written to first. Just saying ...
    – tink
    Feb 1, 2016 at 2:01
  • @tink any way to look for the deleted/hidden unused files?
    – jj282890
    Feb 1, 2016 at 2:04
  • @Joey2890 : sure lsof | awk '/deleted/ && $5 ~ /w/' should do the job...
    – tink
    Feb 1, 2016 at 2:10
0

The author of the question provided the following solution:

FINAL EDIT: found the solution (temporary one at least). Did a service php-fgm restart function and it reduced the storage from 99% to 34%. But guess I'd have to do log rotation and other things to my server to get a more permanent solution.

-1

Seems you have a big mysql database and a small partition so you could either repartition using Gparted or change the place where mysql saves its data to /home, which is sometimes not safe(hacking) or buggy.

To change the place where mysql saves its data:

assuming the new directory is /home/mynewsqldir

1. Obvious enough.

service stop mysql

2. copy your database to the new position

cp -R /var/lib/mysql /home/mynewsqldir

cp -R /var/lib/mysql/users /home/mynewsqldir

3. Backup the my.cnf file

cp /etc/mysql/my.cnf /home/my.cnf.backup

4. edit the original my.cnf file and look for datadir and socket lines then replace them with

datadir=/home/mynewsqldir

socket=/home/mynewsqldir

5. update the directory permissions for safety

chown -R mysql:mysql /home/mynewsqldir

6. rename mysql directory in /var/lib/ to anything just in case something goes wrong but if everything goes as intended , you can delete it .

7. security thingie to update Apparmor

ln -s /home/mynewsqldir/ /var/lib/mysql

echo "alias /var/lib/mysql/ -> /home/mynewsqldir/," >> /etc/apparmor.d/tunables/alias

8. Reload the apparmor profiles and start mysql

sudo /etc/init.d/apparmor reload

start mysql

15
  • thank you for the guide. i restart my web server and for now the usage went down from 99% to 34%. But i have a feeling that it was only a temporary solution. I'll try your solution. Btw, i tried doing yum update and it says error cannot open package etc etc. do i have to install the GParted on my server/local computer first?
    – jj282890
    Feb 1, 2016 at 3:23
  • i think it is temporary too,but you should wait, if usage goes up again then you can try my guide. for Gparted centos.org/forums/viewtopic.php?t=5231 or generally search google install gparted centos as i have never owned a centos and i don't wanna break your system. Feb 1, 2016 at 3:28
  • you could install it or just download gpartedlive.iso , then burn it to a usb using unetbootin or whatever, boot from it, and repartition which is better in my opinion and please delete all your i did it or i edited type of comment as they are very non useful Feb 1, 2016 at 3:31
  • @user3566929 Are you kidding? A log file held > 60% of the disk-space in / and you suggest moving data base files around?
    – tink
    Feb 1, 2016 at 18:10
  • @tink where did you get it was log file? Feb 1, 2016 at 23:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .