How to increase disk space in root?

Question

So I check on my server using df -h and my centos-root is at 99% capacity.

Just a few days ago, the root was at 87%. For some reason, it increases so quick. Is there any suggestions on how to reduce the used space?

[root@host ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G   50G  713M  99% /
devtmpfs                 3.8G     0  3.8G   0% /dev
tmpfs                    3.8G     0  3.8G   0% /dev/shm
tmpfs                    3.8G  401M  3.4G  11% /run
tmpfs                    3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/mapper/centos-home  873G  136G  738G  16% /home
/dev/sda1                494M  172M  323M  35% /boot

[root@host ~]# du -csh /*
0       /bin
146M    /boot
0       /dev
23M     /etc
136G    /home
0       /lib
0       /lib64
0       /media
0       /mnt
24K     /nginxweblog
990M    /opt
du: cannot access `/proc/61778/task/61778/fd/4': No such file or directory
du: cannot access `/proc/61778/task/61778/fdinfo/4': No such file of directory
du: cannot access `/proc/61778/fd/4': No such file of directory
du: cannot access `/proc/61778/fdinfo/4': No such file of directory
0       /proc
41M     /root
401M    /run
0       /sbin
0       /srv
0       /sys
376K    /tmp
2.6G    /usr
14G     /var
153G    total

[root@host ~]# cd /; du -sh * 2> /dev/null
0       bin
146M    boot
0       dev
23M     etc
136G    home
0       lib
0       lib64
0       media
0       mnt
24K     nginxweblog
990M    opt
0       proc
41M     root
401M    run
0       sbin
0       srv
0       sys
376K    tmp
2.6G    usr
14G     var

[root@host ~]# vgs
  VG     #PV #LV #SN Attr   VSize   VFree
  centos   1   3   0 wz--n- 931.02g    0

[root@host ~]# cd /var/log
[root@host log]# ls
anaconda       cron           dmseg      lastlog           maillog-20160201   messages-20160124  ppp              secure-20160124   spooler-20160124  yum.log
audit          cron-20160111  dmesg.old  maillog           mariadb            messages-20160201  samba            secure-20160201   spooler-20160201  yum.log-20150101
boot.log       cron-20160117  firewalld  maillog-20160111  messages           nginx              secure           spooler           tallylog          yum.log-20160101
btmp           cron-20160124  grubby     maillog-20160117  messages-20160111  ntpstats           secure-20160111  spooler-20160111  tuned
btmp-20160201  cron-20160201  httpd      maillog-20160124  messages-20160117  php-fpm            secure-20160117  spooler-20160117  wtmp

[root@host ~]# du -csh /var/*
0       /var/adm
1.3M    /var/cache
0       /var/crash
36K     /var/db
0       /var/empty
0       /var/games
0       /var/gopher
0       /var/kerberos
13G     /var/lib
0       /var/local
0       /var/lock 
90M     /var/log
0       /var/mail
0       /var/nis
0       /var/opt
0       /var/preserve
0       /var/run
51M     /var/spool
4.0K    /var/tmp
0       /var/var
4.0K    /var/www
0       /var/yp
14G     total

[root@host andi]# du -csh /var/lib/*
64K     /var/lib/alternatives
8.0K    /var/lib/authconfig
0       /var/lib/cs
0       /var/lib/dav
0       /var/lib/dbus
0       /var/lib/dhclient
0       /var/lib/dnsmasq
0       /var/lib/games
0       /var/lib/initial-setup
0       /var/lib/initramfs
4.0K    /var/lib/logrotate.status
0       /var/lib/misc
13G     /var/lib/mysql
8.0K    /var/lib/NetworkManager
40K     /var/lib/nginx
0       /var/lib/ntp
0       /var/lib/os-prober
320K    /var/lib/pear
64K     /var/lib/php
0       /var/lib/phpMyAdmin
4.0K    /var/lib/plymouth
0       /var/lib/polkit-1
4.0K    /var/lib/postfix
4.0K    /var/lib/rhsm
58M     /var/lib/rpm
0       /var/lib/rpm-state
4.0K    /var/lib/rsyslog
0       /var/lib/samba
0       /var/lib/stateless
16K     /var/lib/systemd
0       /var/lib/tpm
0       /var/lib/udisks2
0       /var/lib/upower
0       /var/lib/virtuoso
13M     /var/lib/yum
13G     total

Answer 1

The outputs of du and df don't line up in terms of used disk; search for deleted files that are being written to. Stop (restart) the process writing to the file(s), or reboot the box.

 lsof | awk '/deleted/ && $5 ~ /w/'

Implement log rotation and find out why the log keeps growing at that tremendous rate (and rectify the root cause).

Answer 2

Short answer; increase space either by removing data or increase the actual size of root filesystem.

Long answer;

When you have such a big disk at your expense and you are using lvm as well, it is encouraged that you crate separate file systems for /var, /tmp, /var/log, /opt etc.

First of all, find what is eating your disk space by running the following as root cd /; du -sh * 2> /dev/null

Looks like you have already run the command.

I can see that /var is taking too much space, probably it will be a good idea to put it on a separate filesystem. Can you share output of `vgs' command?

Here are the steps you need to take; Detailed steps here

• Reboot the system, press any key at grub menu and append the following to the kernel line systemd.unit=rescue.target
• It will ask for root password, type the root password in
• lvchange --activate y
• lvreduce -r -L 800G /dev/mapper/centos-home you can specify new size 700G, 800G
• lvextend -r -L 100G /dev/mapper/centos-root say you want to make root as 100G file system Don't forget -r option, otherwise you will have to run resize2fs /dev/mapper/centos-root

Answer 3

The author of the question provided the following solution:

FINAL EDIT: found the solution (temporary one at least). Did a service php-fgm restart function and it reduced the storage from 99% to 34%. But guess I'd have to do log rotation and other things to my server to get a more permanent solution.

Answer 4

Seems you have a big mysql database and a small partition so you could either repartition using Gparted or change the place where mysql saves its data to /home, which is sometimes not safe(hacking) or buggy.

To change the place where mysql saves its data:

assuming the new directory is /home/mynewsqldir

1. Obvious enough.

service stop mysql

2. copy your database to the new position

cp -R /var/lib/mysql /home/mynewsqldir

cp -R /var/lib/mysql/users /home/mynewsqldir

3. Backup the my.cnf file

cp /etc/mysql/my.cnf /home/my.cnf.backup

4. edit the original my.cnf file and look for datadir and socket lines then replace them with

datadir=/home/mynewsqldir

socket=/home/mynewsqldir

5. update the directory permissions for safety

chown -R mysql:mysql /home/mynewsqldir

6. rename mysql directory in /var/lib/ to anything just in case something goes wrong but if everything goes as intended , you can delete it .

7. security thingie to update Apparmor

ln -s /home/mynewsqldir/ /var/lib/mysql

echo "alias /var/lib/mysql/ -> /home/mynewsqldir/," >> /etc/apparmor.d/tunables/alias

8. Reload the apparmor profiles and start mysql

sudo /etc/init.d/apparmor reload

start mysql