No space left on device on CentOS7

Question

My machine which OS is CentOS7 have to backup 50G data into MySQL. But, there is no space left on the device. But there are many spaces on /home. How can I make user that there are enough space for MySQL to store data. I think there is no space in /dev/mapper/centos-root, how can I move space to from /home to /.

$ df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G   50G   20K 100% /
devtmpfs                 7.8G     0  7.8G   0% /dev
tmpfs                    7.8G   84K  7.8G   1% /dev/shm
tmpfs                    7.8G  2.6G  5.2G  34% /run
tmpfs                    7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/centos-home  500G   20G  480G   4% /home
/dev/sda1                497M  241M  257M  49% /boot
tmpfs                    1.6G   16K  1.6G   1% /run/user/42
tmpfs                    1.6G     0  1.6G   0% /run/user/0
tmpfs                    1.6G  4.0K  1.6G   1% /run/user/1000

And g++ compile code failed.

$ g++ test.cpp
Cannot create temporary file in /tmp/: No space left on device
[1]    6642 abort (core dumped)  g++ test.cpp

Answer 1

You first need to get your root partition ("/") down from being full or many bad things will happen in your system. It needs /tmp to be writeable for many system and user tasks.

Your root partition is only 50GB so you're never going to get the space you need there with current partition structure.

Your options would be to repartition to get a separate /var (out of space in /home) to give yourself space for MySQL data files. the utility parted will do that. If it's a hosted machine that may not be easy or possible to do, and I don't recommend playing around with that utility if you're not sure what you're doing.

Failing that, it's a bit of a hack but you could get around it with symlinks. Again not normally recommended and you'd probably need to take your machine down to single-user mode as running daemons will have file locks stopping you moving. The essence would be:

mv /var /home
ln -s /home/var /var

And if you do that you'll probably need to reboot the machine to bring it up cleanly:

shutdown -r now

Please research what you want to do, playing around with disk partitions can get very messy if you don't have a clear plan.

Answer 2

If you store only backups of MySQL on your / partition then it's easy to move the data.

Assume you're storing your data in /opt/backup/ you can:

• Stop the backup from writing to that directory ( if you're using some backup program )

• Move backup folder in / to /home/ ( or any folder in /home as you want)

  # mv /opt/backup/ /home/

• Create a symbolic link to old directory:

  # ln -s /home/backup/ /opt/backup

• Give proper permission to make sure your backup program have write permission to /home/backup with chmod.

• Start your backup program back

Answer 3

Clone operating system using professional tools like acronis to larger disk with bigger partitions as hard links may cause trobule at later stage. Use boot rescue to rebuild grub as after moving to new disk layout centos may not boot.

Answer 4

Using CentOs 7, I tried two IDE's but had the same problem handling a project with approx 16000 files.

"no space left on device" was logged by the IDE.
Both IDE's were hanging, or became unresponsive, or debugging actions stop or pause would crash the whole system or terminate the desktop session.

And in the message log it said:

Oct  2 10:44:08 localhost gnome-session: inotify_add_watch("/home/tools/trunk/ext/boost/libs/hana/example/functional") failed: "No space left on device"

Turns out the Centos Kernel has a buffer for folder watches etc..
i found the solution by sfuser0000 on this site.. This is a copy of the solution:

Common problem, temporary solution, increase the watches limit (to, say, 512K):

sudo sysctl fs.inotify.max_user_watches = 524288

(worked for me, but I'll probably go permanent and maybe larger, info below)

To make the change permanent, edit the file /etc/sysctl.conf and add this line to the end of the file:

fs.inotify.max_user_watches = 524288

Size info below:

"Can you increase the watches? By how much? Any tradeoffs? Short answer: Sure, no sweat. Go to straight to a half-million (524288) if you want...the additional memory used should be negligible on a modern system with 4GB+ of memory.

Each used inotify watch takes up 540 bytes (32-bit system), or 1 kB (double - on 64-bit) This comes out of kernel memory, which is unswappable. So, assuming you set the max at 524288, and all were used (improbable), you'd be using approx. 256MB/512MB of 32-bit/64-bit kernel memory

Note that your application will also use additional memory to keep track of the inotify handles, file/directory paths, etc. -- how much depends on its design."

Answer 5

Permanent solution:

fs.inotify.max_user_watches=1048576 #just add to this line

to

/etc/sysctl.conf

fixed the limit value permanently

then do

sysctl -p