I want to replace /home with a symlink to my nfs-mounted home dirs.

Only root is logged in, /home is not a separate filesystem, lsof shows no locks, selinux is permissive. What am I missing?

I'm logged in directly as root via ssh:

[root@usil01-sql01 /]# uname -a
Linux usil01-sql01 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@usil01-sql01 /]# w
 15:30:33 up  1:41,  1 user,  load average: 0.00, 0.02, 0.22
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/2     15:13    1.00s  0.19s  0.01s w

[root@usil01-sql01 /]# lsof | grep /home

[root@usil01-sql01 /]# lsof +D /home

[root@usil01-sql01 /]# df -h /home
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        63G  4.1G   56G   7% /

[root@usil01-sql01 /]# mount | grep -w /
/dev/sda2 on / type ext4 (rw,relatime,seclabel,data=ordered)

[root@usil01-sql01 /]# ls -lFd /home
drwxr-xr-x. 3 root root 4096 Mar  7 13:36 /home/

[root@usil01-sql01 /]# getenforce

[root@usil01-sql01 /]# mv /home /home-old
mv: cannot move "/home" to "/home-old": Device or resource busy

What else can I check?

More system info:

[root@usil01-sql01 /]# lsblk
sda      8:0    0 836.6G  0 disk 
|-sda1   8:1    0 768.6G  0 part /storage
|-sda2   8:2    0    64G  0 part /
`-sda3   8:3    0     4G  0 part [SWAP]
sr0     11:0    1  1024M  0 rom  

[root@usil01-sql01 /]# blkid
/dev/sda2: UUID="5ba6a429-4c65-4023-82b4-3673bfcf6a88" TYPE="ext4" 
/dev/sda3: UUID="b5eb680f-8789-43b2-9f7e-c52570b0eb73" TYPE="swap" 
/dev/sda1: UUID="cb22d57d-4a5b-4963-a990-890abe0c56dc" TYPE="ext4" 
  • Try the lazy umount option as in umount -f -l /home Mar 7 '17 at 21:41
  • Try to look up a suspicious process that consumes to much resources while not being known to you with top or ps. Them try to watch what it's doing right now using strace. That situation is weird tmm. UPDATE: also double check lsof output with fuser one just in case.
    – ddnomad
    Mar 7 '17 at 21:48
  • 1
    @val0x00ff as you can see in the output above, /home is not a mount point.
    – TheAmigo
    Mar 7 '17 at 22:17
  • 1
    @ddnomad fuser /home also has no output. It's a freshly-installed system sitting idle, no busy processes.
    – TheAmigo
    Mar 7 '17 at 22:17
  • @TheAmigo I see. I presume you've already tried to simply reboot the host. You might also try to boot into a safe mode from GRUB menu and try it there.
    – ddnomad
    Mar 7 '17 at 22:24

mv: cannot move "/home" to "/home-old": Device or resource busy

The only "use"[*] I can think of, which holds the name of a file from changing, is a mount point.

What else can I check?

I am not certain, but perhaps this could happen if the mount still exists in another mount namespace. Because it's not getting unmounts propagated from the root namespace, for some reason? Or looking at the result on my system, maybe systemd services with ProtectHome?

$ grep -h home /proc/*/task/*/mountinfo | sort -u
121 89 0:22 /systemd/inaccessible/dir /home ro,nosuid,nodev shared:142 master:24 - tmpfs tmpfs rw,seclabel,mode=755
275 243 253:2 / /home ro,relatime shared:218 master:33 - ext4 /dev/mapper/alan_dell_2016-home rw,seclabel,data=ordered
321 288 253:2 / /home rw,relatime shared:262 master:33 - ext4 /dev/mapper/alan_dell_2016-home rw,seclabel,data=ordered
84 64 253:2 / /home rw,relatime shared:33 - ext4 /dev/mapper/alan_dell_2016-home rw,seclabel,data=ordered
85 46 253:2 / /home rw,relatime master:33 - ext4 /dev/mapper/alan_dell_2016-home rw,seclabel,data=ordered

Note this issue - unable to rename /home despite it not showing as a mount point (in the current namespace) - should be fixed in Linux kernel version 3.18+.


how to find out namespace of a particular process?

lsns might be useful if you can install it. More possible commands:

List mount namespaces:

# readlink /proc/*/task/*/ns/mnt | sort -u

Identify root mount namespace:

# readlink /proc/1/ns/mnt

Find processes with a given mount namespace

# readlink /proc/*/task/*/ns/mnt | grep 4026531840

Inspect the namespace of a given process:

# cat /proc/1/task/1/mountinfo

[*] EBUSY The rename fails because oldpath or newpath is a directory that is in use by some process (perhaps as current working directory, or as root directory, or because it was open for reading) or is in use by the system (for example as mount point), while the system considers this an error. (Note that there is no require‐ ment to return EBUSY in such cases—there is nothing wrong with doing the rename anyway—but it is allowed to return EBUSY if the system cannot otherwise handle such situations.)

  • /home has never been a mount in any namespace.
    – TheAmigo
    Mar 8 '17 at 14:14
  • Does that mean you used commands along the lines I suggested? Because "never" is a very strong assertion, my commands wouldn't demonstrate that. I've added a command at the start which would be simpler (and hopefully functional :), and a second speculation on why this might happen based on the result on my system.
    – sourcejedi
    Mar 8 '17 at 15:00
  • I found another "broken" machine and your grep command indeed points the finger at NetworkManager. So my "stop NM, rename, restart NM" works, but your grep of mountinfo is what find the culprit.
    – TheAmigo
    Mar 8 '17 at 16:22
  • (because /lib/systemd/system/NetworkManager.service uses ProtectHome, at least on the systems we're looking at)
    – sourcejedi
    Mar 8 '17 at 16:39

It was NetworkManager.

Running systemctl stop mysqld httpd postfix ipmievd tuned atd rsyslog smartd crond irqbalance gssproxy polkit chronyd didn't help, but made the process table very small.

After systemctl stop NetworkManager, I was able to rename /home.

  • I needed to do only "service NetworkManager stop" and then was able to move /home then did a reboot to make sure all was well.
    – sdjuan
    Apr 9 '19 at 22:49
  • Thank you very much. This is not at all obvious and I had to read dozens of pages talking about /home as a mount point, which is clearly not in the OP question. I found stopping NetworkManager also helped in my case, running Redhat 7.6.
    – labradort
    Jul 8 '19 at 13:00
  • You just saved one of my peers an entire afternoon of grief, I think. I already hated NetworkManager, so this was oddly cheering. He's on CentOS 7.8 Jun 5 '20 at 19:07
  • Thanks for the tip. For me, I just had to stop chronyd systemctl stop chronyd
    – IanB
    Jan 13 '21 at 23:29

You can boot on single user and make any changes on home directory.

  1. Rebooted my system and durring grub selection edit the line by pressing e
  2. In linux16 line removed rhgb & quite options and put in their placed init=/bin/bash.
  3. The pressed ctrl+x to start. This will promt you on bash console.
  4. Remount / with read/write options by issuing mount -o remount,rw /
  5. After that you can edit your /home directory rename it etc.
  6. After work is done relabel the selinux by issuing touch /.autorelabel
  7. Finally execute exec /sbin/init to start boot as normal.

@sourcejedi: thank you for your answer.

  • 1
    While that works for local machines, it can be a challenge for a physical machine in a remote data center unless you have an IPMI or some means of remote access to the console.
    – TheAmigo
    Jan 21 '21 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.