Background: I am exploring how to copy an ordinary LVM-on-LUKS Debian 9 ("Stretch") installation from a thumb drive (the "source drive") onto a ZFS-formatted drive (the "target drive") in order to achieve a ZFS-on-LUKS installation. My process is based on this HOWTO.* I think the ZFS aspect is irrelevant to the issue I would like help with, but I am mentioning it just in case it matters.

As part of my process, while Stretch is running from the source drive, I mount the target ZFS root (/) filesystem at /mnt. I then recursively bind:

  • /dev to /mnt/dev
  • /proc to /mnt/proc
  • /sys to /mnt/sys.

I then chroot into /mnt.

(In the future, when I am chrooted into /mnt, I intend to run update-initramfs, update-grub, etc, to configure the contents of the /boot partition.)

I then exit the chroot, and my trouble begins. I find that I can unmount /mnt/dev and /mnt/proc, but not /mnt/sys. The latter refuses to unmount because it contains /mnt/sys/fs/cgroup/systemd, which the system for some reason thinks is "in use". Reformatting the ZFS drive and rebooting fixes the problem, but hugely slows the iterations of my learning and documentation process.

My questions are:

- How can I unmount /mnt/sys after the chroot, without rebooting?

- Is the failure (umount: /mnt/sys/fs/cgroup/systemd: target is busy) expected? If not, against which piece of software should I file a bug report: umount, cgroups, systemd, the Linux kernel, or something else?

Here is (I think) a minimal working example. (If you are having difficulty reproducing this and think I might have missed out a step, let me know.) First, the boilerplate:

# Activate the ZFS kernel module
/sbin/modprobe zfs

# Set variables
DIRS_TO_COPY=(boot bin etc home lib lib64 opt root sbin srv usr var)
FILES_TO_COPY=(initrd.img initrd.img.old vmlinuz vmlinuz.old)

## Partition target drive
# 1MB BIOS boot partition
sgdisk -a2048 -n1:2048:4095     -t1:EF02 $1 -c 1:"bios_boot_partition"
# 510MB partition for /boot ZFS filesystem
sgdisk -a2048 -n2:4096:1052671  -t2:BF07 $1 -c 2:"zfs_boot_partition"
# Remaining drive space, except the last 510MiB in case of future need:
# partition to hold the LUKS container and the root ZFS filesystem
sgdisk -a2048 -n3:1052672:-510M -t3:8300 $1 -c 3:"luks_zfs_root_partition"

# Before proceeding, ensure /dev/disk/by-id/ knows of these new partitions

# Create the /boot pool
zpool create -o ashift=12            \
             -O atime=off            \
             -O canmount=off         \
         -O compression=lz4      \
         -O normalization=formD  \
             -O mountpoint=/boot     \
             -R /mnt                 \
             $BOOT_POOL "$1"-part2

# Create the LUKS container for the root pool
cryptsetup luksFormat "$1"-part3               \
                      --hash sha512            \
                      --cipher aes-xts-plain64 \
              --key-size 512

# Open LUKS container that will contain the root pool
cryptsetup luksOpen "$1"-part3 "$DRIVE_SHORTNAME"3_crypt

# Create the root pool
zpool create -o ashift=12           \
             -O atime=off           \
             -O canmount=off        \
             -O compression=lz4     \
             -O normalization=formD \
             -O mountpoint=/        \
             -R /mnt                \
             $ROOT_POOL /dev/mapper/"$DRIVE_SHORTNAME"3_crypt

# Create ZFS datasets for the root ("/") and /boot filesystems
zfs create -o canmount=noauto -o mountpoint=/      "$ROOT_POOL"/debian
zfs create -o canmount=noauto -o mountpoint=/boot  "$BOOT_POOL"/debian

# Mount the root ("/") and /boot ZFS datasets
zfs mount "$ROOT_POOL"/debian
zfs mount "$BOOT_POOL"/debian

# Create datasets for subdirectories
zfs create                     -o setuid=off              "$ROOT_POOL"/home
zfs create -o mountpoint=/root                            "$ROOT_POOL"/home/root
zfs create -o canmount=off     -o setuid=off  -o exec=off "$ROOT_POOL"/var
zfs create -o com.sun:auto-snapshot=false                 "$ROOT_POOL"/var/cache
zfs create                                                "$ROOT_POOL"/var/log
zfs create                                                "$ROOT_POOL"/var/mail
zfs create                                                "$ROOT_POOL"/var/spool
zfs create -o com.sun:auto-snapshot=false     -o exec=on  "$ROOT_POOL"/var/tmp
zfs create                                                "$ROOT_POOL"/srv
zfs create -o com.sun:auto-snapshot=false     -o exec=on  "$ROOT_POOL"/tmp

# Set the `bootfs` property. ***TODO: IS THIS CORRECT???***
zpool set bootfs="$ROOT_POOL"/debian "$ROOT_POOL"

# Set correct permission for tmp directories
chmod 1777 /mnt/tmp
chmod 1777 /mnt/var/tmp

And here's the core part of the issue:

# Copy Debian install from source drive to target drive
for i in "${DIRS_TO_COPY[@]}"; do 
    rsync --archive --quiet --delete /"$i"/ /mnt/"$i"
for i in "${FILES_TO_COPY[@]}"; do
    cp -a /"$i" /mnt/
for i in "${VIRTUAL_FILESYSTEM_DIRS[@]}"; do
    # Make mountpoints for virtual filesystems on target drive
    mkdir /mnt/"$i"
    # Recursively bind the virtual filesystems from source environment to the
    # target. N.B. This is using `--rbind`, not `--bind`.
    mount --rbind /"$i"  /mnt/"$i"

# `chroot` into the target environment
chroot /mnt /bin/bash --login

# (Manually exit from the chroot)

# Delete copied files
for i in "${DIRS_TO_COPY[@]}" "${FILES_TO_COPY[@]}"; do
    rm -r /mnt/"$i"

# Remove recursively bound virtual filesystems from target
for i in "${VIRTUAL_FILESYSTEM_DIRS[@]}"; do
    # First unmount them
    umount --recursive --verbose --force /mnt/"$i" || sleep 0
    # Then delete their mountpoints
    rmdir /mnt/"$i"

At this last step, I get:

umount: /mnt/sys/fs/cgroup/systemd: target is busy
    (In some cases useful info about processes that
     use the device is found by lsof(8) or fuser(1).)

In case it helps: findmnt shows the full sys tree mounted twice: once at /sys and identically at /mnt/sys.

* Debian Jessie Root on ZFS, CC BY-SA 3.0, by Richard Laager and George Melikov.

  • Try using the '--lazy` option, for lazy unmount. Then it should complete the command and detach, then wait with the cleanup until the device is not busy anymore in the background. – Mio Rin May 3 '17 at 16:57
  • @Mioriin, thanks :) I tried --lazy before posting my question above, but found that /mnt/sys remained mounted for long enough afterwards that the next command I wanted to run (zpool destroy $ROOT_POOL) failed. What I didn't try was adding a sleep command between the umount and the zpool destroy, to allow some time for umount --lazy to take effect. Any idea how many seconds it should take? – user6860 May 3 '17 at 17:03
  • I wouldn't know. I haven't played with ZFS yet. You might be havng issues with propagation flags, though. Try mount --make-rslave /mnt/dev and mount --make-rslave /mnt/sys after the initial mounts (yes, on top of the previous mounting of them) when setting up for chroot. (source: wiki.gentoo.org/wiki/Chroot#Configuration ) – Mio Rin May 3 '17 at 18:17
  • Thanks for the additional suggestion :) I'm not familiar with --make-rslave. Will look into it. Found some docs here and here. – user6860 May 3 '17 at 20:21
  • 1
    @Mioriin, hats off! Adding mount --make-rslave /mnt/"$i" after mount --rbind /"$i" /mnt/"$i" does the trick - no need for --lazy or sleep. Please feel free to write it up as an answer and I'll mark it as correct :) Thanks! – user6860 May 3 '17 at 22:36

You need to add mount --make-rslave /mnt/"$i" after your first mount command, to set the correct propagation flags for those mount points.

They protect the host from changes made inside the chroot environment, and help prevent blocking situations like yours.

  • Thanks for the edit. I typed the answer up before coffee. ^_^; – Mio Rin May 4 '17 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy