Boot Linux system from a subdirectory on a partition?

Question

I would like to try to set up a computer so that it has multiple Linux installs all in the same filesystem. For example, the filesytem would have 3 folders: /Ubuntu_Precise, /Ubuntu_Oneiric, and /Ubuntu_Natty.

(I know you can do this with BTRFS and subvolumes, but I would like to use EXT4 for speed).

I once set up multiple installs of different distros using BTRFS, and from getting that working, I know Grub does just fine with booting the vmlinuz and initrd image from 'nonstandard' paths. But when I was doing the BTRFS thing, there was the rootflags=subvol=@<subvolume_name> that told the kernel to mount that subvolume as / in the filesystem. Is there any argument that you could pass the kernel that would make it bind mount a subfolder in a partition as / and then boot?

I think for the other parts, I'm fairly close. I know how to specific a bind mount in /etc/fstab. Also, from when I set up my system with multiple linux installs in BTRFS subvolumes, I'm used to installing a distro in a VM and then migrating it using rsync, so I'm not too worried about what I would need to do to get the right configuration, I'm just trying to find out what the right configuration would be. Once I know that, I should be able to do the migration into the subfolders and file editing easily enough.

I already know about virtualization and partitions, but that's not what I'm looking for. The target computer does not have enough power to do virtualization, and partitions do not share free space. I'm looking to set up a system that dual/triple/quad/etc boots linux distros, but that does it with one filesystem, so that there is no case of "I have free space, but it's in the wrong partition!'

If anyone has suggestions how to edit my question or its title to be clearer, I'm all ears.

Answer 1

Short answer -- there is as far as I know no out of the box working solution for your specific requirements. You will have to adjust each initramfs of each distribution to support your specific needs.

Long answer -- yes it is possible. Nowadays most Linux distributions use an initramfs which will be loaded into memory by the bootloader and then unpacked by the kernel. There it will run /sbin/init which is responsible for setting up the early userspace (running udev, loading modules, starting plymouth, asking for crypto passphrase, setting up the network for network mounts, … you name it). As you can run your own scripts and evaluate custom boot parmaters.

Example for Debian

If you are using Debian (should be the same with Ubuntu) you should be able to place a script in /etc/initramfs-tools/scripts/init-bottom/ which will be executded before init is started. For more information about the script, the different directories and the layout have a look at man initramfs-tools. You will have to adjust rootmnt and add the target directory.

Sample (untested) script which should be installed either as /etc/initramfs-tools/scripts/local-bottom/00-myroot or /usr/share/initramfs-tools/scripts/init-top/00-myroot :

#!/bin/sh -e

PREREQS=""

prereqs() { echo "$PREREQS"; }

case "$1" in
  prereqs)
  prereqs
  exit 0
;;
esac

for opt in $(cat /proc/cmdline); do
  case $opt in
    rootdir=*)
      new_mntdir="${opt#rootdir=}"
      ;;
    esac
done

if [ -n "$new_mntdir" ] ; then
  echo rootmnt="$rootmnt/$new_mntdir" >> /conf/param.conf
fi

The idea is to adjust rootmnt which is used in the initramfs init script to start/execute the real init. As the root device is already mounted in the init-bootom stage you can just adjust/alter the target directory.

To use this script just add a new boot parameter, copy the script, make it executable, regenerate your initramfs and add a boot parameter for your Linux distribution, e.g. rootdir=/Ubuntu_Precise.

Answer 2

Here are two ways which work in ubuntu focal&bionic (and likely elsewhere). i've not enough rep to comment, but, bionic:/usr/share/initramfs-tools/init looks in /etc/fstab for /usr right after calling mountroot and before calling the *-bottom scripts, so adding an init-bottom script (as suggested in another answer here) is "too late". instead i recommend these:

#!/bin/bash -f
#copyleft 2018 greg mott

#set a subdirectory as root (so multiple installs don't need partitions)
#these work in ubuntu bionic, might need tweaking to work elsewhere
#1st choice:  tweak initramfs-tools/scripts/local
#   pro:  $sub becomes root directly, nothing gets any chance to see the partition root
#   con:  requires the subdirectory's initramfs/initrd to be tweaked and rebuilt
#2nd choice:  specify this scriptfile as init= on the kernel commandline
#   pro:  no need to rebuild initramfs
#   con:  bin/bash in partition root must be executable by $sub/vmlinux (partition root older than $sub likely will work)
#   con:  if the partition root etc/fstab mounts /usr, the $sub initramfs will mount the partition root /usr
#   con:  additional initramfs scripts might also look in the partition root rather than $sub

#for either choice copy /etc/grub.d/40_custom to /etc/grub.d/07_custom and add one or more menuentries that specify subroot:
#menuentry "subroot foo" {
#     echo "subroot foo"
#              sub=/foo
#             uuid=22e7c84a-a416-43e9-ae9d-ee0119fc3894         #use your partition's uuid
#     search --no-floppy --fs-uuid --set=root $uuid
#            linux $sub/vmlinuz ro root=UUID=$uuid subroot=$sub
#     echo "initrd $sub/initrd.img"
#           initrd $sub/initrd.img      #works in recent releases where the /initrd.img softlink is relative
#}

#for the 2nd choice, in addition to subroot= on the kernel commandline also specify:
#   init=/path/to/script        #pathname from partition root to this scriptfile (chmod 744)

#for the 1st choice, the tweak for bionic:/usr/share/initramfs-tools/scripts/local is replace:
#          mount ${roflag} ${FSTYPE:+-t ${FSTYPE} }${ROOTFLAGS} ${ROOT} ${rootmnt}
#          mountroot_status="$?"
#with:
#          set -x
#          karg=" $(cat<proc/cmdline) " m=${karg#* subroot=}
#          [ "$m" = "$karg" ]||subroot=${m%% *}                                  #extract subroot from kernel commandline
#          [ $subroot ]&&part=part||part=$rootmnt                                #no subroot, just mount partition as root
#          mkdir part
#          mount ${roflag} ${FSTYPE:+-t ${FSTYPE} }${ROOTFLAGS} ${ROOT} $part&&  #mount partition
#             if [ "$subroot" ]
#             then mount --bind part/$subroot $rootmnt&&                         #mount subroot
#                  umount part                       #&&sleep 15                 #unmount partition root (uncomment &&sleep for time to watch)
#             fi
#          mountroot_status="$?"
#          [ $mountroot_status = 0 ]||sleep 90                                   #if error pause to see it
#          set +x
#once you've edited /usr/share/initramfs-tools/scripts/local, update-initramfs -u will rebuild for the current kernel,
#and it will automatically build into every new initrd/initramfs installed thereafter

subroot(){ karg=" $(cat<proc/cmdline) " m=${karg#* subroot=}
           [ "$m" = "$karg" ]||subroot=${m%% *}                 #extract subroot from kernel commandline
           [ $subroot ]||return 0                               #no subroot, just proceed in partition root
           while read -r m r m
           do for m in $M x                                     #build list of what's already mounted
              do    [[ $r = $m* ]]&&break                       #exclude subtrees (eg dev/**)
              done||[[ $r = /   ]]||M=$M\ $r                    #exclude /
           done<proc/mounts
           (set -x;mount --bind $subroot mnt)||{ set -x         #mount subroot
                                                 sleep 30          #if not found pause to see error
                                                 return 0;}        #then reincarnate as partition root init
           for m in $M
           do (set -x;mount -n --move $m mnt$m)||return         #move listed mounts to subroot
           done
           set -x
           cd           mnt&&
           pivot_root . mnt&&                                   #subroot becomes root
           umount -l    mnt&&                                   #unmount partition root
          #sleep 15        &&                                   #so far so good?  uncomment for time to look
           exec chroot . init "$@"                              #reincarnate as subroot init
}
subroot "$@"&&exec init "$@"||exec bash                         #land in a shell if moves or pivot fail

Answer 3

Booting different linux without messing with the partition table is interesting for different purposes, an alternative solution to a shared filesystem is to use loop volumes, here the few changes needed supposing you have a /debian loop file/volume into the /dev/sdb1 filesystem (I'm using current GNU/Debian sid/unstable for both main and loop os).

/etc/grub.d/40_custom: # outside from loop volume
menuentry 'label' --class gnu-linux --class gnu --class os {
    ...
    loopback loop (hd2,msdos1)/debian
    linux   (loop)/boot/vmlinuz root=/dev/sdb1 loop=/debian ro
    initrd  (loop)/boot/initrd
}

Arguments defined into grub as linux command line are set to env by initrd /init, so:

ROOT=/dev/sdb1
rootmnt=/root
loop=/debian 

loop allow to mount the volume over "itself", the default script flow do a mount /dev/sdb1 /root we just optionally remount the /dev/sdb1 as rw if it was ro then always append a mount -o loop /root/debian /root.

/etc/initramfs-tools/scripts/local-bottom/loop: # inside the loop volume
#!/bin/sh

[ "$1" = "prereqs" ] && echo && exit 0

if [ -n "${loop}" ]; then
        if [ "${readonly}" = "y" ]; then
                roflag=-r
                mount -o remount,rw ${ROOT} ${rootmnt}
        else
                roflag=-w
        fi
        mount ${roflag} -o loop ${rootmnt}${loop} ${rootmnt}
fi

Need also to pre-load some module into the initram (then don't forget to run update-initramfs)

/etc/initramfs-tools/modules: # inside the loop volume
...
loop
ext4

Don't know how much using loop influence performances or waste resources, I'm wondering if mounting ext4 over ext4 double the probabilities of a filesystem failure, but guess some tuneup could be done. Maybe there's a better way to use loop, less hackish, if there's please let me know because I haven't found.

Answer 4

This is not an answer but I want to clarify some point about Ulrich's answer and comments (I can't comment above).

The solution Ulrich propose "may" work (untested yet) but then you will get a non-remountable filesystem. As a workaround (IMHO ugly) you can mount the fs as rw before chrooting (as suggested here) but be careful about broken init scripts. I guess this workaround have more side effects (like broken fs trying to remount ro and failing).

I'm using kernel 3.2 with ext4 and mounting an already mounted dev inside the chroot still give EBUSY as psusi commented.