NixOS installation on multi-boot system with GRUB (from Arch installation)

Question

I have one physical disk (/dev/sda) which is divided in /dev/sda1 (which is /boot bootable GRUB2 partition used and setup with Arch) and /dev/sda2 which is my cryptroot that, when unlocked, is a single large lvmpool with the full size of the partition /dev/sda2. I have installed Arch on 3 lvs, as seen in the image below. This setup is basically LVM on LUKS as described on the Arch Wiki on Dm-crypt.

Since I read lots of great things about NixOS, I am trying to install NixOS next to my current Arch install, so I created 3 more lvs similar to the arch- lvs above, and I followed the NixOS installation guide. I have specified boot.loader.grub.device = "nodev" so that NixOS will not install GRUB anywhere else, because I want to re-use my existing GRUB2 from Arch which resides on /dev/sda1. It will however write a grub.cfg file for NixOS to /boot/grub/grub.cfg that, in this case, is on the NixOS nix-root lv (/, the filesystem root in NixOS) and thus not on the actual (bootable) GRUB2 partition /dev/sda1.

However, to achieve that, I want to add an extra menu-entry to my grub.cfg and point to the configfile that the NixOS-installer has created on my nix-root lv (/, the filesystem root in NixOS), pretty much what is described here. However, I am unsure how to create an entry that will point to the nix-root lv, since it seems that GRUB wants some syntax like set root='(hd0,X)', but how can I make something like this set root aware of my lv to point to instead of to a physical disk and partition number? Both the options configfile and chainloader are not very well-described in GRUB, and neither is the set root as far as I could find.

I hope that somebody could point me in the right direction, tell me what I am doing wrong or why/how I am making this too hard on myself, to just use one bootable GRUB2 partition for multiple Linux distributions that reside on different LVM lvs on LUKS.

EDIT: I tried various options for set root= such as (lv/nix-root), (lvmpool/nix-root) and (/dev/mapper/lvmpool-nix--root), however the problem is that the LVM lvs seem not visible to GRUB yet, since at the point that GRUB loads, /dev/sda2/ is still an encrypted LUKS cryptroot.

Answer 1

With the help above from Alexander Batischev, and some very useful information from Reddit user cookie_enthusiast on the /r/linuxquestions page on Reddit, I managed to get this working.

It turns out that GRUB2 works well with UUIDs, and not so well with device names. With this knowledge in mind, we need the following 4 UUIDs at hand, before we can (manually) create an extra GRUB menuentry to our NixOS grub.cfg configfile:

1. The UUID of the LUKS device.
2. The UUID of the LVM Volume Group.
3. The UUID of the LVM Logical Volume.
4. The UUID of the filesystem containing the grub.cfg file we want to load via the configfile directive in GRUB2.

I will list here how to obtain these four UUIDs:

1. Execute cryptsetup luksUUID /dev/sda2 and remove all the dashes (-) from the UUID, a0cb535a-8468-485f-a220-a5f49e85c9f4 would become a0cb535a8468485fa220a5f49e85c9f4 in my case.
2. Execute vgdisplay and look for the VG UUID, lvmpool with UUID 5atKN9-PQBi-T9wb-Iyz8-qP4y-HN2E-c5uLOT in my case.
3. Execute lvdisplay and look for the LV UUID of the LV Name or LV Path that contains your grub.cfg file, nix-root or /dev/lvmpool/nix-root with UUID C9zkjF-IHu0-qQkP-KgLf-8rAy-TVPu-HQ7gtj in my case.
4. Execute lsblk -p -o +UUID and look for the UUID of the Device Path that contains your grub.cfg file, /dev/mapper/lvmpool-nix--root with UUID cc6a06bb-336f-4e9f-a5f0-fdd43e7f548f in my case.

This will allow you to create the following extra GRUB menuentry for referencing our NixOS grub.cfg configfile, which is on my nix-root lv and because of the boot.loader.grub.device = "nodev"; in my /etc/nixos/configuration.nix there is no GRUB installed for my NixOS installation (in Arch, this would go into /etc/grub.d/40_custom):

menuentry 'NixOS' {
    insmod crypto
    insmod cryptodisk
    insmod luks
    insmod lvm
    cryptomount -u a0cb535a8468485fa220a5f49e85c9f4
    set root='lvmid/5atKN9-PQBi-T9wb-Iyz8-qP4y-HN2E-c5uLOT/C9zkjF-IHu0-qQkP-KgLf-8rAy-TVPu-HQ7gtj'
    search --fs-uuid --set=root cc6a06bb-336f-4e9f-a5f0-fdd43e7f548f
    configfile '/boot/grub/grub.cfg'
}

To clarify this even more, this contains some literal values such as lvmid which is not supposed to be replaced with the name or ID of your LVM. This is not properly documented anywhere, it seems. The same issue applies for when your put the UUID of your LUKS device in the cryptomount -u line with dashes in it, GRUB will just tell you Press any key to continue, which is (obviously) not very helpful.

The bare template for a manual GRUB menuentry to boot from crypt -> LVM -> root with an LVM on LUKS setup would thus be:

menuentry 'NixOS' {
    insmod crypto
    insmod cryptodisk
    insmod luks
    insmod lvm
    cryptomount -u <LUKS UUID without dashes>
    set root='lvmid/<LVM Volume Group UUID>/<LVM Logical Volume UUID>'
    search --fs-uuid --set=root <Filesystem UUID>
    configfile '/boot/grub/grub.cfg'
}

For those who are interested in the other half also, I modified my /etc/nixos/configuration.nix file to look like this:

boot.loader.grub.enable = true;
boot.loader.grub.version = 2;
boot.loader.grub.device = "nodev";
boot.initrd.luks.devices = [ { name = "cryptroot"; device = "/dev/sda2"; preLVM = true; } ];