Problem with booting a newly LUKS encrypted root partition

Question

I didn't encrypt my root partition when I was installing my Ubuntu 18.04 when 18.04 first came out (first few months of it's release) So for reasons I decided to encrypt it now.

I know this question has been asked before (I posted the links of some of the asked ones, actually!) but I posted this because this issue is specific to me, so please don't remove my question, thank you very much, dear moderators and admins. I did not find anyone who had an issue same as mine, and after doing some MAJOR linux stuff, I had to ask for help since I was unable to solve this by myself. (Which hasn't happened a lot over my past 11 years of GNU/Linux-*nix experience. I solved/fixed my problems by myself and dear Google like 99% of the times... and as you know problems happen a lot in linux! :D )

Anyway, Here are the steps of which I did it: (In a 18.04 live usb)

sudo cryptsetup -v -y -c aes-xts-plain64 --key-size 512 --hash sha512 luksFormat --uuid=049172c6-5376-4b9c-bd27-b503b6f25423 /dev/sda5
sudo cryptsetup -v luksOpen /dev/sda5 myroot
sudo mkfs.ext4 -m 0 /dev/mapper/myroot
#Then for copying the contents of my root partition I used dd:
sudo dd if=/dev/sda6 of=/dev/mapper/myroot bs=4M
#/dev/sda6 is a duplicate of my original sda5 root...
#After I dd'ed, there was unallocated space on the partition so I did a check on it in GParted (Which also applies cryptsetup resize command to it)
sudo mount /dev/mapper/myroot /media/myroot #I created /media/myroot beforehands

Then after this I did A LOT of things I thought would help for booting this new encrypted root of mine with GRUB, and also A LOT OF MORE things I found by searching. So I basically tried everything people suggested in these links:

https://askubuntu.com/questions/1134998/booting-19-04-from-luks-system-drive

Set initramfs to prompt for luks passowrd on startup on Mint 18?

https://askubuntu.com/questions/1006867/cant-get-ubuntu-to-boot-from-luks-lvm-group-on-external-drive-on-imac-with-re

https://askubuntu.com/questions/729673/ubuntu-full-disk-encryption-with-encrypted-boot

https://askubuntu.com/questions/450895/mount-luks-encrypted-hard-drive-at-boot

https://askubuntu.com/questions/1082131/how-to-get-grub-to-boot-from-a-newly-encrypted-partition

(And a lot of more links)

Here's the output of fdisk -l:

Disk /dev/sda: 238.5 GiB, 256060514304 bytes, 500118192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: FE0857E8-E0DE-40A0-96C5-C4FEC80B8742

Device         Start       End   Sectors   Size Type
/dev/sda1       2048    534527    532480   260M EFI System
/dev/sda2     534528   1067007    532480   260M EFI System
/dev/sda3    2582528   4630527   2048000  1000M Lenovo boot partition
/dev/sda4    4892672 259438591 254545920 121.4G Microsoft basic data
/dev/sda5  259438592 332343295  72904704  34.8G Linux filesystem
/dev/sda6  332343296 400898047  68554752  32.7G Linux filesystem
/dev/sda7  410068992 425521151  15452160   7.4G Microsoft basic data

These are my UUIDs:

/dev/sda5:
LUKS UUID: 049172c6-5376-4b9c-bd27-b503b6f25423
Partition UUID: 1db5df50-7000-48df-a281-74bad5689ce1

/dev/sda6:
Former UUID which I recently changed it to a new one because it was same as sda5's due to using dd for copying root filesystem to new LUKS partition: 1db5df50-7000-48df-a281-74bad5689ce1
New UUID: 7876a195-7219-4440-892a-61b57c706443

And finally the things I tried in GRUB: http://s000.tinyupload.com/index.php?file_id=00739971820320475700

UPDATE: So basically when I try booting with the configs provided in the zip file that I posted it's link above, for each conf I get a different error or problem or something... Like when I added some stuff I found in the links I posted in the /etc/default/grub and crypttab and fstab and grub.cfg of the unencrypted root (sda6) and booted, while on the quiet splash screen it asks for the luks PW but at the end of the line where it asks me this it says something like "Enter pw for sda5 luks to mount on /!", IDK (or kinda know!) why at the end it says /! instead of just /.

And when I try booting from sda5 sometimes (with one of the confs) it does ask for the passphrase on grub but then it drops into busybox initramfs prompt or it says "No such device found with 1db5df50-7000-48df-a281-74bad5689ce1" or something similar...

I think the problem is either what dear telcoM said in his answer about initramfs, or with the UUIDs and the UUIDs I put in the conf files you know? But hopefully it's because of initramfs, I'm gonna try telcoM's suggestion and see what happens.

Any help will be much appreciated I really don't want to reinstall and encrypt from the installer, I need the things I have on my ubuntu and I really can't do a reinstall, this ubuntu install of mine is just so perfect, I've had a lot of experience with GNU/Linux (Over 10 years now) and had a lot of linux installations and I'm no noob, but I just can't loose all my data or configurations or customizations I made... I know I can back them up but I just... you know?

Answer 1

Your LUKS container was created with the same size as the underlying partition /dev/sda5, but since the /dev/sda6 you dd'd into it was smaller than that, the image of /dev/sda6 would not completely fill your /dev/mapper/myroot. A simple resize2fs /dev/mapper/myroot would have been enough to fix it.

You're talking a lot about things you have done, but very little about the actual symptoms: exactly what happens when you attempt to boot with your new encrypted root? What is the actual error message, if there is any? Does it recognize the existence of your encrypted partition, and prompt for a passphrase, or is it failing to detect it by UUID? Or does GRUB succeed in loading the kernel and initramfs files, but then the initramfs fails to mount the encrypted root filesystem?

You seem to be using UEFI native boot scheme. That's OK, but it brings with it the possibility that Secure Boot might be enabled. If so, it will restrict GRUB from loading any other GRUB modules from /boot/grub/x86_64-efi once the main grubx64.efi has been loaded, so all the necessary modules must be built into the main grubx64.efi bootloader file.

When grub-install is executed, it will typically run grub-mkimage, which will be able to include any GRUB modules into the grubx64.efi it's building. Normally, grub-install will only include the minimum necessary number of modules... that is, only the modules that were detected as necessary at the time grub-install was executed. So it's possible that your GRUB is failing because it does not have the encryption support built in, because the root filesystem wasn't encrypted when grub-install was executed.

To fix that, you have two options. Either use grub-install to re-install GRUB when you have chrooted into your new root filesystem and have a valid /etc/crypttab entry in place for it... or more simply, make sure you have the grub-efi-amd64-signed package installed, and replace the existing grubx64.efi on your ESP (= probably /boot/efi/EFI/ubuntu/grubx64.efi) with /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed:

cp /boot/efi/EFI/ubuntu/grubx64.efi /boot/efi/EFI/ubuntu/grubx64-backup.efi
cp /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed /boot/efi/EFI/ubuntu/grubx64.efi

This is a pre-built UEFI GRUB bootloader that is signed to be maximally acceptable for Secure Boot, and it has all the GRUB modules built in, so it should be noticeably bigger than your existing grubx64.efi file.

Even if Secure Boot is disabled, the grubx64.efi.signed file should work just fine: the fact that it has all the GRUB modules built in is still useful.

But if your GRUB can successfully load your kernel and initramfs files, then the problem must be in the initramfs stage. If your Ubuntu has the cryptsetup-initramfs package available, make sure it's installed. Then, again while you're chrooted onto your new encrypted root and have a valid /etc/crypttab entry for it in place, run update-initramfs -u to rebuild your initramfs. Running it inside the chroot should let it auto-detect that the root filesystem is encrypted, and include the necessary tools and scripts into the initramfs file.