Out of memory on "Loading initial ramdisk" after kernel upgrade (4.15 to 4.19) on Ubuntu 18

Question

I'm trying to upgrade my kernel to 4.19, because I need to run some benchmarks that require it (with some kernel options turned on). I'm completely stuck as to why this isn't working. I did two ubuntu 18 clean installs already, download the 4.19 kernel, make oldconfig (or olddefconfig), install modules and the kernel itself. After a reboot, the output just says

Loading Linux 4.19.237
Loading initial ramdisk
error: out of memory
Press any key to continue

After a key press it just shows the initial boot messages and an error stack:

If I reboot, the older (4.15) kernel still boots and works perfectly.

Both entries on grub.cfg are similar

menuentry 'Ubuntu, with Linux 4.19.237' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.237-advanced-cbc1623f-b651-454a-87dd-da2056dd55\05' {
                recordfail
                load_video
                gfxmode $linux_gfx_mode
                insmod gzio
                if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
                insmod part_gpt
                insmod ext2
                set root='hd0,gpt2'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  cbc1623f-b651-454a-87dd-da2056dd5505
                else
                  search --no-floppy --fs-uuid --set=root cbc1623f-b651-454a-87dd-da2056dd5505
                fi
                echo    'Loading Linux 4.19.237 ...'
                linux   /boot/vmlinuz-4.19.237 root=UUID=cbc1623f-b651-454a-87dd-da2056dd5505 ro
                echo    'Loading initial ramdisk ...'
                initrd  /boot/initrd.img-4.19.237
        }

menuentry 'Ubuntu, with Linux 4.15.0-20-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.15.0-20-generic-advanced-cbc1623f-b651-45\4a-87dd-da2056dd5505' {
                recordfail
                load_video
                gfxmode $linux_gfx_mode
                insmod gzio
                if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
                insmod part_gpt
                insmod ext2
                set root='hd0,gpt2'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  cbc1623f-b651-454a-87dd-da2056dd5505
                else
                  search --no-floppy --fs-uuid --set=root cbc1623f-b651-454a-87dd-da2056dd5505
                fi
                echo    'Loading Linux 4.15.0-20-generic ...'
                linux   /boot/vmlinuz-4.15.0-20-generic root=UUID=cbc1623f-b651-454a-87dd-da2056dd5505 ro
                echo    'Loading initial ramdisk ...'
                initrd  /boot/initrd.img-4.15.0-20-generic
        }

I'm totally out of ideas. I have recompiled the kernel many times, disabled TPM in the BIOS, updated the BIOS, reinstalled ubuntu without LVM, tried recovery mode, changed from RAID to AHCI (there's only one disk).

Any ideas?

Edit: After searching around and changing configs everywhere I made it boot but I still run into the out of memory error, which requires me to press a key. I've enabled a bunch of VIRTIO, AHCI and other BLK parameters in the kernel. Then,when I pressed a button after the out of memory error, it showed a completely different list of UUIDs:

Which made me try not using UUIDs. So I changed grub to use root=/dev/sda2. One out of memory error and a key press after, I'm greeted by a long boot log and a login prompt.

For some reason the network device changed names (from enp4s0 to enp3s0) with the new kernel, so I had to edit the netplan file and I had network.

Now, I still suffer from the out of memory error and I have no idea why. I need to fix this because I use this machine remotely and can't go to it whenever I need to reboot it. Still open to any ideas as to why this is happening in such a simple kernel upgrade.

Answer 1

Turns out the initrd image was huge. 500MB in comparison to the default's 50MB. The key to reducing size was here: How to reduce the size of the initrd when compiling your kernel?

Basically: In /etc/initramfs-tools/initramfs.conf change MODULES to MODULES=dep.

When installing modules, pass a variable to strip debug symbols: make INSTALL_MOD_STRIP=1 modules_install

After this the out of memory error is gone, and so are the vfs errors for some reason. The initrd is less than 50MB now. Before applying these changes I did upgrade all packages, including the kernel, but it might not have had any impact.

Answer 2

In addition to changing MODULES you can also improve the compression level of initrds.

In /etc/initramfs-tools/initramfs.conf

MODULES=dep
COMPRESS=xz

Run the following after updating the conf file:

sudo update-initramfs -c -k all
sudo update-grub