Overlayfs not working on kernel 6.1.31 on a Gentoo VPS

Question

I have a VPS with Gentoo on it and the kernel overlay fs is not working.

These are the kernel options related to overlay

compiled into the kernel

cat .config | grep -i overlay

CONFIG_EFI_CUSTOM_SSDT_OVERLAYS=y
CONFIG_OVERLAY_FS=y
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
# CONFIG_OVERLAY_FS_XINO_AUTO is not set
# CONFIG_OVERLAY_FS_METACOPY is not set

compiled as a module

cat .config | grep -i overlay

CONFIG_EFI_CUSTOM_SSDT_OVERLAYS=y
CONFIG_OVERLAY_FS=m
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
# CONFIG_OVERLAY_FS_XINO_AUTO is not set
# CONFIG_OVERLAY_FS_METACOPY is not set

Message from dmesg when loaded into the kernel

dmesg | grep -i overlay
[   10.144242] request_module fs-overlay succeeded, but still no fs?

Message from dmesg when compiled as a module

dmesg | grep -i overlay
[   10.000001] request_module fs-overlay succeeded, but still no fs?

When compiled as module, the module is on the right place

locate overlay.ko
/lib/modules/6.1.46-gentoo/kernel/fs/overlayfs/overlay.ko
/usr/src/linux-6.1.46-gentoo/fs/overlayfs/.overlay.ko.cmd
/usr/src/linux-6.1.46-gentoo/fs/overlayfs/overlay.ko

cat /proc/filesystems | grep -i overlay comes empty.

The system is:

uname -a
Linux patito 6.1.31-gentoo #3 SMP PREEMPT_DYNAMIC Sun Jun 25 09:04:44 GMT 2023 x86_64 QEMU Virtual CPU version 2.5+ AuthenticAMD GNU/Linux

There is only one partition for the whole system running ext4, except /boot on a different ext4 partition.

The system has 2GB memory and plenty of space. I downloaded a snapshot of the system and ran it on my computer with virt-manager adding more memory and cpu cores, just for testing, and I got same results. Did a qemu-img convert to change from .img to qcow2 in case that helped but no change.

I have tried different kernels, recompiling with different options, removed all overlayfs sections, rebooting and reenabling them, searched for similar problems over Internet but I haven't found anything, any theories will be appreciated.

The overlayfs in the kernel is compiled as part of the kernel as shown above but I also tried to compile it as a module, it didn't load automatically and when I tried to load it manually with modprobe, nothing happened but it didn't load, after running lsmod I wasn't there. I'll try again and add the kernel configuration to the above section.

Answer 1

Googling the message you saw in dmesg lead to the following file in puppet source code describing the cause of the issue:

    # On a fresh node overlay may be unloaded automatically by the OS
    # if no fs needs it. In this case the kern.log should look like:
    # kernel: request_module fs-overlay succeeded, but still no fs?
    # This may lead to unwanted side effects, like Docker not finding
    # the overlay kernel module loaded and falling back to
    # the device-mapper storage driver.
    # Therefore we explicitly load the overlay module when the overlayfs
    # option is true.
    kmod::module { 'overlay':
        ensure => 'present',
    }

So on your host you will see the overlay kernel module isn't loaded if you run the following command:

lsmod |grep overlay

So you can just need to reload the overlay kernel module.

modprobe overlay

Answer 2

After posting a reply to aviro, I realized that my kernel version, 6.1.46, was not the same as in the original message, 6.1.31. That could perfectly be because I was copying most of the information from the original site where I posted the question, the docker community forums, and I have done a lot of tests and attempts to fix the problem, but that got me thinking about the kernel version loaded.

Checked that /usr/src/linux was pointing to the right kernel, 6.1.46, and it was, recompiled with overlay fs as a module to post the information related on the original post, did make && make modules_install && make install followed by grub-mkconfig -o /boot/grub/grub.cfg and rebooted.

After rebooting, I ran a uname -a again and the kernel was 6.1.31, why?

May be I did the mkconfig with the wrong path so I did it again, carefully, and rebooted again. Same thing, still 6.1.31.

After that reboot I checked /boot/grub/grub.cfg and the right option, 6.1.46 was there as the first option to boot the system. So just in case there was a problem with some random line not being updated, I did a rm /boot/\*6.1.31* and ran grub-mkconfig -o /boot/grub/grub.cfg, checked with grep and there was no trace of 6.1.31, only 6.1.46.

Rebooted again and there it was, 6.1.31 again. But everything looked fine, it should have booted 6.1.46.

Then I rebooted and controled manually the boot process, editing by hand the numbers to load 6.1.46 but the system failed, couldn't find the kernel.

That was key.

I changed back to 6.1.31, let it boot and checked my mount points (/boot is a different partition).

And there it was, or should I say, it wasn't, /boot was not being mounted. At some point on the life of this system, I made a mistake on /etc/fstab and let /boot with noauto so it was not being mounted.

I changed that, removed the noauto for /boot on /etc/fstab, rebooted the system and voila!. The system was 6.1.46 and overlayfs was working.

Although, and this is the new part that I don't get. even when /boot was a folder and not a partition, the 6.1.46 kernel was there and the /boot/grub/grub.cfg had the correct configuration, so why was it not being read and loaded?. On the grub menu presented at boot, there was only one option, 6.1.31 but I couldn't find it on the disk not with locate and not with find, so where is it? how was it being loaded? the /boot/grub/grub.cfg read from the disk with cat had 6.1.46.

Well, after this long story, I have solved the problem and got a new enigma, which, for now is going to remain unsolved because I had used too much time already.

Thanks for the help, with out trying to answer that last comment from aviro I wouldn't have stumbled into the solution.