6

I am building a custom Live Linux boot image based on Arch Linux. I've modified the stock initrd to contain a custom script which does the following:

  • After boot, only the initramfs exists. I first locate and mount the boot media itself on /image.
  • Then, I loopback-mount my squashfs root filesystem on /ro.
  • Then, I mount a tmpfs on /rw and create data and work directories in that directory.
  • Finally, I mount an overlayfs using /ro as the bottom layer, /rw/data as the top layer and /rw/work as the work directory. I mount this overlayfs on /new_root.
  • I now use switch_root to switch to /new_root and continue booting.

The problem with this approach is that the initramfs contents remain in RAM well after the system has booted.

Is there any way to rework the process such that, after switching to the real root, the initramfs can be removed from memory? The reason this is important is because I am trying to boot my live Linux on a small memory footprint machine (192MB) and the size of the initrd (I based it on initramfs-fallback) means that after everything has booted, I only have about 30MB of free RAM. In contrast, a typical installed Arch Linux will be using only about 18MB of RAM. This indicates to me that the unpacked initramfs (about 66MB) is definitely still in RAM, along with another 66MB or so of RAM which I can't figure out the usage from. (Perhaps for some reason the initramfs is actually duplicated in RAM?)

I could obviously shrink my initramfs (I'm working on figuring out how to specify specific modules to include in the package) but even so, a large chunk of RAM will remain used after boot on low-memory systems.


EDIT: I tried reworking my initramfs to simply mount one tmpfs on /image, and then the script creates the rw, ro, and work directories inside of /image. Then, the script does the mounts and then mounts the overlayfs on /new_root. The result is exactly the same - the RAM usage is approximately double the size of the initramfs plus the actual executables in RAM.

I requested a pre-mount break and, at that stage, only the size of the initramfs is used in RAM - this makes sense. I then requested a post-mount break and found that the RAM usage was the double amount that I observed in the booted system. So it looks like something in the various mounts is causing the RAM usage. I'm going to try tracing the process by adding free and sleep commands to the initrd scripts to see if I can isolate which command is causing the memory usage...


EDIT AGAIN: So the culprit of the "double" RAM usage was SquashFS. So, it looks like I should recompress the filesystem using gzip instead of using lzma. Ok, part of the mystery solved.

But this still doesn't deal with the initramfs continuing to use RAM after the root is switched. So that's still a mystery.

I'm wondering if perhaps the kernel tries to unmount the initramfs internally but can't do so because there's things mounted on it? But that wouldn't make sense because you will always have something mounted off the initramfs before you switch root anyway...

  • Just wanna mention the AUR and some awesome utils that can do things like dynamic tmpfs or clearing of one or more items to free room. Perhaps you could gander at the source and bash scripts to get ideas. – nannerpuss May 11 '17 at 1:22
  • Not sure that applies a whole lot in this specific scenario. The issue is with the initramfs at boot time not being expunged from RAM after the switch_root operation to move into the real root filesystem. Since my initramfs is 66MB or so, keeping it in RAM after it is no longer needed is a huge waste especially since I'm trying to boot an older machine with only 192MB of RAM, and also would like to be able to boot up an old laptop with 128MB RAM with my live image. That 66MB of RAM being used is a big deal when you have 128MB total. :-) – fdmillion May 11 '17 at 1:59
  • Gotcha. I was just messing with some ramdisk tools and thought this was more theorycraft. Hard to imagine 128MB of RAM nowadays. I feel like my alarm clock must have that :-P – nannerpuss May 11 '17 at 5:43
  • I suspect that the .cpio archive memory area that is unpacked to provide the contents of initramfs is not being freed. – Kaz Feb 6 at 22:15
2

I'm assuming you're referring to the "rootfs". And no, there is no way to get rid of it. The kernel documentation even specifically mentions this: https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt

Rootfs is a special instance of ramfs (or tmpfs, if that's enabled), which is always present in 2.6 systems. You can't unmount rootfs for approximately the same reason you can't kill the init process; rather than having special code to check for and handle an empty list, it's smaller and simpler for the kernel to just make sure certain lists can't become empty.

Most systems just mount another filesystem over rootfs and ignore it. The amount of space an empty instance of ramfs takes up is tiny.

You also don't need to worry about cleaning up rootfs. See the documentation for switch_root: http://man7.org/linux/man-pages/man8/switch_root.8.html

WARNING: switch_root removes recursively all files and directories on the current root filesystem.

Also, going back to the kernel documentation, slightly below the referenced "rootfs" documentation is another blurb which reiterates the above 2 points:

  • When switching another root device, initrd would pivot_root and then umount the ramdisk. But initramfs is rootfs: you can neither pivot_root rootfs, nor unmount it. Instead delete everything out of rootfs to free up the space (find -xdev / -exec rm '{}' ';'), overmount rootfs with the new root (cd /newmount; mount --move . /; chroot .), attach stdin/stdout/stderr to the new /dev/console, and exec the new init.

    Since this is a remarkably persnickety process (and involves deleting commands before you can run them), the klibc package introduced a helper program (utils/run_init.c) to do all this for you. Most other packages (such as busybox) have named this command "switch_root".

  • rootfs is populated by unpacking a .cpio archive that is combined with the kernel. What frees the original .cpio archive after it is unpacked? I can't find the code. Deleting the content of the unpacked rootfs isn't the same thing. – Kaz Feb 6 at 22:14
  • rootfs isn't always populated by an archive embedded within the kernel. It can be an external file. However that said, while I don't know exactly how the kernel handles this, there is nothing that forces the kernel to remain in memory. The kernel controls the memory, it can do whatever it wants. If it wants to release the portion of memory used by the image, it can. – Patrick Feb 8 at 3:38
0

So, for whatever reason the issue seems to be that if you mount anything on the initramfs other than at /new_root or anything else that was pre-mounted, the initramfs is not cleared from memory. No idea why that is.

So I looked at the scripts on the Arch Linux boot ISO and found that it's basically doing something similar to my idea, but it's creating all its temp mount directories under /run. Since /run is a tmpfs to begin with, this ends up saving a few steps anyway.

Adapting my scripts to do the same, I was ultimately able to create an ISO that boots in 128MB RAM and leaves 80MB of free RAM for applications. Not bad.

So the trick is to mount anything you need to mount under /run, and then switch_root should properly clear the root filesystem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.