How to use OverlayFS to protect the root filesystem?

Question

How do I correctly use OverlayFS to protect my root filesystem?

I have an embedded system that boots and runs from an SD card. Since it will experience sudden power loss, I want to protect the root file system. OverlayFS seems like the easiest solution, but the examples I find typically don't involve the root file system and/or use a tmpfs which is not good for me since I have very little memory.

I am using Linux Kernel 4.4.0 with CONFIG_OVERLAY_FS=y enabled. My file system is xenial-base-armhf.tar.gz and I have done apt install -y overlayroot.

My SD Card looks like:

# fdisk -l /dev/mmcblk1
Disk /dev/mmcblk1: 29 GiB, 31104958464 bytes, 60751872 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7f56a0ab

Device         Boot    Start      End  Sectors  Size Id Type
/dev/mmcblk1p1 *        2048  1050623  1048576  512M  c W95 FAT32 (LBA)
/dev/mmcblk1p2       1050624  1052671     2048    1M da Non-FS data
/dev/mmcblk1p3       1052672  7344127  6291456    3G 83 Linux
/dev/mmcblk1p4       7344128 60751871 53407744 25.5G  5 Extended
/dev/mmcblk1p5       7346176 13637631  6291456    3G 83 Linux
/dev/mmcblk1p6      13639680 60751871 47112192 22.5G 83 Linux

Prior to creating the OverlayFS, everything is mounted as:

# mount
/dev/mmcblk1p3 on / type ext4 (rw,noatime,data=ordered)
devtmpfs on /dev type devtmpfs (rw,relatime,size=170440k,nr_inodes=42610,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-    cgroups-agent,name=systemd)
configfs on /sys/kernel/config type configfs (rw,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=35752k,mode=700)
/dev/mmcblk1p6 on /opt type ext4 (rw,noatime,data=ordered)
/dev/mmcblk1p5 on /overlay type ext4 (rw,noatime,data=ordered)

My plan was to use /dev/mmcblk1p5 as the overlay file system mounted at /overlay.

# tree /overlay
/overlay
├── lost+found
├── root-fs
└── work

Either I'm doing things wrong, or I have some configuration issue, because:

# mount -t overlay overlay -o lowerdir=/,upperdir=/overlay/root-fs,workdir=/overlay/work /
# mount
/dev/mmcblk1p3 on / type ext4 (rw,noatime,data=ordered)
devtmpfs on /dev type devtmpfs (rw,relatime,size=170440k,nr_inodes=42610,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
configfs on /sys/kernel/config type configfs (rw,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=35752k,mode=700)
/dev/mmcblk1p6 on /opt type ext4 (rw,noatime,data=ordered)
/dev/mmcblk1p5 on /overlay type ext4 (rw,noatime,data=ordered)
overlay on / type overlay (rw,relatime,lowerdir=/,upperdir=/overlay/root-fs,workdir=/overlay/work)

Looks like it worked, yet if I create a file like:

# touch /root/test_file_write

Then, poweroff and look at the SD card in my desktop, I see /dev/mmcblk1p3/root/test_file_write not what I expected /dev/mmcblk1p5/root-fs/root/test_file_write.

Should this work?

Answer 1

This answer is based on my own experience, but not with embedded devices. Maybe it's useful to others who stumble on this setup -- you should adapt it to your specific circumstances, or at least hopefully learn a thing or two from it.

A simple method is to hijack the place when the root filesystem gets mounted (in initramfs, otherwise hijack the init process), and mount the overlay there, then continue as usual. If you mount it without initramfs, don't forget to move all the virtual filesystems (e.g. /proc and /sys) in your script before passing control to the normal init.

If you don't want an initramfs, then Mathieu Maret's linked solution to how it is done in raspberry should work. Basically, you override the init process/script with your own via a command line to the kernel. Let's say you create the script into the root filesystem at /sbin/init-overlay, then you need to add the init=/sbin/init-overlay to the kernel boot parameters of your bootloader's config.

The init-overlay script can do anything before it passes control to init, in this case it mounts the overlay to another directory and then chroot into it.

---

A possible way to do it with initramfs is to simply hijack the /init script inside the initramfs after the point it mounts the root filesystem. For example, let's say you want to have an overlay root, but also access to the original mount points, at /run/rootfs/ro and /run/rootfs/rw (the former being the read-only root, the latter being the modifications done, the upperdir) after the system is up and running. I'll also assume that the drive that boots your system has a root.squashfs file that contains the read-only root filesystem and you want to mount it. Let's say you also want access to this drive, for convenience, at /media/drive or similar, again, after the system is up and running.

The reason we hijack the initramfs script is because this offers us more flexibility in the process than using some premade parameters that end up running default script commands anyway. So you need to edit the boot config to pass the kernel the information that the "root" filesystem is actually the one where root.squashfs is found, as we'll mount it later.

A typical command with syslinux.cfg on a VFAT partition would be (change the UUID if needed):

label linux
    linux vmlinuz
    append root=UUID=ABCD-1234 rootfstype=vfat rootflags=ro,umask=022,quiet ro quiet splash
    initrd initrd.img

This assumes you'll place the root.squashfs on the VFAT partition, which might not be ideal (what you need to specify as root= above is the partition or filesystem that contains the root.squashfs or similar, such as the real root filesystem if you don't want it compressed). However, I'm explaining it assuming you place it on the boot partition itself. I don't know what kind of embedded system you run so you'll have to use your own judgement here.

First you'll have to extract the initramfs somewhere in /tmp so you can modify its /init script. Don't forget to do it as root (superuser) to retain ownership properly before we pack it back. You can probably script the whole thing after you understand how it can be done. For example unpack it to /tmp/initramfs for edits:

mkdir /tmp/initramfs 2>/dev/null; (cd /tmp/initramfs && zcat /initrd.img | sudo cpio -idmv)

Now we need to find where the default script mounts the root. Look for something like this in /tmp/initramfs/init (edit it as root):

maybe_break mount
log_begin_msg "Mounting root file system"
. /scripts/${BOOT}
parse_numeric ${ROOT}
maybe_break mountroot
mountroot
log_end_msg

You don't have to understand how this works. All you need to understand is that this mounts the normal filesystem containing your root.squashfs to a mountpoint that apparently is given via the ${rootmnt} shell variable.

In other words, what we have at ${rootmnt} at this point is our VFAT partition (or whatever we specified via the root= command line parameter). This script would now do other things such as move all the virtual filesystems to the ${rootmnt} mount point so we need to be sure to do all of our custom stuff after the code above.

All you have to do is just insert something like this after the code above in the initramfs's /init:

# create some temporary directories under the initramfs's /run
# they will be our mountpoints and such, which will get moved
# by the default script to the actual root filesystem...
mkdir -m 755 /run/rootfs
mount -t tmpfs -o size=90%,mode=755,suid,exec tmpfs /run/rootfs
mkdir -m 755 /run/rootfs/drive /run/rootfs/ro /run/rootfs/rw /run/rootfs/.workdir

# move the original root that was mounted, temporarily
mount -n -o move "${rootmnt}" /run/rootfs/drive

# mount the squashfs and then the overlay to our designated locations
mount -t squashfs -o defaults,ro /run/rootfs/drive/root.squashfs /run/rootfs/ro
mount -t overlay -o lowerdir=/run/rootfs/ro,upperdir=/run/rootfs/rw,workdir=/run/rootfs/.workdir root "${rootmnt}"

# at this point we have our overlay root at ${rootmnt}!
# however, move the drive's filesystem mount to the new root
# this allows it to be accessed afterwards from /media/drive
# NOTE: this assumes you have the /media/drive dir in the root squashfs
mount -n -o move /run/rootfs/drive "${rootmnt}/media/drive"
rm -d /run/rootfs/drive

That's it. The script will continue as usual but with the root filesystem being the overlay, and all its parts accessible afterwards easily. Note that it does no error checking, that's up to your own discretion to add for the commands above or to make sure the overlay module is loaded.

Now just pack the initramfs:

sudo sh -c 'cd /tmp/initramfs && find . -print0 | cpio --null -ov --format=newc' | gzip -9 > /tmp/initrd.img

And copy /tmp/initrd.img to your SD card or wherever. Don't forget to place root.squashfs at the VFAT partition's root directory, though that's obviously easily customizable and you don't have to do it this way. This is just the "easiest" way with syslinux (and even UEFI boot), by no means the best.

And sorry, I know you asked for embedded device but I don't know how the boot process works there, I'm just using an example for x86, but that part is less important (only the syslinux.cfg portion).

Please note that this will make the writeable part of the overlayfs as tmpfs, which is not what you wanted. However it is easily changed, if you look at above, just mount something else for /run/rootfs instead of tmpfs, wherever you want the writes to go to.

Answer 2

One solution could be to use an initramfs to mount the rootfs with overlayfs.

You can have a look how it is done on the Raspberry Pi

or, if not using an ARM device, at the overlayroot Ubuntu package

Answer 3

Overlay read only 'protection' for directories

I could not make it work for my entire root filesystem or using /boot/inird solution. But I could make it work for specific folders.

The code bellow must run once your board/system is ready for 'production'. You can adapt it to suit your needs. My case was a SOC board running a Network Video Recorder (NVR).

This code can be used to 'protect' any directory on your system. Directory you know has lots of I/O. And you don't care with what get's written on it like logs etc..

Example 'protecting' /var folder

I use /dev/shm ramdisk folder, if your distribution don't have it you must mount a RAM disk yourself (tmpfs etc.).

# create folders needed by overlay filesystem on /dev/shm (RAM)
mkdir -p /dev/shm/var_upper /dev/shm/var_workdir /dev/shm/var_overlay
mkdir -p /var_
# since /var will be 'hidden' by overlay mouting 'over' it
# we need it 'visible' somewhere to be usable as lowerdir 
# mount --bind does that to /var_
mount --bind /var /var_
sudo mount -t overlay overlay -o lowerdir=/var_,upperdir=/dev/shm/var_upper,workdir=/dev/shm/var_workdir /var

Some explanation

Overlay mounts a union filesystem (or directory tree) where upper directory is protected. This example makes /var read only. Write is done only on RAM /dev/shm hence logs etc. are lost on reboot. It is safe for reboot since unbind will be done automatically and everything will go back to its default place. Can be replicated to any other folder that needs read-only 'protection'.

My original search was for a solution to prevent sd card corruption on an event of power loss on my SOC board.