Layered or Virtual filesystem on Linux

Question

I basically would like to do something like this: Let's imagine a utility called layer.

# sudo layer somefile apt install gcc

layer would launch apt in some sort of container. Apt would think it installs gcc on the actual filesystem but actually it is writing everything into somefile. apt still has access to the whole filesystem. So it's not like docker.

Then, I could do something like

# sudo layer somefile bash
bash# gcc -c foo.c

And bash would see gcc as installed by apt even though gcc was not actually installed on the system but was installed in somefile. Some form of container/jail but which would let access to the rest of the filesystem. My purpose is to create virtual environment for development without polluting my filesystem.

Is there a way to do this?

Answer 1

If you don't mind some custom scripting, you can use OverlayFS (included in the Linux kernel since 2014).

OverlayFS layers two filesystem locations on top of each other: the overlay mountpoint shows everything from lower, unless it has been modified or shadowed by something in upper. All modifications done to the overlay mountpoint are recorded in upper; lower is never touched.
Additionally, OverlayFS requires a work directory that isn't clearly documented, but apparently serves as a temporary area before modifications are moved to upper.

Using this, you can create the kind of container you are looking for (see below for possible edge cases):

mkdir -p /var/tmp/myoverlay/{upper,work,mount}
mount -t overlay -o lowerdir=/,upperdir=/var/tmp/myoverlay/upper,workdir=/var/tmp/myoverlay/work overlayfs /var/tmp/myoverlay/mount

To "enter" the container, you can use the chroot ("change root directory") command to run a command (shell or other) inside your newly created /var/tmp/myoverlay/mount:

chroot /var/tmp/myoverlay/mount
# or
chroot /var/tmp/myoverlay/mount /usr/bin/apt moo # paths are relative to the new root directory

Note that this might not be enough for programs that try to access hardware devices, pseudo-terminals, processes, or system functions: those are provided by the kernel as special mountable filesystems — see the output from mount below:

proc on /proc type proc
sysfs on /sys type sysfs
udev on /dev type devtmpfs
devpts on /dev/pts type devpts

You can mount these into the overlay mountpoint (before entering it):

mount -t proc proc /var/tmp/myoverlay/mount/proc
mount -t sysfs sys /var/tmp/myoverlay/mount/sys
mount -t devtmpfs dev /var/tmp/myoverlay/mount/dev
mount -t devpts devpts /var/tmp/myoverlay/mount/dev/pts

Note that this will give your container more access to the system than you might like. Ultimately, you need to know your exact use case to decide whether an overlay filesystem is sufficiently separated from your normal system for that particular use case.

---

There can be an additional difficulty, depending on your filesystem layout: the lower directory is taken as is, without following any further filesystems mounted below it. If, for example, your /home is a separate filesystem, the OverlayFS will only show you the empty mountpoint at /home.

In those cases, you'll need to create separate overlays for each of these mountpoints, and then mount the additional overlays into the root overlay.
At that point, you're approaching the scenario where proper virtualisation is usually a more sensible choice.

Answer 2

You can use the overlay file system.

modprobe overlay
mkdir -p /path/data /path/work /path/mount
mount -t overlay overlay -o lowerdir=/,upperdir=/path/data,workdir=/path/work /path/mount

Now you can chroot to /path/mount and see everything from your root directory, but changes will be written to /path/data. You can unmount /path/mount and remount it later with the same options.