OverlayFS Seamlessly Edit File in Lower Directory

Question

I'm trying to set up a PXEboot environment in which the base system (served over NFS to the PXE clients) is read only, and the root filesystem is an overlayfs filesystem with the read-only NFS base system as the lowerdir and a tmpfs as the upper/work dir's.

I edited an AuFS initramfs script to use OverlayFS, and it's working well, except that when you try to edit a file that's in the lowerdir (say, for example, /etc/environment), it is opened as read-only, which is not the case for new files (say, for example, /etc/foobar) or files that have already been copied up to the upper directory. The problem can be mitigated by simply doing a touch before attempting to edit the file, but, it's less than ideal and is likely to break other applications.

AuFS didn't have this issue. Any advice? Here's the relevant part of the initramfs script (in /etc/initramfs-tools/scripts/init_bottom/00_overlayfs_init), edited for brevity.

mkdir /overlay
mkdir /local
mkdir /remote



# mount the temp file system and move real root out of the way
mount -t tmpfs none /local
mount --move ${rootmnt} /remote

mkdir /local/rw
mkdir /local/work

mount -t overlayfs -o lowerdir=/remote,upperdir=/local/rw,workdir=/local/work overlay /overlay

#test for mount points on overlay file system
[  -d /overlay/ro ] || mkdir /overlay/remote
[  -d /overlay/rw ] || mkdir /overlay/local

mount --move /remote /overlay/remote
mount --move /local /overlay/local

mount --move /overlay ${rootmnt}

edit: more info

Trying to edit (with an editor, e.g. Vim) results in vim stating that the file is read only, and on :wq!, E166: Can't open linked file for writing.

root@dark-node:~# echo FOO=bar >> /etc/environment 
-bash: /etc/environment: Permission denied
root@dark-node:~# echo FOO=bar > /etc/environment 
-bash: /etc/environment: Permission denied
root@dark-node:~# touch /etc/environment
root@dark-node:~# echo FOO=bar >> /etc/environment 
root@dark-node:~# cat /etc/environment 
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/game::/usr/local/games"
FOO=bar
root@dark-node:~# uname -a
Linux dark-node 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Answer 1

I ran into the exact same problem (trying to do the exact same thing: a read-only base system for a PXE boot environment).

The behavior I'm seeing exactly matches that in this Ubuntu bug report. I cannot write changes to existing files, but I can delete them (but they're preserved in the lower filesystem... with the upper layer just recording their absence) and then write them back again (at which point the upper layer has a copy which I can edit).

From what I can dig up, this is happening because OverlayFS and NFS don't play nice together. The thing that we're expecting to happen, when we try to modify a file from the lower filesystem, is called "copy up", and it breaks when OverlayFS tries to use a NFS filesystem because NFS doesn't support xattrs.

Thus far, the only promising avenue I can find is that it's possible to use fuse_xattrs to emulate xattrs on your NFS mount (described here), but it requires that you have two mounts of your NFS share: the "real" one, and the "xattr-enhanced" one that needs the first.