1

Synopsis

I'm serving a NFS share from my local machine and, while mounting and accessing the share works as expected, when I attempt to reference the NFS as a lower directory in overlayfs, operations requiring exec/read/write permission from a subdirectory fail with Operation not supported

Setup

Here's how I setup the mounts locally

sudo mkdir -p tmpfs_dir
sudo mount -t tmpfs -o size=4096m tmpfs tmpfs_dir

mkdir -p tmpfs_dir/{lower,upper,work}
sudo mount -t nfs -o ro,nolock,noacl,vers=3,proto=tcp 127.0.0.1:/nfsshare tmpfs_dir/lower

mkdir -p overlay_dir
sudo mount -t overlay -o lowerdir=tmpfs_dir/lower,upperdir=tmpfs_dir/upper,workdir=tmpfs_dir/work none overlay_dir

Calling ls on the root of overlayfs - Good

ls -lah overlay_dir
total 108K
drwxr-xr-x  1 myuser   myuser   40 Feb 12 00:31 .
drwxr-xr-x  7 myuser   myuser   4.0K Feb 11 17:19 ..
drwxr-xr-x  2 root     root     4.0K Aug 20 16:07 bin
drwxr-xr-x  4 root     root     4.0K Aug 16 14:36 boot
drwxr-xr-x 20 root     root     4.0K Aug 20 15:57 dev
drwxr-xr-x 98 root     root     4.0K Sep 10 18:13 etc
drwxr-xr-x  3 root     root     4.0K Aug  7  2019 home
...

Calling ls on a subdirectory of overlayfs - Breaks

ls -lah overlay_dir/bin
ls: cannot open directory 'overlay_dir/bin': Operation not supported

Calling ls on the root of NFS - Good

ls -lah tmpfs_dir/lower
total 108K
drwxr-xr-x  1 myuser   myuser   40 Feb 12 00:31 .
drwxr-xr-x  7 myuser   myuser   4.0K Feb 11 17:19 ..
drwxr-xr-x  2 root     root     4.0K Aug 20 16:07 bin
drwxr-xr-x  4 root     root     4.0K Aug 16 14:36 boot
drwxr-xr-x 20 root     root     4.0K Aug 20 15:57 dev
drwxr-xr-x 98 root     root     4.0K Sep 10 18:13 etc
drwxr-xr-x  3 root     root     4.0K Aug  7  2019 home
...

Calling ls on a subdirectory of NFS - Good

ls -lah tmpfs_dir/lower/bin
total 16M
drwxr-xr-x  2 root root 4.0K Aug 20 16:07 .
drwxr-xr-x 26 root root 4.0K Feb  7 14:57 ..
-rwxr-xr-x  1 root root 1.1M Jun  6  2019 bash
-rwxr-xr-x  1 root root 700K Mar 12  2018 btrfs
lrwxrwxrwx  1 root root    5 Mar 12  2018 btrfsck -> btrfs
-rwxr-xr-x  1 root root 368K Mar 12  2018 btrfs-debug-tree
-rwxr-xr-x  1 root root 364K Mar 12  2018 btrfs-find-root
-rwxr-xr-x  1 root root 388K Mar 12  2018 btrfs-image
...

Calling an executable from a subdirectory of overlayfs - Breaks

overlay_dir/bin/bash
bash: overlay_dir/bin/bash: Operation not supported

It appears accessing anything below the root directory from with overlayfs will show the Operation not supported error. I have discovered, however, that recursively calling chown fixes the files in the overlay share, but (i) it seems like an unnecessary hack to retouch permissions post mount and (ii) it's a little cumbersome detecting the owner on the file as failing to do so (e.g. blindly running sudo chown on every subdirectory) will throw an error on instances where the owner is different

sudo chown -R root bin

ls -lah bin/bash
-rwxr-xr-x 1 root root 1.1M Jun  6  2019 bin/bash

overlay_dir$ bin/bash -c "echo 'why does it work now?'"
why does it work now?

tl;dr

How do I properly mount NFS from within overlayfs, so I don't have to retouch permissions post-mount?

Specs

  • Distribution Ubuntu 18.04
  • Kernel 5.3.0-28-generic
  • mount version libmount 2.31.1
  • NFS server serving v4 and v3

Further considerations

  • Read-only nfsroot with NFS v4 and overlayfs is the closest I could find to a similar issue
  • My current research has led me to believe serving NFS over an older linux kernel might work ~4.15, so I'll see if that helps.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.