OverlayFS doesn't work with unprivileged user namespace

Question

OverlayFS mount works weird when accessed from within the unprivileged user namespace. Best to be explained in example:

~# uname -a
Linux host 4.1.0-1-amd64 #1 SMP Debian 4.1.3-1 (2015-08-03) x86_64 GNU/Linux
~# runuser - test -c id
uid=2000(test) gid=2000(test) groups=2000(test)
~# cat /etc/subuid /etc/subgid | grep test
test:200000:65536
test:200000:65536
~# cd ~test
/home/test# mkdir -p upper/test1 lower/test2 target target.work
/home/test# chown -R test:test upper lower target target.work
/home/test# mount -t overlay -o lowerdir=lower,upperdir=upper,workdir=target.work overlay target
/home/test# mount | grep test
overlay on /home/test/target type overlay (rw,relatime,lowerdir=lower,upperdir=upper,workdir=target.work)

Overlay is mounted and works as expected:

/home/test# runuser - test
~$ cd target
~/target$ ls -l
total 8
drwxr-xr-x 2 test test 4096 Sep 15 13:50 test1
drwxr-xr-x 2 test test 4096 Sep 15 13:50 test2
~/target$ mkdir test3
~/target$ mkdir test2/test2-3
~/target$ mkdir test1/test1-3

Lets try unprivileged user namespace now

~/target$ ^D
/home/test/target# cd ..
/home/test# umount target
/home/test# rm -rf upper lower target target.work
/home/test# mkdir -p upper/test1 lower/test2 target target.work
/home/test# chown -R 200000:200000 upper lower target target.work
/home/test# mount -t overlay -o lowerdir=lower,upperdir=upper,workdir=target.work overlay target
/home/test# mount | grep test
overlay on /home/test/target type overlay (rw,relatime,lowerdir=lower,upperdir=upper,workdir=target.work)

Make sure that unprivileged namespaces are allowed:

/home/test# sysctl -w kernel.unprivileged_userns_clone=1

Okay, lets try it:

/home/test# runuser - test
~$ lxc-usernsexec -m u:0:200000:65536 -m g:0:200000:65536 -m u:65536:2000:1 -m g:65536:2000:1 -- /bin/bash
~# cd target
~/target# ls -l
total 8
drwxr-xr-x 2 root root 4096 Sep 15 13:57 test1
drwxr-xr-x 2 root root 4096 Sep 15 13:57 test2

So far so good.

~/target# mkdir test3
~/target# mkdir test2/test2-3
~/target# mkdir test1/test1-3
mkdir: cannot create directory 'test1/test1-3': Permission denied

And this is where it gets broken. Aufs works fine in the same scenario (except for debian 4.1 kernel doesn't support aufs anymore).

Is there any way to make it working?

Answer 1

I think this is because target.work/work on mount belongs to root. Could you try to chown that dir after mount? However I found simpler reproduction case, we just need dir which is in lower, but not in upper:

# from user
mkdir -p upper lower/test2 target target.work
# from root
mount -t overlay -o lowerdir=lower,upperdir=upper,workdir=target.work overlay target
# from user again
unshare -Ur touch target/test2/1

If I insert chown user:user target.work/work after mount then all works okay. Not sure if we should think about this as overlayfs bug or feature :)

Answer 2

I don't know if you noticed, but your second mount is incorrect:

-m g:0:2000000:65536 should be -m g:0:200000:65536. Your command has an extra 0.

I followed your steps and I couldn't find any error:

root@clone-newuser:/home/test# mount -t overlay -o lowerdir=lower,upperdir=upper,workdir=target.work overlay target
root@clone-newuser:/home/test# runuser - test
test@clone-newuser:~$ lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -m u:65536:1000:1 -m g:65536:1000:1 -- /bin/bash
root@clone-newuser:~# ls
lower  target  target.work  upper
root@clone-newuser:~# cd target
root@clone-newuser:~/target# ls
test1  test2
root@clone-newuser:~/target# mkdir test3
root@clone-newuser:~/target# mkdir test2/test2-3
root@clone-newuser:~/target# mkdir test1/test1-3
root@clone-newuser:~/target#

I'm using a slightly newer kernel though:

root@clone-newuser:/home/test# uname -srm
Linux 4.1.7-040107-generic x86_64

I don't really know if this helps you.