Docker error : permission denied 'Hello World'

Question

Today I am trying to set up postgresql with Docker. However I have an error when I test with the 'hello-world' :

docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\"proc\\" to rootfs \\"/var/lib/docker/overlay2/1ef83761740dcfec79baa94e1ef3073bf1fa130a21f56e9201a80f4523e073ab/merged\\" at \\"/proc\\" caused \\"permission denied\\"\"": unknown. ERRO[0003] error waiting for container: context canceled

I learned about this problem, which happens a lot, however I found no solution (I tested to set docker groupe, launch with sudo, etc...)

This is 'docker info':

Client:
 Debug Mode: false

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 1
 Server Version: 19.03.10
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.3.18-3-pve
 Operating System: Debian GNU/Linux 9 (stretch)
 OSType: linux
 Architecture: x86_64
 CPUs: 3
 Total Memory: 7.938GiB
 Name: 18507
 ID: 6GVD:Z44L:34QD:NS5H:UXZM:IARY:PI66:3NZN:XRWH:WP7C:KEJM:7KDF
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Do you have any questions or suggestions ?

Answer 1

The command virt-what you ran and which outputs lxc tells your whole system is already running in an unprivileged LXC container (itself probably provisionned by Proxmox' own API/method), as determined by seing cat /proc/1/uid_map didn't output 0 0 4294967295 but 0 100000 65536.

LXC puts restrictions to containers it's running. Such containers don't have standard access to containerization features themselves without special settings in place on the host (providing the LXC container, owned by your VPS provider) and sometimes also the containerization tool to work along.

I found this discussed in an LXC forum, and it appears to match exactly your problem:
Working install of Docker-CE in LXC unprivileged container in Proxmox

Now the docker daemon should be OK ; it’s time for the second error: docker run hello-world returns an error « mounting proc to rootfs…permission denied »

As expected, the solution is to enable nested containerization on LXC/Proxmox. This must be done by your VPS provider (or some API it would provide to its users).

Fix it (trick #2) by inserting manually a line containing the following :

features:  keyctl=1,nesting=1

when looking at Proxmox' documentation:

keyctl=<boolean> (default = 0)

For unprivileged containers only: Allow the use of the keyctl() system call. This is required to use docker inside a container.
[...]

and:

nesting=<boolean> (default = 0)

Allow nesting. Best used with unprivileged containers with additional id mapping. Note that this will expose procfs and sysfs contents of the host to the guest.

Those two features must be activated for you by your provider. As there are security considerations, your provider might not be inclined to do it. So you should check your contract with your provider to see if you are supposed to get such restriction or not and act accordingly.

You won't get this problem in an environment where the result of virt-what gives a VM hypervisor rather than a container result (eg: kvm, vmware, hyperv, even qemu (which would run very slowly) ... but not docker or lxc), or else if the provider explicitly allows nesting by relaxing some restrictions.