I'm trying to build a portable cross-arch Docker container using QEMU. However, depending on the host system and the programs installed on the host the container behaves differently.
Running the container on Windows, the emulation works fine. It also works fine on Ubuntu 16.04 when qemu-user-static is installed. But it will stop emulating if it isn't.
Emulation works as long as binfmt-support is enabled in the Host kernel (for Docker on Linux) or in the VM's kernel (for Docker on Windows) AND the necessary binaries are avaiblable in the containers filesystem.
My goal is to make the container portable and runnable without the need of registering qemu as an interpreter in the kernel since any modifications to the Host-OS itself or its kernel are forbidden by our customer. But currently, that is the result of running the container:
$ sudo docker run -ti qemu:xenial_arm64 qemu-aarch64-static: /usr/bin/groups: cannot execute binary file: Exec format error /usr/bin/lesspipe: line 28: /usr/bin/basename: cannot execute binary file: Exec format error /usr/bin/lesspipe: line 282: /usr/bin/dirname: cannot execute binary file: Exec format error /usr/bin/lesspipe: line 295: [: =: unary operator expected qemu-aarch64-static: /usr/bin/dircolors: cannot execute binary file: Exec format error root@41f2795e2569:/# uname -m qemu-aarch64-static: /bin/uname: cannot execute binary file: Exec format error
However if I install qemu-user-static (qemu will also be enabled in binfmt_misc by installing it):
$ sudo apt-get -yqq install qemu-user-static Selecting previously unselected package qemu-user-static. (Reading database ... 245713 files and directories currently installed.) Preparing to unpack .../qemu-user-static_1%3a2.5+dfsg-5ubuntu10.42_amd64.deb ... Unpacking qemu-user-static (1:2.5+dfsg-5ubuntu10.42) ... Processing triggers for man-db (2.7.5-1) ... Setting up qemu-user-static (1:2.5+dfsg-5ubuntu10.42) ... $ sudo docker run -ti qemu:xenial_arm64 root@7903b45a3c1e:/# uname -m aarch64
My explanation for that is:
- qemu-aarch64-static starts emulating /bin/bash
- /bin/bash uses execve(..) to call some other executables
- Linux Host:
- if binfmt_misc is enabled either by the host or by the container (on Linux, the host and the Docker guest share a kernel), qemu will be dynamically assigned as an interpreter for the arm64 executables
- else, executation will fail and "Exec format error" will be thrown
- Windows Host:
- since Windows and Linux have different kernels, Linux guests cannot be executed by passing syscalls onto the host kernel
- so on Windows, containers run inside of a VM with a separate kernel; that kernel is then shared with the containers
- if qemu-user-static with binfmt-support was once installed and enabled in any container, it will still serve its purpose and dynamically assign qemu as interpreter for executables of another architecture
Execution on Linux and on Windows yields equivalent results.
A possible fix would be to launch the container using the --privileged flag. Then I would be able to mount binfmt_misc and thereby register qemu-user-static as an interpreter inside the container. But our customer forbids to use that flag.
I've already tested the approach described here as well as qemu-user's -0 flag. Neither one does work.
FROM scratch ADD xenial-arm64-rootfs / ADD qemu-aarch64-static /usr/bin/qemu-aarch64-static RUN chmod +x /usr/bin/qemu-aarch64-static ENTRYPOINT ["/usr/bin/qemu-aarch64-static", "-0", "/usr/bin/qemu-aarch64-static"] CMD ["/bin/bash"]
qemu-aarch64-static is taken from /usr/bin/qemu-aarch64-static (installed with qemu-user-static) xenial-arm64-rootfs is created using:
qemu-debootstrap --arch=arm64 --components=main,universe,multiverse,restricted --variant=buildd --foreign xenial xenial-arm64-rootfs http://ports.ubuntu.com/ubuntu-ports/
What can I do in order to keep the emulation going?