I'm trying to build a portable cross-arch Docker container using QEMU. However, depending on the host system and the programs installed on the host the container behaves differently.

Running the container on Windows, the emulation works fine. It also works fine on Ubuntu 16.04 when qemu-user-static is installed. But it will stop emulating if it isn't.

Emulation works as long as binfmt-support is enabled in the Host kernel (for Docker on Linux) or in the VM's kernel (for Docker on Windows) AND the necessary binaries are avaiblable in the containers filesystem.

My goal is to make the container portable and runnable without the need of registering qemu as an interpreter in the kernel since any modifications to the Host-OS itself or its kernel are forbidden by our customer. But currently, that is the result of running the container:

$ sudo docker run -ti qemu:xenial_arm64
qemu-aarch64-static: /usr/bin/groups: cannot execute binary file: Exec format error
/usr/bin/lesspipe: line 28: /usr/bin/basename: cannot execute binary file: Exec format error
/usr/bin/lesspipe: line 282: /usr/bin/dirname: cannot execute binary file: Exec format error
/usr/bin/lesspipe: line 295: [: =: unary operator expected
qemu-aarch64-static: /usr/bin/dircolors: cannot execute binary file: Exec format error
root@41f2795e2569:/# uname -m
qemu-aarch64-static: /bin/uname: cannot execute binary file: Exec format error

However if I install qemu-user-static (qemu will also be enabled in binfmt_misc by installing it):

$ sudo apt-get -yqq install qemu-user-static
Selecting previously unselected package qemu-user-static.
(Reading database ... 245713 files and directories currently installed.)
Preparing to unpack .../qemu-user-static_1%3a2.5+dfsg-5ubuntu10.42_amd64.deb ...
Unpacking qemu-user-static (1:2.5+dfsg-5ubuntu10.42) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up qemu-user-static (1:2.5+dfsg-5ubuntu10.42) ...

$ sudo docker run -ti qemu:xenial_arm64
root@7903b45a3c1e:/# uname -m

My explanation for that is:

  • qemu-aarch64-static starts emulating /bin/bash
  • /bin/bash uses execve(..) to call some other executables
  • Linux Host:
    • if binfmt_misc is enabled either by the host or by the container (on Linux, the host and the Docker guest share a kernel), qemu will be dynamically assigned as an interpreter for the arm64 executables
    • else, executation will fail and "Exec format error" will be thrown
  • Windows Host:
    • since Windows and Linux have different kernels, Linux guests cannot be executed by passing syscalls onto the host kernel
    • so on Windows, containers run inside of a VM with a separate kernel; that kernel is then shared with the containers
    • if qemu-user-static with binfmt-support was once installed and enabled in any container, it will still serve its purpose and dynamically assign qemu as interpreter for executables of another architecture

Execution on Linux and on Windows yields equivalent results.

A possible fix would be to launch the container using the --privileged flag. Then I would be able to mount binfmt_misc and thereby register qemu-user-static as an interpreter inside the container. But our customer forbids to use that flag.

I've already tested the approach described here as well as qemu-user's -0 flag. Neither one does work.


FROM scratch
ADD xenial-arm64-rootfs /
ADD qemu-aarch64-static /usr/bin/qemu-aarch64-static
RUN chmod +x /usr/bin/qemu-aarch64-static
ENTRYPOINT ["/usr/bin/qemu-aarch64-static", "-0", "/usr/bin/qemu-aarch64-static"]
CMD ["/bin/bash"]

qemu-aarch64-static is taken from /usr/bin/qemu-aarch64-static (installed with qemu-user-static) xenial-arm64-rootfs is created using:

qemu-debootstrap --arch=arm64 --components=main,universe,multiverse,restricted --variant=buildd --foreign xenial xenial-arm64-rootfs http://ports.ubuntu.com/ubuntu-ports/

What can I do in order to keep the emulation going?

  • 1
    can you add the dockerfile or a minimal version that was use to build qemu:xenial_arm64 . – EchoMike444 Nov 28 '19 at 18:47
  • I realized that the execution on Windows and on Linux yields the same results. That is because Docker on Windows runs containers inside of a Linux-VM and thus, containers running on Windows share the same kernel as containers running on Linux do. Got that when I installed qemu-user-static in a container running on Windows and apt-get output contained "update-binfmts: warning: qemu-aarch64 already enabled in kernel." even though it was an unedited ubuntu:bionic image. – Areopag Dec 10 '19 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.