Why systemd-nspawn is slower than docker,podman and qemu?! how to Improve nspawn performance?

Question

Why nspawn is slow compared to docker podman and even qemu?! CPU tasks take twice of the time it takes in docker, podman or qemu

Here is a benchmark test I did:

First I disabled all the spectre/meltdown mitigations in the host kernel (and the qemu guest kernel in the case of qemu benchmark) using:

GRUB_CMDLINE_LINUX_DEFAULT=noibrs noibpb nopti nospectre_v2 nospectre_v1 l1tf=off nospec_store_bypass_disable no_stf_barrier mds=off tsx=on tsx_async_abort=off mitigations=off spectre_v2_user=off spec_store_bypass_disable=off nx_huge_pages=off kvm.nx_huge_pages=off kvm-intel.vmentry_l1d_flush=never srbds=off

then I used this benchmark test:

git clone https://github.com/tsuna/contextswitch
cd contextswitch
time make

I tested nspawn with super full privileges:

export SYSTEMD_NSPAWN_USE_CGNS=0
systemd-nspawn  --keep-unit --register=no --boot --capability=all --private-users=false --system-call-filter="@default @aio @basic-io @chown @clock @cpu-emulation @debug @file-system @io-event @ipc @keyring @memlock @module @mount @network-io @obsolete @privileged @process @raw-io @reboot @resources @setuid @signal @swap @sync @system-service @timer" --bind=/sys/fs/cgroup  --machine=testtt -D busterdir

I tested podman with privileges too:

podman run --rm -it --privileged debian:10 bash

I tested docker with privileges too:

docker run --rm -it  --privileged  debian:10 bash

I tested qemu with:

qemu-system-x86_64 -name buster20210121210102 -m 2G -enable-kvm -cpu host -smp cores=4,threads=2,sockets=1 -object iothread,id=myio1 -device virtio-blk-pci,drive=mydisk0,iothread=myio1 -drive file=buster20210121210102.qcow2,if=none,id=mydisk0,format=qcow2,aio=native,cache=none

and here are the results:

# baremetal
real    0m12.998s

# nspawn
real    0m30.777s  <==== :(

# docker
real    0m15.127s

#podman
real    0m15.207s

# qemu without mitigations
real    0m15.979s

here I filled a request to improve nspawn performance which contain the full test result: https://github.com/systemd/systemd/issues/18370

Do you know why systemd-nspawn is slower? how can I improve it?

Answer 1

The problem of performance was because I thought that whitelisting the syscalls en nspawn with --system-call-filter will improve the performance, but as they explained me in systemd mail list I should use export SYSTEMD_SECCOMP=0, because nspawn will still be processing syscalls when I whitelist them.

SYSTEMD_SECCOMP was added in systemd v247 (debian buster have v241 but backport repository have v247).

so to make nspawn as quick as the baremetal host do:

export SYSTEMD_SECCOMP=0
systemd-nspawn --capability=all -D ./bbusterboot --boot

this is equivalent to --privileged in docker/podman, and there is no need to use --system-call-filter if we use SYSTEMD_SECCOMP.

of course this is not good for security, so do it in a safe environment when running a trusted code only.

and if you want max performance which will add performance to baremetal, nspawn, docker, podman or what ever you are using, then disable all the spectre/meltdown mitigations as I did in the question above (but this is not good for security too if you run untrusted code like browsers with ad's for example).

---

read this for more details: https://github.com/systemd/systemd/issues/18370