2

I've got sort of a niche use case (not "production", it's a lab environment for a classroom) where I am using LXC containers on a Proxmox VE machine to run GNS3 Server and KVM virtualization.

To make a long story short, I need to run KVM inside an LXC container, which means I have to pass through the host machine's CPU virtualization features to the container. I already have this working based on some tutorials I read, but I don't understand why it works. I have no idea what these config file lines actually do, I just blindly copy-pasted.

Here is my container config (from /etc/pve/lxc/102.conf):


arch: amd64
cores: 2
hostname: ubuntu-gns3vm2-clone
memory: 4096
net0: name=eth0,bridge=vmbr0,hwaddr=86:4B:53:B7:66:0A,ip=dhcp,type=veth
ostype: ubuntu
rootfs: local-lvm:base-102-disk-0,size=16G
swap: 2048
template: 1

# LINES IN QUESTION
# This line seems to pass through the host CPU's virtualization features
lxc.cgroup.devices.allow: c 10:232 rwm

# These lines were needed for virtual machine networking to behave, I don't really understand them either
lxc.cgroup.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

Can someone decipher these for me? Where do the 10:232 and 10:200 numbers come from? What is a cgroup? What does /dev/net/tun actually do?

Like I said, nothing is misbehaving, it's all working fine, I just don't want to deploy something I don't fully understand. Feel free to ELI5.

0

As far as I known, there's no hardware passthrough in containers: USB devices, PCI devices, KVM, etc. are always available as long as permissions to use them are not removed by various isolation mechanisms used in containers, or perhaps unless their driver/layer include namespace awareness allowing them to change behaviour depending on namespace. Here's a non exhaustive list of such isolation mechanisms:

  • capabilities . Eg in /usr/share/lxc/config/common.conf:

    # Drop some harmful capabilities
    lxc.cap.drop = mac_admin mac_override sys_time sys_module sys_rawio
    
  • cgroups. Eg in /usr/share/lxc/config/common.conf:

    # CGroup whitelist
    lxc.cgroup.devices.deny = a
    ## Allow any mknod (but not reading/writing the node)
    lxc.cgroup.devices.allow = c *:* m
    lxc.cgroup.devices.allow = b *:* m
    ## Allow specific devices
    ### /dev/null
    lxc.cgroup.devices.allow = c 1:3 rwm
    

    [...]

  • seccomp. Eg in /usr/share/lxc/config/common.seccomp:

    kexec_load errno 1
    open_by_handle_at errno 1
    init_module errno 1
    finit_module errno 1
    delete_module errno 1
    

These are the main machanisms, there are others (eg: pivot_root/mount when the container is started, or mounting fake entries over some portions of /proc/ or /sys/ to prevent access).

Mimic-ing a passthrough can be done in multiple ways depending on the kind of ressource. Eg: most network interfaces (not their underlying hardware (or virtual...) network device) have direct support for namespaces and can be moved around namespaces without trouble. It's a base concept for containers. It's not the same as passing through the whole device like is done with a VM.

Now if it's an USB serial device exposing for example /dev/ttyUSB0, there's no such namespace support. This requires copying (using mknod) the relevant /dev/ entry, which usually has a dynamic device node value which won't help making a simple configuration, and adding various permissions that were removed by default (see cgroups above) to restore access to this device. Again, there was no device passthrough involved.


So, for your KVM case. Access was first removed (that's the default lxc.cgroup.devices.deny = a), and you add it back (lxc.cgroup.devices.allow: c 10:232 rwm). c 10:232 describes a device node of type character with char major 10 and minor 232: 232 = /dev/kvm Kernel-based virtual machine (hardware virtualization extensions), that's the reserved tuple for KVM.

QEMU/KVM is creating and using TUN/TAP devices (instead of veth interfaces used with containers) to have network connectivity, so permission to use them by QEMU/KVM should be added back, that's what are the lines related to tun below (the Linux documented entry for c 10:200 is 200 = /dev/net/tun TAP/TUN network device). Also, as mount is forbidden inside the container, LXC pre-mount the original /dev/net for it to be able to use the original /dev/net/tun. I think an other method would have been, in the container, to create the directory /dev/net and create the same node (c 10:200) /dev/net/tun.

If you have to add more devices, you can first do some tests with lxc-attach -e which bypasses all limitations (eg: access to a device, using mount or rmmod all work) and then see what permissions should be added back in the configuration.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.