0

I am trying to build a little project using Linux network namespaces, but am a bit overwhelmed by all the linux networking features and containerization-tech available, and thus unsure if i'm approaching this problem the right way.

THE PROBLEM / PROJECT

I currently have a simple Linux box with a single network device (eth0) which has a dozen static public IPv4 addresses assigned to it (1.1.1.1, 1.1.1.2, 1.1.1.3, ..., 1.1.1.12). I wish to create a situation where i have a bunch of (network) namespaces, each of those effectively having its own single exclusive public IPv4 addr/interface.

My goal is to spin up multiple shells, each isolated to their own network namespace (and also pid,ipc,... namespaces). So for example, the 7th shell would use network namespace ns7 which has a single (virtual) ethernet interface which has static ip 1.1.1.7. Within that shell i could (for example) start apache/nginx, let it listen on *:80 which would then serve websites to the public on 1.1.1.7:80. Any traffic directed at port 80 on any of the other IPv4 addresses would never reach ns7, and likewise any traffic directed at 1.1.1.7 should only reach processes in namespace ns7.

The basic idea is that the namespaces are effectively permanent. The namespaces themselves, and the related virtual network devices, will be (re)created and spun up when the system boots.

POTENTIAL SOLUTION (am i on the right track ?)

From what i have been able to piece together without hands-on experimentation, the solution should be something like outlined below, am i on the right track here ?

  1. Create and bring up a (virtual) L2 Bridge device br0. Making sure we operate at L2 (and not L3).

  2. Change the current ethernet device eth0 configuration, so that it still comes up on boot but doesn't set any IP address anymore (neither static nor DHCP), also assign eth0 to the br0 Bridge.

  3. Create some network namespaces ip netns add ns2, ip netns add ns3, ..., ip netns add ns12. I will treat the default/root namespace as if it is ns1.

  4. Create multiple virtual ethernet interface pairs net2a~net2b, net3a~net3b, net4a~net4b, ... For each pair, hook up the a-version to the br0 Bridge, and assign the b-version to it's respective namespace.

  5. Within each namespace, assign the local veth device (like net2b in ns2) the appropriate IPv4 address info and bring the device up.

  6. I perhaps may have to enable IPv4 forwarding (?) (/proc/sys/net/ipv4/ip_forward) and/or enable ARP filtering (?) (/proc/sys/net/ipv4/conf/all/arp_filter).

  7. Each namespace would have it's own firewall configuration (as i understand it), so within each namespace run some iptables/nftables shell script that sets some sane defaults and adjust them to local needs.

Does this sound like a (roughly) workable plan, to those more familiar with these virtual linux networking devices ?

EXTRA DETAILS (in case they matter)

  • OS Info: I run CentOS 8, which notably includes kernel 4.18, systemd and SELinux, only i use nftables for manual firewall configuration (instead of firewalld stuff).

  • Background: All the addresses provided (like 1.1.1.1) and interface names (like eth0) and such were all hopefully obviously fictional, to some degree for privacy reasons, but more so for brevity/simplicity sake, the same goes for my stated intention of just "running a shell in each namespace".

  • Actual Requirements: There is a bunch of different kinds of software that will be running in these namespaced environments, each namespace will have a unique job and often involve multiple services; One of my main wishes is to completely isolate the IPv4 addresses from each other; Also, many of the target programs are servers/daemons (like my apache httpd example), i want them to be able to bind to the actual public-facing interface/port, instead of say binding to a port on a private ipv4 or unixsocket, and then having software in the root-namespace act as reverse proxy middleware;

  • Why not just docker: (TL/DR) It is an option, but i want to custom tailor something together for fun (Longer Rant) Almost all of these namespaced environments are for personal use, one will run my private mailserver, one webserver for hosting some personal low-traffic sites, one webserver for some live webdev work, a couple running their own sshd+webstack to act as a free mini-VPS for some hobbyist friends, some running some largely automated processes, that kind of stuff. I know there is a huge overlap between what i'm trying to do and what common containerization stacks like docker offer, infact most of the things i've described the system already does, much of it using Podman (near identical to docker) and much of the rest just running side by side in the common root namespace. For various reasons i like to cleanly micromanage some of it, which would be a lot easier if i can implement the separation of stuff into multiple permanent namespaces, i have found that the more i try to tune things to my wishes, the more the containersoftware gets in my way. And since the only things containersoftware provides, that i actually make use of, are actually linux kernel features, i felt it's a worthy project to ditch that wrapper and figure out how to accomplish these things without it. And i also enjoy the education value that comes with diving deeper in this stuff, as it's not something my day job usually touches on.

  • IPC: I have no significant reasons for the (processes inside the individual) namespaces to have to communicate over IP with the other namespaces (nor with the default namespace). Though, if i change my mind on that (and assuming my idea so far was mostly correct), i imagine i could just repeat part of the proces by setting up an additional L2 bridge with additional veth-pairs for each namespace, and assign those private 192.168.x.x style IPv4-addresses.

  • VPS / Cloud: The machine in question is not a physical machine but a VPS / virtual server / cloud server. The only/single network device that's currently present on the machine. This eth0, which in reality is called ens3, automatically functioned right away, even before completing the CentOS installation, and is thus already a virtual device itself. lsmod currently reveals veth and virtio_net are loaded. I think my hosting provider provides this VPS using Qemu. I'm not sure if any of this matters or not, i imagine it shouldn't. Though i did spend some time looking into if it was possible to just clone (or create more) interfaces like the current ens3 and then simply assigning them all a single IPv4 and namespace directly, eliminating the need for a bridge device and veth-pair. Nothing really came out of that search, i kind of just ended up assuming this wouldn't be possible without help from the hosting providers personnel changing settings at the hypervisor level. And while they are generally helpful, and i imagine potentially open to such actions, it would make my solution less flexible to future changes, so if possible i'd prefer to handle this on the device that i can fully control.

  • IPv6: I have avoided mentioning IPv6 for brevity, i do have a block of IPv6 addresses too and plan to use those in a similar fashion. But i think it's likely best to get IPv4-only working correctly first, then enable IPv6 and go from there, i can't imagine it'll be too different.

  • My implementation: I'm not entirely sure yet on how to implement it all, once i do actually manage to get things working for the first time. I imagine that for the network part of the project, i'll create a network-setup.sh shell script that tests for the presence of all the network-related neccecities (like the namespaces, bridge device, veth devices, ...) and then recreates or sets up whatever is missing. Then accompany it with a systemd unit file that runs that shell script, and flag that unit file as required by network-online.target. And then another shell script and unit file that run later in the (boot)process, that uses unshare or the systemd-version of init to actually start the relevant processes. But if anyone has a better idea i'd love to hear it.

  • Different namespace implementations: One potential complication i can smell coming is that i am under the impression that what util-linux (man unshare) calls a network namespace is not the same as what iproute2 (man ip-netns) means by that same term. I'm however still not sure if the former is a superset/extension over the latter, or if it is an entirely different incompatible implementation. Infact this seems to be a recurring problem when reading up on container related tech.

1

The approach with a bridge a veth-pairs will work, but there's a simpler one:

Use a macvlan, see e.g. here or here for some details and discussion.

That is a virtual interface that uses the physical interface (in your case, eth0) as parent (or "master"), is completely transparent to other devices that use the same parent, and can be moved into a network namespace.

You then can assign IPv4 or IPv6 addresses inside the namespace to this interface, and it will work like you had an additional physical network interface that's exclusive to that container.

There are different flavours depending on whether you want your containers to talk to each other or not. Read the documention for details.

And yes, if you want a firewall (iptables), you'll have to do that in each namespace as well.

Incidentally, Docker and other virtualization approaches that use namespaces also use macvlans, so if all you want is apache, nginx and so on, consider using Docker and/or Docker Compose, and it will do all the rest of the work (different filesystems, local DNS, port mapping).

2
  • Interesting. I did know about the macvlan/ipvlan family of devices (indeed from working backwards away from docker), but your description of it makes it sound a lot more suiting than how i initially walked away understanding it. I'll dive into it, thanks ! – Raxi Jun 29 '20 at 9:38
  • 1
    Yes, the use of macvlan worked nicely ! severely cuts down the amount of virtual devices to manage (as the alternative required both a bridge and veth devices which come in pairs). – Raxi Jul 3 '20 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.