2

I have two docker containers running on my machine where a very restrictive nftables configuration is active. I'd like to keep it that way but whitelist access to the docker containers from outside.

The containers open ports 80 and 6200. The docker service is started with iptables disabled.

Below is the current firewall configuration, including my attempt. icmp, ssh, http and https are already open. For docker, only the http port 80 and the application specific port 6200 are needed. I tried to allow access to docker only from 192.168.0.0/16 to be as restrictive as possible.

 table inet filter {
    chain input {
            type filter hook input priority 0; policy drop;
            iif lo accept
            iif eno2 icmp type echo-request accept
            iif eno2 ip 192.168.0.0/16 tcp dport 22 accept
            iif eno2 ip 192.168.0.0/16 tcp dport { http, https, 6200 } accept
    }

    chain forward {
            type filter hook forward priority 0; policy drop;
    }

    chain output {
            type filter hook output priority 0; policy drop;
    }
}

I tried adding additional rules for thr docker0 interface, but without any success. I suspect I have to modify chain forward?

4
  • Notwithstanding that it's in the wrong chain, do you also intent to further restrict Docker's containers reachability for their published ports 80 and 6200 to work only from 192.168.0.0/16 source instead of any source? If that's the intent you should state this intent in the question and not let people guess. Likewise, port 443 (or port 22) is not stated to be part of a Docker container, right?
    – A.B
    Jul 8, 2021 at 23:25
  • @A.B I clarified the question accordingly. I don't see how I'm further restricting access to the docker containers when there is no access at all to begin with. Could you please answer which chain is the correct one to edit, when you already mention that the input chain is the wrong one?
    – flowit
    Jul 9, 2021 at 16:23
  • The chain to edit is the forward chain of course. But really it's more complex than just that.
    – A.B
    Jul 9, 2021 at 16:35
  • Managed to overcome the "bridge path" problem too. I made an answer, and tried to keep it generic so it can help more than just for your specific case with two exposed containers (and wouldn't you be glad to know nothing more has to be done when adding a new container?).
    – A.B
    Jul 10, 2021 at 12:55

1 Answer 1

11

While the problem might appear simple, it's nothing but. Having Docker around always imposes several challenges to other parts in the system dealing with networking. Once nftables gets more widely adopted and gets directly used by Docker in the future, and especially once Docker stops using br_netfilter, things might become simpler.

If you think it's still worth using nftables along Docker, I present below a method intended to let Docker handle its part and not requiring having to duplicate Docker settings in other firewall rules whenever changes as simple as starting a new container with a new exposed port, are done.

Problems to be solved

iptables is still needed

Currently (2021) Docker still uses iptables and only iptables (It could also use firewalld but only with firewalld with an iptables backend. I'm not considering this case anyway). There's thus currently no way to have a pure nftables system when using Docker. The fact that iptables can be iptables-legacy or iptables-nft doesn't really matter.

Here are a few relevant excerpts from Docker and iptables that are useful for this case:

Docker installs two custom iptables chains named DOCKER-USER and DOCKER, and it ensures that incoming packets are always checked by these two chains first.

All of Docker’s iptables rules are added to the DOCKER chain. Do not manipulate this chain manually. If you need to add rules which load before Docker’s rules, add them to the DOCKER-USER chain. These rules are applied before any rules Docker creates automatically.

Nitpicking: actually Docker does -A DOCKER-USER -j RETURN so rules should be added in it before starting docker, or better: inserted which works in all cases.

Rules added to the FORWARD chain -- either manually, or by another iptables-based firewall -- are evaluated after these chains.

Docker also sets the policy for the FORWARD chain to DROP. If your Docker host also acts as a router, this will result in that router not forwarding any traffic anymore.

Docker enables IP forwarding but firewalls it for other uses than itself by default.

It is possible to set the iptables key to false in the Docker engine’s configuration file at /etc/docker/daemon.json, but this option is not appropriate for most users. It is not possible to completely prevent Docker from creating iptables rules, and creating them after-the-fact is extremely involved and beyond the scope of these instructions. Setting iptables to false will more than likely break container networking for the Docker engine.

Can't avoid having iptables.

br_netfilter

Moreover, Docker also loads the kernel module br_netfilter in order to have this property set:

# sysctl net.bridge.bridge-nf-call-iptables
net.bridge.bridge-nf-call-iptables = 1

So bridged frames (with here IPv4 type frames temporarily converted into IPv4 packets) are filtered by iptables and also by nftables (even if this is not clearly documented, nftables just like iptables hooks into Netfilter and Netfilter will call these hooks, be they from iptables or nftables).

This feature is the major point causing problems when interacting with Docker. Without knowledge about it, one would wonder why containers in the same internal bridged LAN can't communicate between themselves anymore, be they handled by Docker or something else (LXC, libvirt/QEMU...) running along Docker.

Here's the Packet flow in Netfilter and General Networking:

Packet flow in Netfilter and General Networking

A single chain from iptables or nftables in the ip/inet families can thus be traversed in two different ways: from usual routing path (green boxes inside green Network Layer field) but also from bridge path (green boxes inside blue Link Layer field). This documentation also tells:

A bridged packet never enters any network code above layer 1 (Link Layer). So, a bridged IP packet/frame will never enter the IP code.

So there's a guarantee that a packet won't traverse twice the same chain which is a relief.

Interactions between iptables and nftables

Since the goal is to use nftables one has to know how to use them together.

Here are Q/A having an answer of mine about this:

To summarize:

  • iptables and nftables can be used together
  • nftables can have its priority adjusted to have a deterministic order of evaluation between iptables and nftables (for this case: nftables after iptables)
  • a dropped packet stays definitively dropped whenever/wherever this happens
  • an accepted packet (which will be by iptables) continues evaluation in the next chain in the same hook (which will be nftables' chain).
  • packet marks can be used to convey messages between iptables and nftables

Method to solve this in a generic way

Dealing with bridge path

nftables rules in the ip/inet families should avoid doing anything in bridge path. Without Docker activating br_netfilter this would never even have to be considered. Detecting being in the bridge path from the ip/inet family should be left to iptables to avoid having nftables to deal with this and stay generic, with Docker installed or not. It's also easier to do this with iptables than with nftables from the ip/inet family because there's the specific iptables -m physdev --physdev-is-bridged test:

[!] --physdev-is-bridged

Matches if the packet is being bridged and therefore is not being routed. This is only useful in the FORWARD and POSTROUTING chains.

Note that this match depends on and loads br_netfilter if this wasn't done by Docker already: to work around issues caused by br_netfilter, br_netfilter is needed!

Using marks to link iptables and nftables

The idea is to use a mark to have messages from iptables passed to nftables, to differentiate cases:

  • rules evaluation is happening in the bridge path instead of the routing path

    Always accept such case.

  • packet was ACCEPT-ed by Docker

    Further restrictions can be added but mostly accept such case.

  • packet was ignored by Docker

    Use normal nftables rules that don't have to consider the presence of Docker.

  • packet was DROP-ed for any reason in iptables

    That's a moot case, nftables won't see this packet and nothing has to or can be done about it.

iptables

If done before Docker is started, create the filter chain DOCKER-USER:

iptables -N DOCKER-USER

If done after, Docker will have created it.

Add a rule to mark packets before Docker evaluation in the DOCKER chain overriden by the bridge path detection case with a different mark (inserting them here as explained before, but numbering them to preserve natural order, which does matter here):

iptables -I DOCKER-USER 1 -j MARK --set-mark 0xd0cca5e
iptables -I DOCKER-USER 2 -m physdev --physdev-is-bridged -j MARK --set-mark 0x10ca1

0x10ca1 and 0xd0cca5e are arbitrarily chosen values.

Append (before or after Docker was run, effect is the same since Docker always inserts its DOCKER chain before) a final rule that resets the packet's mark only if it was the tentative Docker evaluation mark, and add a final ACCEPT rule to override Docker's default DROP policy set on the FORWARD chain: the idea is to defer further evaluation to nftables for packets unrelated to Docker.

iptables -A FORWARD -m mark --mark 0xd0cca5e -j MARK --set-mark 0
iptables -A FORWARD -j ACCEPT

nftables

Change the inet filter forward priority value to a value slightly greater than NF_IP_PRI_FILTER (0), for example 10 to ensure nftables's forward chain happens after iptables filter/FORWARD in order to respect this chronology. The base chain line in OP's ruleset should be changed from:

    chain forward {
            type filter hook forward priority 0; policy drop;

to:

      chain forward {
              type filter hook forward priority 10; policy drop;

The 4 previous described cases can be detected in nftables by checking the mark on the packet. Adding counter expressions to help debug.

  • mark 0x10ca1 : bridge path

    Add bridge path pass-through rule:

    nft add rule inet filter forward meta mark 0x10ca1 counter accept
    
  • mark 0xd0cca5e: Docker case

    • create a regular/user chain to treat the Docker case and add a rule calling it:

      nft add chain inet filter dockercase
      nft add rule inet filter forward meta mark 0xd0cca5e counter jump dockercase
      
    • add additional restrictions about Docker, but accept by default

      For example to restrict incoming packets arriving from the eno2 interface to only be accepted if from private address within 192.168.0.0/16:

      nft add rule inet filter dockercase iif eno2 ip saddr != 192.168.0.0/16 counter drop
      nft add rule inet filter dockercase counter accept
      
  • no mark: general case not related to Docker

    Add anything that would be done without having to consider the presence of Docker, including nothing and having a default drop policy, else probably starting with the usual ct state related,established accept

  • (no packet: dropped in iptables, non-case)

example above becomes:

...
    chain forward {
        type filter hook forward priority 10; policy drop;
        meta mark 0x10ca1 counter accept
        meta mark 0xd0cca5e counter jump dockercase
    }

    chain dockercase {
        iif eno2 ip saddr != 192.168.0.0/16 counter drop
        counter accept
    }
...

Generic handling achieved

The ports 80 and 6200 don't have to appear in the nftables rules anymore. Should a new container that needs to expose new ports be added using Docker commands, nothing at all has to be done in nftables: it's already being taken care of thanks to the marks.

Adding more chains

Still because of br_netfilter's effects, should any other base nftables chain with the property hook forward or hook postrouting contain dropping rules or more usefully, altering rules (nat...) without using tricks described in previous link below figure 7b, then the same kind of arrangement has to be done:

  • its priority value should be above the iptables' equivalent chain priority

  • such iptables equivalent chain (except filter/FORWARD where it's already done in DOCKER-USER) should receive:

    iptables -t foo -I BAR -m physdev --physdev-is-bridged -j MARK --set-mark 0x10ca1

    with foo among raw, mangle, or nat and BAR among PREROUTING or POSTROUTING depending on the case

  • and the very first rule of the nftables chain should be again:

    meta mark 0x10ca1 accept
    
  • and if the chain's policy is again drop it should probably again include an user/regular chain jump from a rule using the 0xd0cca5e mark, as previously done.

For hook prerouting, documentation about --physdev-is-bridged tells this might not work in PREROUTING: don't ever use a default drop policy there. Anyway for hook prerouting cases there can't either be any 0xd0cca5e mark inherited from filter/FORWARD yet, but the same would be true using only iptables: PREROUTING can't foresee what happens later.

If you really want to do something at the bridge level, just use nftables in the bridge family, don't rely on this special case of the ip/inet family called from bridge path because of br_netfilter.

Caveat

Now marks are used to handle this, it becomes more difficult to use marks simultaneously for something else, but not impossible with some care. For example by using bitwise operations and masks with these marks. This is available in iptables and nftables. Even ip rule accepts a mask when using a mark as selector.


Important additional required adjustments

Docker adds nat rules to do port forwarding with iptables' DNAT target. In the end all exposed/published ports are routed to the containers instead of being received by the host. That means they will use as seen above iptables's filter/FORWARD chain as well as (with OP's ruleset) nftables inet filter forward chain and won't use INPUT / input.

There are also missing rules preventing correct connectivity for the host.

inet filter input

The input path won't be used at all for Docker's containers, except maybe for the docker-proxy case which usually is for local host's access but that OP already accepts with iif lo accept, so it doesn't have to be further handled in this answer. Anything about Docker shouldn't be present here: references to container's ports 80 and 6200 become useless and should be removed.

Then, unrelated to Docker, the input chain misses a stateful rule. Without it, return traffic from host's output (DNS query replies, ping replies, download for upgrades...) will fail. Use this:

    chain input {
            type filter hook input priority 0; policy drop;
            ct state related,established accept
            iif lo accept
            iif eno2 icmp type echo-request accept
            iif eno2 ip 192.168.0.0/16 tcp dport 22 accept
            iif eno2 ip 192.168.0.0/16 tcp dport 443 accept
    }

The input path can still require additional rules for Docker itself (rather than its containers): rules might be needed to allow remote access to the Docker API (if security considerations allow it) or various features like VxLAN used by Docker swarm.

inet filter output

Likewise, OP's inet filter output chain's drop policy kills host connectivity (DNS queries, ping requests or downloads can't be initiated etc.). There should either be a policy accept, or the exceptions for needed outgoing traffic from host itself should be added. The chain should include at least something like this:

  chain output {
          type filter hook output priority 0; policy drop;
          ct state related,established accept
          oif lo accept
          udp dport { 53, 123 } accept
          tcp dport { 53, 80, 443 } accept
          icmp type echo-request accept
  }

The containers' packets are not evaluated by these chains, but by the forward chain and won't be restricted.

IPv6

Using the inet family without enabling correctly ICMPv6 prevents any IPv6 connectivity, because IPv6 doesn't rely on (almost never-firewalled) ARP but on ICMPv6 for link local connectivity. Either use the ip family (and use an other name than filter for the table to avoid any clash with iptables-nft) or deal correctly with ICMPv6: accept them all or check which are required in input and in output direction for correct SLAAC (NDP: RS, RA, NS, NA, ...), ping ... handling.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.