Understanding docker port forwarding and iptables/nftables

Question

Given a fairly common firewall setup with nftables/iptables (OUTPUT accept, INPUT/FORWARD accept established+related, default drop):

table ip nat {
    chain DOCKER {
        iifname "docker0" return
        iifname != "docker0" meta l4proto tcp ip daddr 172.17.0.1 tcp dport 5000 dnat to 172.17.0.2:5000
        iifname != "docker0" meta l4proto tcp ip daddr 127.0.0.1 tcp dport 5000 dnat to 172.17.0.2:5000
    }

    chain POSTROUTING {
        type nat hook postrouting priority srcnat; policy accept;
        oifname != "docker0" ip saddr 172.17.0.0/16 masquerade
        meta l4proto tcp ip saddr 172.17.0.2 ip daddr 172.17.0.2 tcp dport 5000 masquerade
    }

    chain PREROUTING {
        type nat hook prerouting priority dstnat; policy accept;
        fib daddr type local jump DOCKER
    }

    chain OUTPUT {
        type nat hook output priority -100; policy accept;
        ip daddr != 127.0.0.0/8 fib daddr type local jump DOCKER
    }
}
table ip filter {
    chain INPUT {
        type filter hook input priority filter; policy drop;
        iif "lo" accept comment "Allow loopback"
        ct state invalid drop
        ct state established,related accept
        iifname "bond0-data" tcp dport 22 accept
        udp dport 3052 accept
        iifname "bond0-data" tcp dport 9090 accept
        log drop
    }

    chain FORWARD {
        type filter hook forward priority filter; policy drop;
        jump DOCKER-USER
        jump DOCKER-ISOLATION-STAGE-1
        oifname "docker0" ct state related,established accept
        oifname "docker0" jump DOCKER
        iifname "docker0" oifname != "docker0" accept
        iifname "docker0" oifname "docker0" accept
        log drop
    }

    chain OUTPUT {
        type filter hook output priority filter; policy accept;
    }

    chain FORWARD-OVERRIDE {
        type filter hook forward priority filter + 10; policy accept;
        ct state invalid drop
        ct state established,related accept comment "Accept established, related"
        iifname "bond1-control" drop comment "Drop new connections from bond1-control"
    }

    chain DOCKER {
        iifname != "docker0" oifname "docker0" meta l4proto tcp ip daddr 172.17.0.2 tcp dport 5000 accept
    }

    chain DOCKER-ISOLATION-STAGE-1 {
        iifname "docker0" oifname != "docker0" jump DOCKER-ISOLATION-STAGE-2
        return
    }

    chain DOCKER-ISOLATION-STAGE-2 {
        oifname "docker0" drop
        return
    }

    chain DOCKER-USER {
        return
    }
}

Note that container 172.17.0.2:5000 is mapped to port 5000 using docker's port forwarding.

If I try to connect from container 172.17.0.3 directly to 172.17.0.2:5000 then this works. If I try to connect from container 172.17.0.3 through the host IP (172.17.0.1:5000) then this does not work, unless I add iffname "docker0" accept to the INPUT chain.

I would like to understand why.

The kern.log shows the following when trying to connect:

IN=docker0 OUT= PHYSIN=vethbdc197b MAC=xxxxxxx SRC=172.17.0.3 DST=172.17.0.1 LEN=44 TOS=0x00 PREC=0x00 TTL=58 ID=37344 PROTO=TCP SPT=52535 DPT=5000 WINDOW=1024 RES=0x00 SYN URGP=0

Why is it that this connection is blocked when direct connections are allowed? My best guess at the moment is the direct connection is somehow allowed by the OUTPUT accept and then INPUT sees it as established,related. But it's not clear how things differ when going through the docker port mapping. Maybe DNAT results in a separate INPUT connection that then gets blocked?

Can anyone shed some light on how this works from the iptables/nftables perspective?

Answer 1

The answer for this turns out to be quite straightforward. I was under the impression that containers are just seen as 'extra' IPs by the hosts network stack. This isn't true - containers have their own network stack which is separate to the host (technically containers could also have their own iptables/nftables rules).

This means that direct container-to-container connections don't go through the OUTPUT or INPUT chains of the host, because the IPs involved are not local IPs on the host's network stack. A direction connection looks like this:

• Container A OUTPUT (probably blank)
• Host FORWARD
• Container B INPUT (probably blank)

The fact that this goes through FORWARD on the host is a little surprising - I have tested and this is definitely the case. It appears to be because of the way the bridging works.

In this case the host allows the connection because the FORWARD chain has iifname "docker0" oifname "docker0" accept.

On the flip-side, if container A tries to connect to the forwarded port on the host then it does enter the host's INPUT chain and with the above rules it gets blocked.