Root network namespace as transit between 2 other net namespaces

Question

I am trying to communicate between two network namespaces that are connected through the root namespaces using veth pairs as seen in the diagram. I am unable to perform a ping from netns A to netns B. Additionally I can ping from root namespace to both netns A (VA IP) and B (VB IP).

            +-------+         +-------+
            |   A   |         |   B   |
            +-------+         +-------+
                | VA              | VB
                |                 |
                | RA              | RB
            +-------------------------+
            |                         |
            |      Root namespace     |
            |                         |
            +-------------------------+

ip netns add A
ip netns add B

ip link add VA type veth peer name RA
ip link add VB type veth peer name RB

ip link set VA netns A
ip link set VB netns B

ip addr add 192.168.101.1/24 dev RA
ip addr add 192.168.102.1/24 dev RB

ip link set RA up
ip link set RB up

ip netns exec A ip addr add 192.168.101.2/24 dev VA
ip netns exec B ip addr add 192.168.102.2/24 dev VB

ip netns exec A ip link set VA up
ip netns exec B ip link set VB up

ip netns exec A ip route add default via 192.168.101.1
ip netns exec B ip route add default via 192.168.102.1

I have tried enabling IP forwarding and there are no IP table rules blocking the traffic. The same works when instead of using root namespace I use another namespace called transit and connect it like below.

    +-------+ VA  RA +-------+ RB   VB +-------+
    |   A   |--------|transit|---------|   B   |
    +-------+        +-------+         +-------+

            +-------------------------+
            |                         |
            |      Root namespace     |
            |                         |
            +-------------------------+

Here I am successful in pinging between namespaces A and B.

Why is it that the traffic gets dropped at root namespace and does not when a third transit namespace is used instead?

There are a few iptable rules installed by docker, but I do not see any conflict.

rahul@inception:~$ sudo iptables -L -n -v 
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy DROP 2 packets, 168 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    2   168 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    2   168 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    2   168 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    2   168 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

nft list format

rahul@inception:~$ sudo nft list ruleset
table ip nat {
    chain DOCKER {
        iifname "docker0" counter packets 0 bytes 0 return
    }

    chain POSTROUTING {
        type nat hook postrouting priority srcnat; policy accept;
        oifname != "docker0" ip saddr 172.17.0.0/16 counter packets 1 bytes 90 masquerade 
    }

    chain PREROUTING {
        type nat hook prerouting priority dstnat; policy accept;
        fib daddr type local counter packets 148 bytes 11544 jump DOCKER
    }

    chain OUTPUT {
        type nat hook output priority -100; policy accept;
        ip daddr != 127.0.0.0/8 fib daddr type local counter packets 3 bytes 258 jump DOCKER
    }
}
table ip filter {
    chain DOCKER {
    }

    chain DOCKER-ISOLATION-STAGE-1 {
        iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
        counter packets 2 bytes 168 return
    }

    chain DOCKER-ISOLATION-STAGE-2 {
        oifname "docker0" counter packets 0 bytes 0 drop
        counter packets 0 bytes 0 return
    }

    chain FORWARD {
        type filter hook forward priority filter; policy drop;
        counter packets 2 bytes 168 jump DOCKER-USER
        counter packets 2 bytes 168 jump DOCKER-ISOLATION-STAGE-1
        oifname "docker0" ct state related,established counter packets 0 bytes 0 accept
        oifname "docker0" counter packets 0 bytes 0 jump DOCKER
        iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
        iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
    }

    chain DOCKER-USER {
        counter packets 2 bytes 168 return
    }
}

ip route

rahul@inception:~$ ip route
default via 192.168.0.1 dev wlo1 proto dhcp metric 600 
169.254.0.0/16 dev wlo1 scope link metric 1000 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.0.0/24 dev wlo1 proto kernel scope link src 192.168.0.101 metric 600 
192.168.101.0/24 dev RA proto kernel scope link src 192.168.101.1 
192.168.102.0/24 dev RB proto kernel scope link src 192.168.102.1

Using TCPDUMP I found that the packet is reaching the root namespace. Is there any debugging tool that I can learn and can be used to see where the packet is traversing inside the namespace (like strace or ftrace)?

Answer 1

Docker changes the default forward policy to DROP:

Docker also sets the policy for the FORWARD chain to DROP. If your Docker host also acts as a router, this will result in that router not forwarding any traffic anymore.

As happens in OP's case:

Chain FORWARD (policy DROP 2 packets, 168 bytes)

The documentation also tells:

If you want your system to continue functioning as a router, you can add explicit ACCEPT rules to the DOCKER-USER chain to allow it:

The fix is to enable such traffic in the DOCKER-USER chain (by inserting in this chain rather than appending, because there's a final RETURN target).

The minimal fix to have the experiment unhindered without allowing more would thus be:

iptables -I DOCKER-USER 1 -i RA -o RB -j ACCEPT
iptables -I DOCKER-USER 2 -i RB -o RA -j ACCEPT