Routing traffic to virtual copy of network without namespaces

Question

Given the following diagram, is it possible to route traffic through the linux kernel like this? I wish to simulate an exact copy of the devices outside of my "inner" network, with the same IP-ranges whilst enabling the inner and outer devices to communicate with each other without knowing that the other has the same IP as itself.

For example: Device X on the "inside" contacts 192.168.3.5, this goes to the middleman-bridge and gets forwarded to device Y on the "outside" with IP 192.168.2.5. The response is then sent back to middleman, and sent to device X with IP 192.168.2.5.

I know that this is possible with network-namespaces and have a working simulation with that. However, I wish to avoid namespaces and instead use something like different routing tables for the different directions of traffic. Is this possible?

If I have understood it correctly, I cannot use NAT because of the duplicated IP-ranges. Is this correct?

Answer 1

Topology

We start by configuring the network interfaces on middleman. I'm assuming that either you're logged in on the system console or you have access via an interface that's not involved in either the inner or outer networks. For the purpose of this answer, we're going to assume that interface middleman-eth0 on middleman is connected to the "inner" network and middleman-eth1 is connected to the "outer" network. This gives us the following network topology:

Enable forwarding

We need to ensure that we have enabled packet forwarding on middleman:

sysctl -w net.ipv4.ip_forward=1

And we should start with an empty netfilter configuration. Running iptables-save should produce no output.

Interface configuration

For this to work, both middleman-eth0 and middleman-eth1 will have identical network configurations:

ip addr add 192.168.2.1/24 dev middleman-eth0
ip addr add 192.168.2.1/24 dev middleman-eth1

If you think that looks weird, you're right! At the moment, the routing table on middleman looks like this:

192.168.2.0/24 dev middleman-eth1 proto kernel scope link src 192.168.2.1
192.168.2.0/24 dev middleman-eth0 proto kernel scope link src 192.168.2.1

That's not going to be particularly useful.

VRF configuration

We're going to take advantage of Linux's support for "virtual routing and forwarding" ("VRF"). This allows us to create multiple isolated routing domains on a system, so that traffic coming in on eth0 will see a different routing table than traffic coming in on eth1.

We first create the VRF interfaces:

ip link add vrf-inner type vrf table 100
ip link set vrf-inner up
ip link add vrf-outer type vrf table 200
ip link set vrf-outer up

These commands set up two VRF devices, associating each one with a specific routing table.

Next, we attach each of our physical interfaces to a VRF devices:

ip link set dev middleman-eth0 master vrf-inner
ip link set dev middleman-eth1 master vrf-outer

With these changes, the primary routing table is now empty:

middleman# ip route show
<no output>

In table 100 we see the rules associated with middleman-eth0 (the "inner" network):

middleman# ip route show table 100
broadcast 192.168.2.0 dev middleman-eth0 proto kernel scope link src 192.168.2.1
192.168.2.0/24 dev middleman-eth0 proto kernel scope link src 192.168.2.1
local 192.168.2.1 dev middleman-eth0 proto kernel scope host src 192.168.2.1
broadcast 192.168.2.255 dev middleman-eth0 proto kernel scope link src 192.168.2.1

And in table 200 we see the rules for middleman-eth1 (the "outer" network):

middleman# ip route show table 200
broadcast 192.168.2.0 dev middleman-eth1 proto kernel scope link src 192.168.2.1
192.168.2.0/24 dev middleman-eth1 proto kernel scope link src 192.168.2.1
local 192.168.2.1 dev middleman-eth1 proto kernel scope host src 192.168.2.1
broadcast 192.168.2.255 dev middleman-eth1 proto kernel scope link src 192.168.2.1

At this point, we effectively have two disconnected networks that look like this:

Hosts on the "inner" network can contact 192.168.2.1, and they will be talking to middleman-eth0. Hosts on the "outer" network can also contact 192.168.2.1, but they will be talking to middleman-eth1.

In which the twain do meet

All that we need to do now is set up the mapping so that either side can use addresses from 192.168.3.0/24 to contact nodes on the other side.

First, we need to tell all the nodes that they route to the 192.168.3.0/24 network via middleman; that means on all the nodes, on both the "inner" and "outer" networks, we need:

ip route add 192.168.3.0/24 via 192.168.2.1

On middleman, we need to (a) map addresses in the 192.168.3.0/24 range into the 192.168.2.0/24 range and (b) ensure that when we route the connection we use the correct routing table. To accomplish (a) we can create some NETMAP rules:

iptables -t nat -A PREROUTING -d 192.168.3.0/24 -j NETMAP --to 192.168.2.0/24
iptables -t nat -A POSTROUTING -s 192.168.2.0/24 -j NETMAP --to 192.168.3.0/24

To accomplish (b), we'll first mark packets based on their ingress interface:

iptables -t mangle -A PREROUTING -i middleman-eth0 -d 192.168.3.0/24 -j MARK --set-mark 100
iptables -t mangle -A PREROUTING -i middleman-eth1 -d 192.168.3.0/24 -j MARK --set-mark 200

And then use those marks to select a routing table:

ip rule add prio 100 fwmark 100 lookup 200
ip rule add prio 200 fwmark 200 lookup 100

Recall from earlier that table 100 has the rules for the "inner" network and table 200 has the rules for the "outer" network, so these rules say "for packets arriving on interface middleman-eth0, make a routing decision using the routing table associated with middleman-eth1", and vice-versa.

Following the bouncing ball

With all this in place, if node 192.168.2.10 on the "inner" networks tries to ping 192.168.3.10:

1. The packet gets routed to middleman because of the 192.168.3.0/24 via 192.168.2.1 route entry
2. The packet arrives at middleman-eth0
3. The packet hits the MANGLE table PREROUTING chain and has the fwmark set to 100
4. The packet hits the NAT table PREROUTING chain and has the destination mapped to 192.168.2.10
5. The packet enters the routing subsystem, where it hits the fwmark 100 lookup 200 rule
6. In routing table 200, it hits the 192.168.2.0/24 dev middleman-eth1, so the kernel will send it out device middleman-eth1
7. The packet hits the NAT table POSTROUTING chain, where it has its source mapped to 192.168.3.10.
8. The packet arrives at the "outer" node with address 192.168.2.10.
9. ...take a deep breath...
10. The outer node sends a reply to 192.168.3.10
11. The reply arrives at middleman-eth1
12. The reply hits the MANGLE table PREROUTING chain and has the fwmark set to 200
13. The reply hits the NAT table PREROUTING chain and has the destination mapped to 192.168.2.10
14. The reply enters the routing subsystem, where it hits the fwmark 200 lookup 100 rule
15. In routing table 100, it hits the 192.168.2.0/24 dev middleman-eth0 rule, so the kernel will send it out device middleman-eth0
16. The reply hits the NAT table POSTROUTING chain, where it has its source mapped to 192.168.3.10
17. The reply arrives at "inner" node 192.168.2.10, which sees a reply to the request it initially sent out.

Validation

If on "inner" node 0 (192.168.2.10) we attempt to ping "outer" node 0 using address 192.168.3.10, running tcpdump -nn -i any icmp on inner node 0 shows:

07:01:58.125370 innernode0-eth0 Out IP 192.168.2.10 > 192.168.3.10: ICMP echo request, id 12999, seq 1, length 64
07:01:58.125533 innernode0-eth0 In  IP 192.168.3.10 > 192.168.2.10: ICMP echo reply, id 12999, seq 1, length 64

On middleman we see:

07:01:58.125440 middleman-eth0 In  IP 192.168.2.10 > 192.168.3.10: ICMP echo request, id 12999, seq 1, length 64
07:01:58.125459 middleman-eth1 Out IP 192.168.3.10 > 192.168.2.10: ICMP echo request, id 12999, seq 1, length 64
07:01:58.125514 middleman-eth1 In  IP 192.168.2.10 > 192.168.3.10: ICMP echo reply, id 12999, seq 1, length 64
07:01:58.125518 middleman-eth0 Out IP 192.168.3.10 > 192.168.2.10: ICMP echo reply, id 12999, seq 1, length 64

And on "outer" node 0 we see:

07:01:58.125489 outernode0-eth0 In  IP 192.168.3.10 > 192.168.2.10: ICMP echo request, id 12999, seq 1, length 64
07:01:58.125497 outernode0-eth0 Out IP 192.168.2.10 > 192.168.3.10: ICMP echo reply, id 12999, seq 1, length 64

So I think we have accomplished your goal!

---

I used mininet to test this configuration; you can find the complete sources for my test environment here. There is a video of this configuration in action here.

Update

As A.B. points out in comments, there is a problem with this configuration! By default, the kernel's connection tracking logic looks only at source/destination address and source/destination port. A connection from innernode0 port 4000 to outernode0 port 80 would appear to be the same connection as one in the opposite direction...that is, assuming that I have a webserver running on port 80 on all the nodes, these two commands:

innernode0# curl --local-port 4000 192.168.3.10

And:

outernode0# curl --local-port 4000 192.168.3.10

Would result in a single connection tracking entry on middleman:

middleman# conntrack  -L
tcp      6 118 TIME_WAIT src=192.168.2.10 dst=192.168.3.10 sport=4000 dport=80 src=192.168.2.10 dstroot@mininet-vm:/proc/net=192.168.3.10 sport=80 dport=4000 [ASSURED] mark=0 use=1

We to tell the conntrack subsystem how to differentiate this connections. We can do that by adding a pair of CT rules to the PREROUTING chain in the RAW table:

iptables -t raw -A PREROUTING -s 192.168.2.0/24 -i middleman-eth0 -j CT --zone-orig 100
iptables -t raw -A PREROUTING -s 192.168.2.0/24 -i middleman-eth1 -j CT --zone-orig 200

With these rules in place, we now see two separate connections in the conntrack table:

middleman# conntrack -L
tcp      6 113 TIME_WAIT src=192.168.2.10 dst=192.168.3.10 sport=4000 dport=80 zone-orig=200 src=192.168.2.10 dst=192.168.3.10 sport=80 dport=40568 [ASSURED] mark=0 use=1
tcp      6 112 TIME_WAIT src=192.168.2.10 dst=192.168.3.10 sport=4000 dport=80 zone-orig=100 src=192.168.2.10 dst=192.168.3.10 sport=80 dport=4000 [ASSURED] mark=0 use=1