1

What I would like to do

I would like to set up my Debian Stretch GNU/Linux machine to route IP packets between its network interfaces.

Context

These are the network interfaces of available on my Debian box and soon-to-be router:

Index interface name ip
1 enp2s0 192.168.23.91/24
2 enx00e04c360e75 192.168.1.1/24
3 enx000ec667a74a 192.168.2.2/24
  • sysctl net.ipv4.ip_forward returns 1, hence routing is enabled.
  • cat /proc/sys/net/ipv4/conf/*/rp_filter returns nothing but 0s. Hence, reverse path filtering is disabled. (This was the source of concern for a few other questions I found.)

The routing table looks as follows:

$ sudo route -nn
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.23.254  0.0.0.0         UG    100    0        0 enp2s0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 enp2s0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.1.0     0.0.0.0         255.255.255.0   U     101    0        0 enx00e04c360e75
192.168.2.0     0.0.0.0         255.255.255.0   U     102    0        0 enx000ec667a74a
192.168.23.0    0.0.0.0         255.255.255.0   U     100    0        0 enp2s0

The following works

From my device (192.168.1.111) connected on NIC 2, I can ping both my NIC 2 (192.168.1.1) as well as NIC 3 (192.168.2.2).

tcpdump output on my Debian router looks as expected:

$ sudo tcpdump -i enx00e04c360e75 icmp -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enx00e04c360e75, link-type EN10MB (Ethernet), capture size 262144 bytes
18:10:51.919808 IP 192.168.1.111 > 192.168.1.1: ICMP echo request, id 25404, seq 17, length 64
18:10:51.919863 IP 192.168.1.1 > 192.168.1.111: ICMP echo reply, id 25404, seq 17, length 64
18:10:52.920019 IP 192.168.1.111 > 192.168.1.1: ICMP echo request, id 25404, seq 18, length 64
18:10:52.920130 IP 192.168.1.1 > 192.168.1.111: ICMP echo reply, id 25404, seq 18, length 64
18:10:53.920808 IP 192.168.1.111 > 192.168.1.1: ICMP echo request, id 25404, seq 19, length 64
18:10:53.920895 IP 192.168.1.1 > 192.168.1.111: ICMP echo reply, id 25404, seq 19, length 64
[...]
18:11:03.408546 IP 192.168.1.111 > 192.168.2.2: ICMP echo request, id 25916, seq 0, length 64
18:11:03.408622 IP 192.168.2.2 > 192.168.1.111: ICMP echo reply, id 25916, seq 0, length 64
18:11:04.405006 IP 192.168.1.111 > 192.168.2.2: ICMP echo request, id 25916, seq 1, length 64
18:11:04.405061 IP 192.168.2.2 > 192.168.1.111: ICMP echo reply, id 25916, seq 1, length 64
18:11:05.405147 IP 192.168.1.111 > 192.168.2.2: ICMP echo request, id 25916, seq 2, length 64
18:11:05.405201 IP 192.168.2.2 > 192.168.1.111: ICMP echo reply, id 25916, seq 2, length 64

The following doesn't work

From my device connected via NIC 2, I cannot ping some host accessible through NIC 1.

tcpdump running on my Debian box shows incoming packets on NIC 2:

$ sudo tcpdump -i enx00e04c360e75 icmp -n
listening on enx00e04c360e75, link-type EN10MB (Ethernet), capture size 262144 bytes
18:11:31.837778 IP 192.168.1.111 > 193.99.144.80: ICMP echo request, id 36668, seq 4, length 64
18:11:32.838830 IP 192.168.1.111 > 193.99.144.80: ICMP echo request, id 36668, seq 5, length 64
18:11:33.838249 IP 192.168.1.111 > 193.99.144.80: ICMP echo request, id 36668, seq 6, length 64
[...]

but they never leave through NIC 1:

$ sudo tcpdump -i enp2s0 icmp -n
listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
[nothing]

When I issue the pings directly from my Debian router box, tcpdump looks like expected:

$ sudo tcpdump -i enp2s0 icmp -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:35:16.577795 IP 192.168.23.91 > 193.99.144.80: ICMP echo request, id 13146, seq 1, length 64
18:35:16.600861 IP 193.99.144.80 > 192.168.23.91: ICMP echo reply, id 13146, seq 1, length 64
18:35:17.578164 IP 192.168.23.91 > 193.99.144.80: ICMP echo request, id 13146, seq 2, length 64
18:35:17.599898 IP 193.99.144.80 > 192.168.23.91: ICMP echo reply, id 13146, seq 2, length 64

Question

What is preventing the ping packets arriving on NIC 2 to be relayed through NIC 1? What do I need to do to enable routing of these packets?

8
  • 1
    Is 192.168.1.111 able to use your Debian Router to communicate to the 192.168.23.0/24 network at all, or is it only connections that cross your other gateway(192.168.23.254) that aren't completed? Also can the 192.168.2.0/24 network communicate with the 192.168.23.0/24 network and/or the 193.99.144.80?
    – ReedGhost
    Jun 22 at 17:23
  • 1
    I would keep rp_filter=1 whenever possible. Requiring an other value to work might mean there's an anomaly in the routing configuration. Anyway 0 is fine for troubleshooting. What would be the output of this command (that mimics your example but doesn't send any packet): ip route get from 192.168.1.111 iif enx00e04c360e75 to 193.99.144.80? The result can even change depending on rp_filter. Also, still for troubleshooting, did you disable any firewall in place?
    – A.B
    Jun 22 at 21:25
  • @ReedGhost: (1) Except for the Debian router box in question (192.168.23.91), 192.168.1.111 cannot communicate to network 192.168.23.0/24 at all. I tried to ping a collegues machine 191.168.23.98 but the connections didn't complete. I see the ping packets enter at one interface but they don't leave through the other. (2) Devices on the 192.168.2.0/24 network show the same behaviour as the ones in 192.168.1.0/24. That is, they can ping all the ip addresses of my Debian router box, but nothing else.
    – Multisync
    Jun 23 at 12:33
  • 1
    Try this: disable docker (systemctl disable --now docker; systemctl mask docker.socket), reboot (yes really, docker makes a lot of network changes so unless you know which ones, reboot), and try again your routing attempt.
    – A.B
    Jun 23 at 13:11
  • 1
    @A.B You were right. I didn't notice that after rebooting with docker disabled, ipv4 routing (sysctl net.ipv4.ip_forward) got disabled. After reenabling it, I could see the ping requests entering through NIC 2 (192.168.1.1/24) leaving through NIC 1 (192.168.23.91/24). NIC 1 never received any ping replies, but that's due to how the network infrastructure is configured, I assume. Pings to devices connected on NIC 3 (192.168.2.2/24) got replied to and showed up on the tcpdump output of both NIC 2 and NIC 3. If you provide your docker-related comment as answer, I'm happy to accept.
    – Multisync
    Jun 23 at 14:44
1

Summarizing what was exchanged in comments:

  • the system was not explicitly configured as a router

    Ie there is no explicit configuration that applies something like:

    sysctl -w net.ipv4.ip_forward=1
    
  • the system is running Docker

    That means that:

    • Docker turns the system into a router

      It sets itself:

      sysctl -w net.ipv4.ip_forward=1
      
    • Docker prevents routing in the firewall policy

      Docker on a router

      Docker also sets the policy for the FORWARD chain to DROP. If your Docker host also acts as a router, this will result in that router not forwarding any traffic anymore. If you want your system to continue functioning as a router, you can add explicit ACCEPT rules to the DOCKER-USER chain to allow it:

      $ iptables -I DOCKER-USER -i src_if -o dst_if -j ACCEPT
      

So

  • having Docker dropped forwarded packets (except for containers)
  • stopping Docker didn't enable forwarding anymore (at least on next boot)

In both cases the system didn't route in the end.

You should probably do both things below:

  • enable routing explicitly (somewhere in /etc/sysctl.conf or /etc/sysctl.d/)
  • follow Docker guide in previous link to allow routed traffic
1
  • That is an incredibly structured and well written answer. Indeed, when rebooting the machine without docker engine running, everything behaved as expected. I just had to turn on kernel ip_forward manually.
    – Multisync
    Jun 24 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.