I have two different ISPs. I want to set some kind of load balancing setup that will distribute packets to those providers. I know this can be done using different routing tables, but I wanted to use something called "multipath gateway".

I've configured both interfaces in the /etc/network/interfaces file. Both of the connections work separately. I replaced the default gateways with the one below:

# ip route add default \
    nexthop via dev bond0 weight 1 \
    nexthop via dev wwan0 weight 1

I added masquerade targets in iptables on both of the interfaces:

iptables -t nat -A POSTROUTING -o wwan0 -j MASQUERADE
iptables -t nat -A POSTROUTING -o bond0 -j MASQUERADE

Also I enabled (partially) reverse path filtering via sysctl:

net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2

This setup works. Packets (connections) are sent via both interfaces. There's just one problem I don't get.

When I want to check my IP address using the following commands:

$ curl text.whatisyourip.org
$ curl eko.one.pl/host.php

The IP address is different in both cases, which means that the mechanism works well. Also I can see it working in wireshark. But when I'm trying to send, for instance, multiple requests to the first of the domains above, I always get the same IP address in response. So it looks like that packets that are destined to the specific IP address always go through the same interface. I'm just wondering why. Is there any mechanism that remembers the destination IP addresses of the previous requests and makes the next requests to the same addresses to go through the same interface?

  • what you mean "multiple requests"? IP-packets? What exactly do you do to produce the traffic? – Serge May 20 '16 at 11:50
  • Your kernel may be caching routes, if it's an older version. Does ip route show cache display anything? BTW, be happy, quite a few websites assign a session to an IP address, so if after logging in you connect froma different source IP you need to login again. And things stored in the session aren't always accessible, etc. (I found this out when using a load-balanced proxy setup that connected to the outside with multiple source IPs). – wurtel May 20 '16 at 11:55
  • 1
    There's nothing in ip route show cache . And by "multiple requests" I mean that I send the commands 10-20 times in a row, just by typing them into a terminal. – Mikhail Morfikov May 20 '16 at 11:58

I've managed to solve the problem. In this link you can read the following:

IPv4: Hash-based multipath routing. When the routing cache was removed in 3.6, the IPv4 multipath algorithm changed from more or less being destination-based into being quasi-random per-packet scheduling. This increased the risk of out-of-order packets and made it impossible to use multipath together with anycast services. In this release, the multipath routing implementation is replaced with a flow-based load balancing based on a hash over the source and destination addresses merge commit

So even though the cache was removed in kernel 3.6, the requests are still being cached. Now the source and the destination addresses matter. So that's why the packets go always through the same interface.

  • You should also mention why a hash over source and destination is needed: Normal internet protocols (TCP/UDP) are not "multihomed", and your two ISPs will give you different public addresses. The protocols can't deal with that, so packets randomly distributed to the "other" ISP would just count as packet loss. You need a deterministic choiced based on the connection. – dirkt Aug 15 '18 at 17:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.