20

I got two NICs on the server side, eth0 ? 192.168.8.140 and eth1 ? 192.168.8.142. The client sends data to 192.168.8.142, and I expect iftop to show the traffic for eth1, but it does not. All networks go through eth0, so how can I test the two NICs?

Why does all the traffic go through eth0 instead of eth1? I expected I could get 1 Gbit/s per interface. What's wrong with my setup or configuration?

Server

ifconfig

eth0    Link encap:Ethernet  HWaddr 00:00:00:19:26:B0
        inet addr:192.168.8.140  Bcast:0.0.0.0  Mask:255.255.252.0
        inet6 addr: 0000::0000:0000:fe19:26b0/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
        RX packets:45287446 errors:0 dropped:123343 overruns:2989 frame:0
        TX packets:3907747 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000
        RX bytes:66881007720 (62.2 GiB)  TX bytes:261053436 (248.9 MiB)
        Memory:f7e00000-f7efffff

eth1    Link encap:Ethernet  HWaddr 00:00:00:19:26:B1
        inet addr:192.168.8.142  Bcast:0.0.0.0  Mask:255.255.255.255
        inet6 addr: 0000::0000:0000:fe19:26b1/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
        RX packets:19358 errors:0 dropped:511 overruns:0 frame:0
        TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000
        RX bytes:1772275 (1.6 MiB)  TX bytes:1068 (1.0 KiB)
        Memory:f7c00000-f7cfffff

Server side

# Listen for incomming from 192.168.8.142
nc -v -v -n -k -l 192.168.8.142 8000 | pv > /dev/null
Listening on [192.168.8.142] (family 0, port 8000)
Connection from 192.168.8.135 58785 received!

Client

# Send to 192.168.8.142
time yes | pv |nc -s 192.168.8.135 -4 -v -v -n 192.168.8.142 8000 >/dev/null
Connection to 192.168.8.142 8000 port [tcp/*] succeeded!

Server side

$ iftop -i eth0
interface: eth0
IP address is: 192.168.8.140

TX:             cumm:  6.34MB   peak: 2.31Mb   rates: 2.15Mb  2.18Mb  2.11Mb
RX:                    2.55GB          955Mb           874Mb   892Mb   872Mb
TOTAL:                 2.56GB          958Mb           877Mb   895Mb   874Mb

$ iftop -i eth1
interface: eth1
IP address is: 192.168.8.142

TX:             cumm:      0B   peak:     0b   rates:     0b      0b      0b
RX:                    4.51KB         3.49Kb          3.49Kb  2.93Kb  2.25Kb
TOTAL:                 4.51KB         3.49Kb          3.49Kb  2.93Kb  2.25Kb

$ ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:00:00:19:26:b0 brd ff:ff:ff:ff:ff:ff
$ ip link show eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:00:00:19:26:b1 brd ff:ff:ff:ff:ff:ff
  • 1
    It sounds like what you're really looking for is interface bonding: wiki.linuxfoundation.org/networking/bonding – Flexo Jan 23 '18 at 8:33
  • @flexo is totally right - depending on your ultimate goal, bonding the two network interfaces together can give you more overall bandwidth, but the bonding options vary. Best you can get is 2 streams of ~1 Gbit, and not 1 stream of 2 Gbit. Plus you require the services of a managed ethernet switch. Likewise, bonding 4 could give 4x 1 Gbit flows at once. – Criggie Jan 23 '18 at 9:19
32

There are two possible design models for a TCP/IP network stack: a strong host model and a weak host model. You're expecting behavior that would match the strong host model. Linux is designed to use the weak host model. In general the weak host model is more common as it reduces the complexity of the routing code and thus might offer better performance. Otherwise the two host models are just different design principles: neither is inherently better than the other.

Basically, the weak host model means that outgoing traffic will be sent out the first interface listed in the routing table that matches the IP address of the destination (or selected gateway, if the destination is not reachable directly), without regard to the source IP address.

This is basically why it's generally inadvisable to use two separate physical interfaces if you need two IP addresses on the same network segment. Instead assign two IP addresses for one interface (IP aliases: e.g. eth1 = 192.168.8.142 and eth1:0 = 192.168.8.140). If you need more bandwidth than a single interface can provide, bond (or team, if applicable) two or more interfaces together, and then run both IPs on the bond/team.

By tweaking a number of sysctl settings and using the "advanced routing" functionality to set up independent routing tables for each NIC, it is possible to make Linux behave like a strong-host-model system. But that is a very special configuration, and I would recommend thinking twice before implementing it.

See the answers at Linux Source Routing, Strong End System Model / Strong Host Model? if you really need it.

  • The default mode is also a bad surprise waiting if attempting to shape traffic with iptables :) – rackandboneman Jan 22 '18 at 22:14
  • yeah, I've had fun implementing the strong host model in the past. It was required for that project, but I wouldn't bother going through that headache for a personal machine. – Baldrickk Jan 23 '18 at 12:31
11

An additional point to consider is that the eth1 interface is configured with a subnet mask of 255.255.255.255.

This means that the eth1 interface is configured to expect no other devices(hosts) on its network interface. This means that it won't be able to communicate with your 192.168.8.142 client.

2

After a lot of search, I found Why netcat doesn't use the right interface associated with the IP?, and this is the same problem. As @telcoM said, outgoing traffic will be sent out to the first interface, and that's the problem, so, the easiest way to solve this is:

ip route add default via 192.168.8.142 dev eth1 table 142
ip rule add from 192.168.8.142 table 142

This route will make ip route get 192.168.8.135 from 192.168.8.142 return eth1 instead of eth0. Then everything works as expected.

  • 3
    If you omit the ARP sysctl settings mentioned in the question I referred to, and are dealing with enterprise-grade switches and routers, your network admin will be a bit unhappy with you causing unnecessary "IP address flapping" messages at the router, as the system may still answer to ARP requests for both IPs on both interfaces. Or if you have IP hijacking protection in the network, it might engage and disable all traffic to your system. – telcoM Jan 22 '18 at 8:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.