How to force TCP packets to use a specific NIC

Question

My Linux (kernel 3.17) computer has more than one NIC: eth0, eth1, (and/or eth2...)

I have several PC connected to each of them:

PC0 <---> eth0 (192.168.1.100)
PC1 <---> eth1 (192.168.1.101)
PC2 <---> eth2 (192.168.1.102)

An application on the Linux computer started threads to listen to 192.168.1.100 (and the other two). Upon receiving a packet from PC0, the thread replies. How do I guarantee the reply never go through eth1 or eth2 although they have same subnet mask? Similarly, upon receiving packet from pc1, the reply should never go through eth0 or eth2.

Ideally, to achieve what I said, I should use three completely different subnet, such as 10.0.0.1, and 172...., but my colleague suggests it could work, so I am here to find peer opinions. The kernel is in my control, I build it, so any solution to make it work can be proposed.

Explanation

Why three NIC on same subnet? Eth0 is the main IP address that is statically assigned during Linux boot. It shouldn't change but could change if needed. The question is: what if the field people accidentally assigned eth1 and eth2 to have same subnet with eth0? Will my application reply only to eth0 if the socket listens on eth0 at a certain port?

Possible Routing Table

the rule is to go through the NIC if it is where the request is from. I copied from somewhere

ip route add 172.16.10.0/24 dev eth0 src 172.16.10.10 table 10
ip route add default via 172.16.10.251 table 10
ip route add 172.16.10.0/24 dev eth1 src 172.16.10.20 table 20
ip route add default via 172.16.10.251 table 20
ip route add 172.16.10.0/24 dev eth0 src 172.16.10.10
ip route add 172.16.10.0/24 dev eth1 src 172.16.10.20
ip route add default via 172.16.10.251
ip rule add from 172.16.10.10 table 10
ip rule add from 172.16.10.20 table 20
ip route flush cache

Answer 1

Usually, an application binds to an IP address, not to a specific interface. (The exception is applications that mess with the network directly, such as e.g. tcpdump/wireshark etc - but that's not your case, so disregard this.) Most applications can bind either to only one IP address or to all of them (but usually not to more-than-one-but-not-all of them). Which of these apply to your particular application will depend on how it's coded.

When a client connects to this socket, the connection is identified by four items:

1. Your own IP address (e.g. 192.168.1.100)
2. Your own port (e.g. 8080)
3. The client's IP address
4. The client's port

Any responses sent to the data you are receiving on this connection will be sent using these four characteristics - i.e. they will not be sent from another IP or interface on your server.

So, as long as what your application does is to listen on an {IP,port}, accept connections on that {IP,port} and answering back to the client, there shouldn't be any problems.

Where you might run into problems is if your application will sometimes initiate its own connection. In that case, it will (usually) let the computer decide which interface to use. And if your computer considers all three interfaces equally valid, you may get traffic going out on the wrong one.

It is still possible to make your computer always use a particular interface - exactly how you do it will depend on your particular OS/distro. The term to search for is "unix source routing".