1

Let me start with a diagram:

Workstation -> TrafficShaper -> Internet
                     |             ^
                     V             |
                LinuxProxy ---------

My traffic shaper has a feature which allows me to divert packets to another host (the LinuxProxy in the diagram). It does this by setting the ethernet destination MAC address to the LinuxProxy's interface and places the packets onto the network where the LinuxProxy will receive them. The LinuxProxy is free to do what it wants with the packet, and any responses should be placed on the network with the TrafficShaper set as the ethernet destination.

As an example, let's say the Workstation sends a TCP SYN to www.google.com. The packet flows to the TrafficShaper, and the TrafficShaper sends it to the LinuxProxy. The LinuxProxy tries to reply with a SYN-ACK (which would flow to the TrafficShaper, then back to the Workstation), however there is an issue where the LinuxProxy doesn't know what ethernet destination MAC address to use. The TrafficShaper's data interface does not have an IP bound, so I think that means I can't set a route to send the traffic directly back to the TrafficShaper. To make matters worse, the MAC address of the TrafficShaper could change depending on which Workstation is making the request. I need the LinuxProxy to send reply packets to whichever MAC address they originally came from. If I'm just trying to do this for TCP connections, is there a way to remember the MAC address per connection and always send reply packets to that MAC? Or maybe remember the MAC address for a source IP address and always use that MAC when replying to that IP?

Update w/ tcpdump (tcpdump -n -nn -i eth1 -e -vv port 80 or arp):

1) When a Workstation makes an HTTP request, here's what the LinuxProxy sees if I have no routes configured on LinuxProxy:

16:02:02.382023 TrafficShaper:VirtualInterfaceX > LinuxProxy:eth1, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 122, id 30174, offset 0, flags [DF], proto TCP (6), length 52)
Workstation:IP.21875 > RandomWebserver.80: Flags [S], cksum 0x2ffb (correct), seq 1782577522, win 65535, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0

I don't even see the SYN-ACK because without a route, the LinuxProxy has no idea which interface to respond on. I even see the following in dmesg:

[177553.396852] device eth1 entered promiscuous mode
[177555.286091] IPv4: martian source LinuxProxyIP from WorkstationIP, on dev eth1
[177555.294651] ll header: 00000000: XX XX XX XX XX RT.1....5.:...
[177560.529102] device eth1 left promiscuous mode

2) Let's add a route so LinuxProxy knows which interface to reply on (note, the traffic shaper has no IP, so I only put the interface here):

route add default dev eth1

Now my tcpdump shows LinuxProxy trying to ARP for the Workstation's IP, but the TrafficShaper doesn't answer ARPs, and it doesn't forward the ARPs to the other side. (having the Workstation's MAC is useless anyways)

16:41:14.800321 TrafficShaper:VirtualInterfaceX > LinuxProxy:eth1, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 122, id 30787, offset 0, flags [DF], proto TCP (6), length 48)
Workstation:IP.21975 > RandomWebserver.80: Flags [S], cksum 0xc8cf (correct), seq 2481765147, win 65535, options [mss 1460,nop,nop,sackOK], length 0
16:41:14.800432 LinuxProxy:eth1 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has Workstation:IP tell 10.89.14.11, length 28
16:41:15.800890 LinuxProxy:eth1 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has Workstation:IP tell 10.89.14.11, length 28
16:41:16.802797 LinuxProxy:eth1 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has Workstation:IP tell 10.89.14.11, length 28
16:41:17.792513 TrafficShaper:VirtualInterfaceX > LinuxProxy:eth1, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 122, id 30788, offset 0, flags [DF], proto TCP (6), length 48)
Workstation:IP.21975 > RandomWebserver.80: Flags [S], cksum 0xc8cf (correct), seq 2481765147, win 65535, options [mss 1460,nop,nop,sackOK], length 0

3) Just to verify that this works if I hardcode the MAC from the TrafficShaper, I create a fake ARP entry and change the route to goto that IP. Note: I can't do this in production because TrafficShaper has about 50 different virtual interfaces, each with their own MAC that my many Workstations are load-balanced through:

ip neighbor add FAKE_IP lladdr TrafficShaper:VirtualInterfaceX dev eth1
route del default dev eth1
route add default gw FAKE_IP

And now the 3-way handshake completes:

16:52:17.077903 TrafficShaper:VirtualInterfaceX > LinuxProxy:eth1, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 122, id 31076, offset 0, flags [DF], proto TCP (6), length 52)
Workstation:IP.22033 > RandomWebserver.80: Flags [S], cksum 0x5a9f (correct), seq 3325648471, win 65535, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
16:52:17.078024 LinuxProxy:eth1 > TrafficShaper:VirtualInterfaceX, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
RandomWebserver.80 > Workstation:IP.22033: Flags [S.], cksum 0x99ab (incorrect -> 0x18fa), seq 1232307723, ack 3325648472, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
16:52:17.157231 TrafficShaper:VirtualInterfaceX > LinuxProxy:eth1, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 122, id 31077, offset 0, flags [DF], proto TCP (6), length 40)
Workstation:IP.22033 > RandomWebserver.80: Flags [.], cksum 0xfbdb (correct), seq 1, ack 1, win 53248, length 0
16:52:17.165426 LinuxProxy:eth1 > TrafficShaper:VirtualInterfaceX, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 64, id 20208, offset 0, flags [DF], proto TCP (6), length 40)
... extras removed ...

I know that each Workstation IP will be load balanced to a specific virtual interface on TrafficShaper. If I had a way to send the reply packets for a given Workstation IP back to the MAC which it received them from, my problem would be solved. I am currently looking through netfilter docs to see if there's a way to save a WorkstationIP->MAC mapping in a dictionary, then lookup that MAC when sending packets back.

  • I am using mitmproxy, however this problem I am having is at layer 2/3. I can't get past the TCP 3-way handshake because I don't know how to tell Linux what MAC address to put in the ethernet header. To add more detail, my traffic shaper has many interfaces on it, and the user traffic flowing through the traffic shaper gets load balanced by the user. So user X's traffic will always flow through interface X which always has MAC address X ... – mykospark Mar 20 at 19:34
  • If I hardcode the return MAC address (by adding a "fake" ARP entry using the user's IP address and the traffic shaper's MAC address), then the system seems to work fine. The problem is that I can't hardcode each of these "fake" ARP entries because there is a very large number of users and the group of users is constantly evolving (adding/removing users many times per minute). The simple heuristic is for Linux to remember what MAC address that user's IP was using, then make sure all replies to that user's IP use the same MAC address ... – mykospark Mar 20 at 19:35
  • I see that Linux is actually trying to ARP on the network using the user's IP, but the traffic shaper won't answer the ARP queries. I presume this is because the traffic shaper isn't a router and doesn't want to be bothered with "router" functions. In any case, I have reached out to our vendor asking if the traffic shaper can reply to ARPs for users it services, but I suspect this won't happen. – mykospark Mar 20 at 19:36
  • I think we're on the same page now. I'm trying to find some combination of netfilter/iproute/iprule/etc that might be able to solve my problem. I'll look around for "ARP family mode". Thanks for the hint! – mykospark Mar 20 at 20:01
  • In the end, I think the question should definitely include some captures of arp requests and replies to see what alterations are done to get an idea. – A.B Mar 20 at 22:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.