How to forward packets with keeping the original IP address

Question

I need forward packets from one server ("as a proxy") to another with keeping the original IP address of clients. Topology is: X.X.X.X - public IP1 ("proxy server") Y.Y.Y.Y - public IP2 (host for virtualization) 172.16.0.2 - private IP (virutal machine with web server)

some public IP ---> X.X.X.X ---> Y.Y.Y.Y ---> 172.16.0.2 (web server)

On systems with X.X.X.X and Y.Y.Y.Y forwarding is enabled.

Traffic from some public IP via X.X.X.X is on X.X.X.X routed to Y.Y.Y.Y but it never gets there. I captured it with tcpdump. If I use masquerade on "proxy server" it works OK, but the original ip address is not preserved. If I use DNAT for port 80 only form Y.Y.Y.Y to 172.16.0.2 - iptables -t nat -A PREROUTING -d Y.Y.Y.Y/32 -i venet0 -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.16.0.2:80 and I try Y.Y.Y.Y:80 it works ok. Problem is the most probably on "proxy server" with public IP X.X.X.X.

"Proxy server" host IP X.X.X.X

It has only one interface with connect to the internet - eth0.

iptables rule: (forwarding in filter table is allowed)

iptables -t nat -A PREROUTING -i eth0 -p tcp -m tcp --dport 443 -j DNAT --to-destination Y.Y.Y.Y:443

Route table: It uses main table with:

default via X.X.X.1 dev eth0 onlink

Host IP Y.Y.Y.Y

It has only one interface with connect to the internet - venet0. For VM is used Qemu and interface br0.

iptables rule: (forwarding in filter table is allowed)

iptables -t nat -A PREROUTING -d Y.Y.Y.Y/32 -i venet0 -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.16.0.2:443
iptables -t nat -A POSTROUTING -s 172.16.0.0/24 ! -o br0 -j MASQUERADE

Route table: It uses main table with:

default via 255.255.255.254 dev venet0

Host IP 172.16.0.2

It has only one interface with connect to the internet - ens6 Route table: It uses main table with:

default via 172.16.0.1 dev ens6 proto static

iptables rules according to gapsf answer:

X.X.X.X iptables rules:

iptables -t nat -A PREROUTING -i eth0 -p tcp -m tcp --dport 443 -j DNAT --to-destination Y.Y.Y.Y
iptables -t nat -A POSTROUTING -s Y.Y.Y.Y/32 -o eth0 -p tcp -m tcp --sport 443 -j SNAT --to-source X.X.X.X
iptables -A FORWARD -d Y.Y.Y.Y/32 -p tcp -m tcp --dport 443 -j ACCEPT
iptables -A FORWARD -s Y.Y.Y.Y/32 -p tcp -m tcp --sport 443 -j ACCEPT

Y.Y.Y.Y iptables rules:

iptables -t nat -A PREROUTING -i venet0 -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.16.0.2:443
iptables -t nat -A POSTROUTING -s 172.16.0.2/32 -o venet0 -p tcp -m tcp --sport 443 -j SNAT --to-source Y.Y.Y.Y
iptables -A FORWARD -d 172.16.0.2/32 -i venet0 -o br0 -p tcp -m tcp --dport 443 -j ACCEPT
iptables -A FORWARD -s 172.16.0.2/32 -i br0 -o venet0 -p tcp -m tcp --sport 443 -j ACCEPT

172.16.0.2 iptables rules:

iptables -A INPUT -i ens6 -p tcp -m multiport --dports 80,443 -j ACCEPT
iptables -A OUTPUT -o ens6 -p tcp -m tcp --sport 443 -j ACCEPT

Could you help me please - where is the problem?

Answer 1

In comments you've indicated the web servers to which you want to forward traffic do not have a route to return packets directly to the web clients. This is a very common scenario, but it's one that's too complex for iptables.

The usual solutions involve either:

1. A hardware or software load balancer product in a kind of "DMZ" subnet that is the machine with the default route for the web servers. The load balancer has the public X.X.X.X ip address that the clients know, and via DNAT forwards the traffic to the web servers with the source address unchanged. The load balancer is the default gateway for the web servers, so their return packets can pass back through the balancer and it can perform SNAT on the return traffic. This sophisticated software and/or hardware has a cost that may put it out of reach for many organizations.

2. Perform SNAT when forwarding traffic to the web servers, but use a proxy to add an HTTP header that informs the web server software of the client's true IP address. The header I've usually seen used for this is X-Forwarded-For:. Many HTTP server libs will automatically recognize this header and make the IP address in the header available for the app to use. Some apps expect only a single IP address in the header, some can handle a list of addresses (because the traffic may pass through a proxy before it reaches your proxy). Most web proxy software can add this header.

As with everything else, there are tradeoffs. The 2nd option of proxying the HTTP traffic and adding a header requires the HTTP traffic match the accepted standards so the proxy software can manipulate it. If your clients talk a custom protocol that's loosely based on HTTP but not fully compliant, the proxy's attempts to modify the requests can cause trouble. For example if your protocol uses URLs that are longer than 64k bytes. (I've run into that issue before, though it was a number of years ago).

Given the requirements you've mentioned, I think #2 is the most likely solution for you at first. If your web traffic grows, you'll probably outgrow it and need a more sophisticated (and costly) load balancing solution.

Answer 2

You dont need expensive hardware - only routing+iptables just enough I suppouse X.X.X.X host have one interface only?

on X.X.X.X

ip route add default via eth0
PREORUTING in eth0 dport 80 DNAT to Y.Y.Y.Y
POSTROUTING out eth0 src Y.Y.Y.Y sport 80 SNAT to X.X.X.X
FORWARD dst Y.Y.Y.Y dport 80 ACCEPT
FORWARD src Y.Y.Y.Y sport 80 ACCEPT

on Y.Y.Y.Y

ip route add default via venet0
ip route add 172.16.0.2 via br0
PREROUTING in venet0 dport 80 DNAT to 172.16.0.2
POSTROUTING out venet0 src 172.16.0.2 sport 80 SNAT to Y.Y.Y.Y
FORWARD in venet0 out br0 dst 172.16.0.2 dport 80 ACCEPT
FORWARD in br0 out venet0 src 172.16.0.2 sport 80 ACCEPT

on 172.16.0.2

ip route add default via 172.16.0.1
INPUT in ens6 dport 80 ACCEPT
OUTPUT out ens6 sport 80 ACCEPT

Part 2
X.X.X.X iptables rules:

iptables -t nat -A PREROUTING -i eth0 -p tcp -m tcp --dport 443 -j DNAT --to-destination Y.Y.Y.Y
iptables -t nat -A POSTROUTING -s Y.Y.Y.Y/32 -o eth0 -p tcp -m tcp --sport 443 -j SNAT --to-source X.X.X.X
iptables -A FORWARD -d Y.Y.Y.Y/32 -p tcp -m tcp --dport 443 -j ACCEPT
iptables -A FORWARD -s Y.Y.Y.Y/32 -p tcp -m tcp --sport 443 -j ACCEPT

Y.Y.Y.Y iptables rules:

iptables -t nat -A PREROUTING -i venet0 -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.16.0.2:443
iptables -t nat -A POSTROUTING -s 172.16.0.2/32 -o venet0 -p tcp -m tcp --sport 443 -j SNAT --to-source Y.Y.Y.Y
iptables -A FORWARD -d 172.16.0.2/32 -i venet0 -o br0 -p tcp -m tcp --dport 443 -j ACCEPT
iptables -A FORWARD -s 172.16.0.2/32 -i br0 -o venet0 -p tcp -m tcp --sport 443 -j ACCEPT

172.16.0.2 iptables rules:

iptables -A INPUT -i ens6 -p tcp -m multiport --dports 80,443 -j ACCEPT
iptables -A OUTPUT -o ens6 -p tcp -m tcp --sport 443 -j ACCEPT