iptables --set-mark - Route diferent ports through different interfaces

Question

Short story,
3 interfaces, eth0 (LAN), eth1(ADSL), eth2(4G).
eth0 -> eth1: Works
(ports 80, 443, 4070) eth0 -> eth2: Doesn't happen

This is a graphical representation of the idea:

Port 80 & 443 via eth2
ant the rest via eth1

Netscheme:

eth0: -ip 10.0.0.1 -net 10.0.0.0/8 -gw 10.0.0.1 (the servers own intf) 
eth1: -ip 192.168.1.74 -net 192.168.1.0/24 -gw 192.168.1.254 
eth2: -ip 192.168.1.91 -net 192.168.0.0/24 -gw 192.168.0.1 

---

---

This new script reroutes 22 and 4070 to the proper table, I think.
However, after getting to that table, it doesn't get rerouted to eth2.

---

This script works, except for 22 and 4070!

(Port 80 is un-commented and it works but via eth1 which is wrong.)

modprobe iptable_nat
modprobe ip_conntrack

echo "1" > /proc/sys/net/ipv4/ip_forward

iptables -P INPUT ACCEPT
iptables -F INPUT
iptables -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -F PREROUTING
iptables -t nat -F
iptables -t mangle -F
iptables -F
# This next line restores any issues trying to connect to something
# if you get weird ACK packets when trying to connect (at least i did)!
iptables -t mangle -A PREROUTING -p tcp -j CONNMARK --restore-mark
ip route flush table main

iptables -A PREROUTING -i eth0 -t mangle -p tcp --dport 22 -j MARK --set-mark 1
###  iptables -A PREROUTING -i eth0 -t mangle -p tcp --dport 80 -j MARK --set-mark 1
iptables -A PREROUTING -i eth0 -t mangle -p tcp --dport 4070 -j MARK --set-mark 1

## Setup routes
# LAN
route add -net 10.0.0.0 netmask 255.0.0.0 dev eth0
# ADSL
route add -net 192.168.1.0 netmask 255.255.255.0 dev eth1
# 4G (Only accessible if marking packages with \x01
route add -net 192.168.0.0 netmask 255.255.255.0 dev eth2
# Default via ADSL
## -- Does the same as ip route below? route add default gw 192.168.1.254


echo "201 eth2.out" >> /etc/iproute2/rt_tables

ip rule add fwmark 1 table eth2.out
ip route add default via 192.168.0.1 dev eth2 table eth2.out
ip route add default via 192.168.1.254 dev eth1



## Setup forwards
# From 4G to LAN
iptables -A FORWARD -i eth2 -o eth0 -m state --state ESTABLISHED,RELATED,NEW -j ACCEPT
# From ADSL to LAN
iptables -A FORWARD -i eth1 -o eth0 -m state --state ESTABLISHED,RELATED,NEW -j ACCEPT
# From LAN to ADSL (Default route out)
# - Note: If marked packages is sent to ADSL they will be mangled and rerouted to 4G
iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE

Old script:

---

  Ignore everything below unless you're interested in retracing my steps!!

---

I've created a router.sh script to setup my environment in case I do something bad. I've got 3 ports which I want to send to a 4G connection and the rest via a landline ADSL connection. To do this, I've instructed iptables to mangle packages on the default route and send them via my 4G interface if the --dport == 443 | 80 | 4070

However, this doesn't work; I'm still getting routed through my landline no matter what.

This is what my script looks like:

#!/bin/bash

## routing tables
# wireless = 4G via eth2
# adsl = adsl via eth1

modprobe iptable_nat
modprobe ip_conntrack

echo "1" > /proc/sys/net/ipv4/ip_forward

iptables -P INPUT ACCEPT
iptables -F INPUT
iptables -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -t nat -F
ip route flush table main
ip route flush table wireless
ip route flush table adsl

## Setup routing tables
# ADSL
ip route add table adsl to 192.168.1.0/24 dev eth1
# 4G
ip route add table wireless to 192.168.0.0 dev eth2
ip rule add fwmark 0x1 table wireless

## Setup routes
# LAN
route add -net 10.0.0.0 netmask 255.0.0.0 dev eth0
# ADSL
route add -net 192.168.1.0 netmask 255.255.255.0 dev eth1
# 4G (Only accessible if marking packages with \x01
route add -net 192.168.0.0 netmask 255.255.255.0 dev eth2
# Default via ADSL
route add default gw 192.168.1.254


## Forward ports into the LAN
iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j DNAT --to 10.0.0.3:80


## Lets mark all packets we want for 4G forward
# HTTPS
iptables -A OUTPUT -t mangle -o eth1 -p tcp --dport 443 -j MARK --set-mark 1
# HTTP
iptables -A OUTPUT -t mangle -o eth1 -p tcp --dport 80 -j MARK --set-mark 1
# Spotify
iptables -A OUTPUT -t mangle -o eth1 -p tcp --dport 4070 -j MARK --set-mark 1

## Setup forwards
# From 4G to LAN
iptables -A FORWARD -i eth2 -o eth0 -m state --state ESTABLISHED,RELATED,NEW -j ACCEPT
# From ADSL to LAN
iptables -A FORWARD -i eth1 -o eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
# From LAN to ADSL (Default route out)
# - Note: If marked packages is sent to ADSL they will be mangled and rerouted to 4G
iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT
iptables -A FORWARD -j LOG
#iptables --table nat --append POSTROUTING --out-interface eth2 --jump SNAT --to-source "192.168.1.74"
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE

I've also tried to add these 3 to the bottomg of the script:

iptables -t nat -A POSTROUTING -o eth2 -p tcp --dport 80 -j SNAT --to "192.168.0.91"
iptables -t nat -A POSTROUTING -o eth2 -p tcp --dport 443 -j SNAT --to "192.168.0.91"
iptables -t nat -A POSTROUTING -o eth2 -p tcp --dport 4070 -j SNAT --to "192.168.0.91"

Also tried without success:

iptables -A PREROUTING -t mangle -i eth0 -p tcp --dport 80 -j MARK --set-mark 1

Last but not least, tried:

## Lets mark all packets we want for 4G forward
# HTTPS
iptables -A POSTROUTING -t mangle -o eth1 -p tcp --dport 443 -j MARK --set-mark 1
# HTTP
iptables -A POSTROUTING -t mangle -o eth1 -p tcp --dport 80 -j MARK --set-mark 1
# Spotify
iptables -A POSTROUTING -t mangle -o eth1 -p tcp --dport 4070 -j MARK --set-mark 1

The routing works, I can browse the web, listen to music and what-not, but I'm doing it through the wrong interface. I've Googled around for a long while now and found bits and pieces to understand what I'm doing and why I'm doing it. I could do traffic shaping via tc but if it's possible via marking packages in iptables it would help me a long way.

My guess is that I'm doing the order wrong on the different rules, mainly the MASQUERADE part? or if that should even be there?

Can someone explain how to DNAT port say, tcp:80 from an external interface (either one or BOTH protocols) to a internal 10.0.0.0 address space?

Outputs:

root@Netbridge:~# route -n Kernel IP routing table Destination    

Gateway         Genmask         Flags Metric Ref    Use Iface<br>
0.0.0.0         192.168.1.254   0.0.0.0         UG    0      0        0 eth1<br>
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 eth0<br>
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth2<br>
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1

root@Netbridge:~# ifconfig

eth0      Link encap:Ethernet  HWaddr 00:0c:29:7e:9e:4e  
          inet addr:10.0.0.1  Bcast:10.255.255.255  Mask:255.0.0.0

eth1      Link encap:Ethernet  HWaddr 00:0c:29:7e:9e:58  
          inet addr:192.168.1.74  Bcast:192.168.1.255  Mask:255.255.255.0

eth2      Link encap:Ethernet  HWaddr 00:0c:29:7e:9e:62  
          inet addr:192.168.0.91  Bcast:192.168.0.255  Mask:255.255.255.0

Followed these instructions:
output-traffic-on-different-interfaces-based-on-destination-por
iptables-forward-specific-port-to-specific-nic
Among a few other related threads.

Answer 1

BatchyX already give some very good explanation about iptables and routing, so I will exercise my laziness and go directly to script.

It should NAT all traffic to port 80,443,22,4070 through 192.168.0.91. All the rest will NAT through 192.168.1.254.

I re-do my testing and end up following this guide. What is missing in that guide is the last 3 lines in my script. Which I found out from another port, but I lost track of that link.

It is a tested working script.

Need Default Route

One thing I did not put in the script is setting up the default route. It should be

route add default gw 192.168.1.254

When you do route -n, it should be the only default route (Dest:0.0.0.0)

0.0.0.0    192.168.1.254    0.0.0.0    UG    0    0    0    eth1

fw-router.sh

# Reset/Flush iptables
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

#Reset/Flush/Setup IP Route (table 4)
ip route flush table 4
ip route show table main | grep -Ev ^default | while read ROUTE ; do ip route add table 4 $ROUTE ; done
ip route add table 4 default via 192.168.0.1

#Mark Packet with matching D.Port
iptables -t mangle -A PREROUTING -p tcp --dport 22   -s 10.0.0.0/24 -j MARK --set-mark 4
iptables -t mangle -A PREROUTING -p tcp --dport 80   -s 10.0.0.0/24 -j MARK --set-mark 4
iptables -t mangle -A PREROUTING -p tcp --dport 443  -s 10.0.0.0/24 -j MARK --set-mark 4
iptables -t mangle -A PREROUTING -p tcp --dport 4070 -s 10.0.0.0/24 -j MARK --set-mark 4

#SNAT Rules
iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 192.168.1.74
iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 192.168.0.91

#IP Route
ip rule add fwmark 4 table 4
ip route flush cache

#IP Stack
#This is the missing part from the guide
echo 1 > /proc/sys/net/ipv4/ip_forward
for f in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 0 > $f ; done
echo 0 > /proc/sys/net/ipv4/route/flush

PS1: In short, MASQUERADE does not work (in most case, and definitely in your case) for NAT with multiple external IPs that need some kind of load balancing or need DNAT to handle incoming traffic. You need SNAT for direction control.

PS2: Pure iptables is not sufficient.

Answer 2

Note: I only have considered the first script, ignoring the old one.

• You don't need to modprobe netfilter modules by hand with current iptables. This is only necessary for custom connection trackers.
• Don't mix up route and ip route. This is pure evil. Just use ip everywhere and forget about ifconfig and route

• /etc/iproute2/rt_tables is not reset across reboots. Appending the same entry over and over is not a good idea, you only need to do it once. Remember that rt_tables just define name aliases to numeric values, it does not change any configuration.

• Now for iptables: In your FORWARD chain, you drop packets coming from LAN to 4G. This is bad. the FORWARD hook is used after routing is done. At this point, all policy routing is done, and it is already known whether the packet should be sent to 4G or ADSL. There is no rerouting done in FORWARD or after FORWARD (well, technically, rerouting can be done after POSTROUTING in severe cases, but back to the point).

Now for your routing: Remember that Ubuntu enables reverse path filtering by default. Reverse path filtering works as follow: When the kernel receives a packet (may it be forwarded or not) from an interface A, it will invert the source address and the destination address, and check if the resulting packet should be routed through interface A. If it isn't, the packet is dropped as a address spoofing attempt.

For packets received from eth0, this is not a problem. For packets received from eth1, this is also not a problem, because when reversing the source IP address and the destination IP address, the kernel will it the default route in table main. For packets received from eth2, which you do not mark, this is a problem, because the kernel will hit the default route in table main, and consider that these packets should have been received from eth1. The easiest solution is to disable reverse path filtering on eth1:

sysctl -w net.ipv4.conf.eth1.rp_filter=0