Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Short story,
3 interfaces, eth0 (LAN), eth1(ADSL), eth2(4G).
eth0 -> eth1: Works
(ports 80, 443, 4070) eth0 -> eth2: Doesn't happen


This is a graphical representation of the idea:

Port 80 & 443 via eth2
ant the rest via eth1
enter image description here

Netscheme:

eth0: -ip 10.0.0.1 -net 10.0.0.0/8 -gw 10.0.0.1 (the servers own intf) 
eth1: -ip 192.168.1.74 -net 192.168.1.0/24 -gw 192.168.1.254 
eth2: -ip 192.168.1.91 -net 192.168.0.0/24 -gw 192.168.0.1 






This new script reroutes 22 and 4070 to the proper table, i think..
however after getting to that table it doesn't get rerouted to eth2.




This script works, except for 22 and 4070!

(Port 80 is un-commented and it works but via eth1 which is wrong.)

modprobe iptable_nat
modprobe ip_conntrack

echo "1" > /proc/sys/net/ipv4/ip_forward

iptables -P INPUT ACCEPT
iptables -F INPUT
iptables -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -F PREROUTING
iptables -t nat -F
iptables -t mangle -F
iptables -F
# This next line, restores any issues trying to connect to something
# if you get weird ACK packets when trying to connect (at least i did)!
iptables -t mangle -A PREROUTING -p tcp -j CONNMARK --restore-mark
ip route flush table main

iptables -A PREROUTING -i eth0 -t mangle -p tcp --dport 22 -j MARK --set-mark 1
###  iptables -A PREROUTING -i eth0 -t mangle -p tcp --dport 80 -j MARK --set-mark 1
iptables -A PREROUTING -i eth0 -t mangle -p tcp --dport 4070 -j MARK --set-mark 1

## Setup routes
# LAN
route add -net 10.0.0.0 netmask 255.0.0.0 dev eth0
# ADSL
route add -net 192.168.1.0 netmask 255.255.255.0 dev eth1
# 4G (Only accessible if marking packages with \x01
route add -net 192.168.0.0 netmask 255.255.255.0 dev eth2
# Default via ADSL
## -- Does the same as ip route below? route add default gw 192.168.1.254


echo "201 eth2.out" >> /etc/iproute2/rt_tables

ip rule add fwmark 1 table eth2.out
ip route add default via 192.168.0.1 dev eth2 table eth2.out
ip route add default via 192.168.1.254 dev eth1



## Setup forwards
# From 4G to LAN
iptables -A FORWARD -i eth2 -o eth0 -m state --state ESTABLISHED,RELATED,NEW -j ACCEPT
# From ADSL to LAN
iptables -A FORWARD -i eth1 -o eth0 -m state --state ESTABLISHED,RELATED,NEW -j ACCEPT
# From LAN to ADSL (Default route out)
# - Note: If marked packages is sent to ADSL they will be mangled and rerouted to 4G
iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE









Old script:


  Ignore everything below unless you're interested in retracing my steps!!




I've created a router.sh script to setup my environment in case i do something bad. I've got 3 ports which i want to send to a 4G connection and the rest via a landline ADSL connection. To do this, I've instructed iptables to mangle packages on the default route and send them via my 4G interface if the --dport == 443 | 80 | 4070

However this doesn't work, i'm still getting routed through my landline no matter what.

This is what my script looks like:

#!/bin/bash

## routing tables
# wireless = 4G via eth2
# adsl = adsl via eth1

modprobe iptable_nat
modprobe ip_conntrack

echo "1" > /proc/sys/net/ipv4/ip_forward

iptables -P INPUT ACCEPT
iptables -F INPUT
iptables -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -t nat -F
ip route flush table main
ip route flush table wireless
ip route flush table adsl

## Setup routing tables
# ADSL
ip route add table adsl to 192.168.1.0/24 dev eth1
# 4G
ip route add table wireless to 192.168.0.0 dev eth2
ip rule add fwmark 0x1 table wireless

## Setup routes
# LAN
route add -net 10.0.0.0 netmask 255.0.0.0 dev eth0
# ADSL
route add -net 192.168.1.0 netmask 255.255.255.0 dev eth1
# 4G (Only accessible if marking packages with \x01
route add -net 192.168.0.0 netmask 255.255.255.0 dev eth2
# Default via ADSL
route add default gw 192.168.1.254


## Forward ports into the LAN
iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j DNAT --to 10.0.0.3:80


## Lets mark all packets we want for 4G forward
# HTTPS
iptables -A OUTPUT -t mangle -o eth1 -p tcp --dport 443 -j MARK --set-mark 1
# HTTP
iptables -A OUTPUT -t mangle -o eth1 -p tcp --dport 80 -j MARK --set-mark 1
# Spotify
iptables -A OUTPUT -t mangle -o eth1 -p tcp --dport 4070 -j MARK --set-mark 1

## Setup forwards
# From 4G to LAN
iptables -A FORWARD -i eth2 -o eth0 -m state --state ESTABLISHED,RELATED,NEW -j ACCEPT
# From ADSL to LAN
iptables -A FORWARD -i eth1 -o eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
# From LAN to ADSL (Default route out)
# - Note: If marked packages is sent to ADSL they will be mangled and rerouted to 4G
iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT
iptables -A FORWARD -j LOG
#iptables --table nat --append POSTROUTING --out-interface eth2 --jump SNAT --to-source "192.168.1.74"
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE

I've also tried to add these 3 to the bottomg of the script:

iptables -t nat -A POSTROUTING -o eth2 -p tcp --dport 80 -j SNAT --to "192.168.0.91"
iptables -t nat -A POSTROUTING -o eth2 -p tcp --dport 443 -j SNAT --to "192.168.0.91"
iptables -t nat -A POSTROUTING -o eth2 -p tcp --dport 4070 -j SNAT --to "192.168.0.91"

Also tried without success:

iptables -A PREROUTING -t mangle -i eth0 -p tcp --dport 80 -j MARK --set-mark 1

Last but not least, tried:

## Lets mark all packets we want for 4G forward
# HTTPS
iptables -A POSTROUTING -t mangle -o eth1 -p tcp --dport 443 -j MARK --set-mark 1
# HTTP
iptables -A POSTROUTING -t mangle -o eth1 -p tcp --dport 80 -j MARK --set-mark 1
# Spotify
iptables -A POSTROUTING -t mangle -o eth1 -p tcp --dport 4070 -j MARK --set-mark 1

The routing works, i can browse the web, listen to music and what not but i'm doing it through the wrong interface. I've Googled around for a long while now and found bits and peaces to understand what i'm doing and why i'm doing it. I could do traffic shaping via tc but if it's possible via marking packages in iptables it would help me a long way.

My guess is that i'm doing the order wrong on the different rules, mainly the MASQUERADE part? or if that should even be there?

Also, if someone could explain how to DNAT port say, tcp:80 from an external interface (either one or BOTH protocols) to a internal 10.0.0.0 address space, that would be lovely!

I wouldn't post this here normally but i'm in a real hurry to get this up and running or i'm in a big pickle!



Outputs:

root@Netbridge:~# route -n Kernel IP routing table Destination    

Gateway         Genmask         Flags Metric Ref    Use Iface<br>
0.0.0.0         192.168.1.254   0.0.0.0         UG    0      0        0 eth1<br>
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 eth0<br>
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth2<br>
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1


root@Netbridge:~# ifconfig

eth0      Link encap:Ethernet  HWaddr 00:0c:29:7e:9e:4e  
          inet addr:10.0.0.1  Bcast:10.255.255.255  Mask:255.0.0.0

eth1      Link encap:Ethernet  HWaddr 00:0c:29:7e:9e:58  
          inet addr:192.168.1.74  Bcast:192.168.1.255  Mask:255.255.255.0

eth2      Link encap:Ethernet  HWaddr 00:0c:29:7e:9e:62  
          inet addr:192.168.0.91  Bcast:192.168.0.255  Mask:255.255.255.0

Followed these instructions:
output-traffic-on-different-interfaces-based-on-destination-por
iptables-forward-specific-port-to-specific-nic
Among a few other related threads.

share|improve this question
    
I'm sorry, but is there a reason to don't bind apache only to the localhost and to the address of eth2 interface? –  Jan Marek Dec 27 '12 at 21:00
    
@JanMarek: There is no mention of apache here. And for your information, binding a socket on an ethX's ip address will prevent hosts on the ethX lan to access the local server using ethY's ip address, and will not prevent hosts from the ethY to access the server using ethX's ip address. Remember that Linux uses a weak host model. –  BatchyX Dec 27 '12 at 21:07
    
OK, there is possible more web servers, not only apache server... I though, that bind web server on the interface eth2 can be a simple solution. But I can be wrong, I know. –  Jan Marek Dec 27 '12 at 21:22
add comment

2 Answers 2

up vote 4 down vote accepted
+50

BatchyX already give some very good explanation about iptables and routing, so I will exercise my laziness and go directly to script.

It should NAT all traffic to port 80,443,22,4070 through 192.168.0.91. All the rest will NAT through 192.168.1.254.

I re-do my testing and end up following this guide. What is missing in that guide is the last 3 lines in my script. Which I found out from another port, but I lost track of that link.

It is a tested working script.

Need Default Route

One thing I did not put in the script is setting up the default route. It should be

route add default gw 192.168.1.254

When you do route -n, it should be the only default route (Dest:0.0.0.0)

0.0.0.0    192.168.1.254    0.0.0.0    UG    0    0    0    eth1

fw-router.sh

# Reset/Flush iptables
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

#Reset/Flush/Setup IP Route (table 4)
ip route flush table 4
ip route show table main | grep -Ev ^default | while read ROUTE ; do ip route add table 4 $ROUTE ; done
ip route add table 4 default via 192.168.0.1

#Mark Packet with matching D.Port
iptables -t mangle -A PREROUTING -p tcp --dport 22   -s 10.0.0.0/24 -j MARK --set-mark 4
iptables -t mangle -A PREROUTING -p tcp --dport 80   -s 10.0.0.0/24 -j MARK --set-mark 4
iptables -t mangle -A PREROUTING -p tcp --dport 443  -s 10.0.0.0/24 -j MARK --set-mark 4
iptables -t mangle -A PREROUTING -p tcp --dport 4070 -s 10.0.0.0/24 -j MARK --set-mark 4

#SNAT Rules
iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 192.168.1.74
iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 192.168.0.91

#IP Route
ip rule add fwmark 4 table 4
ip route flush cache

#IP Stack
#This is the missing part from the guide
echo 1 > /proc/sys/net/ipv4/ip_forward
for f in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 0 > $f ; done
echo 0 > /proc/sys/net/ipv4/route/flush

PS1: In short, MASQUERADE does not work (in most case, and definitely in your case) for NAT with multiple external IPs that need some kind of load balancing or need DNAT to handle incoming traffic. You need SNAT for direction control.

PS2: Pure iptables is not sufficient.

share|improve this answer
    
First of all, cheers for going directly to script :) I'll try this tonight as i get back to the environment in which this will run at! –  Torxed Dec 20 '12 at 8:31
    
hmm, may need heavy revise. Let me know result. –  John Siu Dec 20 '12 at 15:46
    
Revised, should be working now. –  John Siu Dec 20 '12 at 19:20
    
Sweet, will get back to you in a short bit :D –  Torxed Dec 21 '12 at 11:24
    
I love you, so much headache and you solved it! Thank you! –  Torxed Dec 21 '12 at 14:45
show 2 more comments

Note: I only have considered the first script, ignoring the old one.

  • You don't need to modprobe netfilter modules by hand with current iptables. This is only necessary for custom connection trackers.
  • Don't mix up route and ip route. This is pure evil. Just use ip everywhere and forget about ifconfig and route
  • /etc/iproute2/rt_tables is not reset across reboots. Appending the same entry over and over is not a good idea, you only need to do it once. Remember that rt_tables just define name aliases to numeric values, it does not change any configuration.

  • Now for iptables: In your FORWARD chain, you drop packets coming from LAN to 4G. This is bad. the FORWARD hook is used after routing is done. At this point, all policy routing is done, and it is already known whether the packet should be sent to 4G or ADSL. There is no rerouting done in FORWARD or after FORWARD (well, technically, rerouting can be done after POSTROUTING in severe cases, but back to the point).

Now for your routing: Remember that Ubuntu enables reverse path filtering by default. Reverse path filtering works as follow: When the kernel receives a packet (may it be forwarded or not) from an interface A, it will invert the source address and the destination address, and check if the resulting packet should be routed through interface A. If it isn't, the packet is dropped as a address spoofing attempt.

For packets received from eth0, this is not a problem. For packets received from eth1, this is also not a problem, because when reversing the source IP address and the destination IP address, the kernel will it the default route in table main. For packets received from eth2, which you do not mark, this is a problem, because the kernel will hit the default route in table main, and consider that these packets should have been received from eth1. The easiest solution is to disable reverse path filtering on eth1:

sysctl -w net.ipv4.conf.eth1.rp_filter=0
share|improve this answer
    
All great comments and yes, there are some faulty logic on my part, for instance appending to rt_tables each time and especially not clearing certain tables as well as i should but i'll get there :) I'll give everything a go and see if it works, if so, +50 to you and most importantly you've saved a mans behind from being chopped into bits and peaces! :) –  Torxed Dec 18 '12 at 20:40
    
Below script works but you sir deserve all the gratitude as well, you gave me a thing or two to think about and rp_filter=0 helped a long way! :D –  Torxed Dec 21 '12 at 14:47
    
@BatchyX very informative, up vote to you too. –  John Siu Dec 22 '12 at 3:18
    
@Torxed The post I saw mention rp_filter for ALL interface need to be disabled. I am not sure if it works other way. –  John Siu Dec 22 '12 at 3:20
    
@JohnSiu: You just need to disable rp filter on interfaces where it is problematic. In that case, it's eth1, because there is no route going to that interface when marks are not set. eth2 has no such problem, as the default route goes into that interface. For eth0, it is actually useful to keep rp filter enabled, otherwise, someone on the LAN could forge an IP packet with a source address being on the internet and a destination address on the LAN, and the router would happily accept it, and forward it back to ... eth0. –  BatchyX Dec 22 '12 at 9:24
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.