Only allow certain outbound traffic on certain interfaces

Question

I have rather an odd issue. I have a server with two network interfaces eth0 and eth1. Each are connected to a different network. Each network has a internet gateway. The server has various outbound connections: http (some scripts on the server scrape websites), nfs client, samba client, dns client and an email fetcher to name but a few.

For reasons I won't go into, I need to split these outbound clients up so outbound http, nfs, samba and dns traffic is only requested over eth0 while everything else goes off through eth1.

I've read around a few Google searches and it looks like iptables is what I'll need but I really haven't got a clue. I'm only used to managing inbound firewall rules through ufw.

Could somebody start me off with a few example rules and tell me how to get the system to adopt these rules on boot? Ideally without locking me out of my SSH connection (I can get physical access, but I'd rather not).

Edit I can split the clients over two users if it's possible to limit all outbound traffic from an account to one interface. on paper that seems like it might be easier.

Answer 1

I would setup a separate routing table and a policy to route marked packets using that table and have iptables/netfilter mark certain packets.

Create a table: echo 1 known >> /etc/iproute2/rt_tables

Create a routing rule (the ip command is from iproute2): ip rule add from all fwmark 1 table known

We created a table called "known" and created a routing rule that says any packet with a mark equal to 1 gets routed according to the "known" table. I only called it known because it's for the list of known protocols - you can name it whatever you want. Now we setup the known table to route the proper way.

ip route add default dev eth0 table known

Create iptables rules:

iptables -t mangle -I PREROUTING -p tcp --dport 111 -j MARK --set-mark 1
iptables -t mangle -I PREROUTING -p tcp --dport 2049 -j MARK --set-mark 1

The example marks packets on the NFS ports (111, 2049) with a 1. We are adding this rule to the 'mangle' iptable. This is different from the routing tables and is not changeable; the mangle table is specifically for altering packets in any way other than NAT.

Now, to route everything else through the other interface, we add a route to the standard routing table.

ip route add default dev eth1

To really understand this, read sections 4 and 11 of the LARTC howto.

Answer 2

Well, the "easiest" way should be to instruct the single programs to use a specific interface (or the ip of the interface. For example:

ping -I eth1 8.8.8.8

instructs ping to use eth1 as a source interface, while

wget --bind-address 10.0.0.1 http://www.google.it/

instructs wget to go throught the interface which has 10.0.0.1 as an IP address.

Honestly I don't know if it's possible with all the programs you need, but it's a start to trim down the rules you need to write for iptables and the iproute programs.

As a start you should read this tutorial on multiple internet connections. A good read is also one of the thousands of iptables tutorials, expecially on outbound filtering, process/pid filtering and port filtering.

Answer 3

The proper way to to this is to bind() to the interface you want to use for outgoing packets. Since You can then set up routes with the ip route and ip rule commands to control how packets are routed based on their outgoing interface. For my example, I will assume the following network:

• eth0:
  • Address: 192.168.0.2/24
  • Default Gateway: 192.168.0.1
• eth1:
  • Address: 192.168.1.2/24
  • Default Gateway: 192.168.1.1

I will create two routing tables, one for traffic outgoing for eth0 called alternate and one table for eth1 called main. Routing table main always exists and is the normal table that is used by the route and ip route commands. Most people never deal with any other tables. To create the table called alternate, we will add the following line to /etc/iproute2/rt_tables:

10    alternate

The table main has a default priority of 254. The rules for which routing table is in effect is controlled by the ip rule command. By default, that command will print out a list of existing rules which should look something like this:

0:      from all lookup local 
32766:  from all lookup main 
32767:  from all lookup default 

This basically says it will look for a route in the table local which is a special table maintained by the kernel for local routes such as my own IP address. It will then try table main and table default. Table default is normally blank so if there's no match in main, there's no route to host. First, lets fill table alternate with rules for eth0.

sudo ip route add table alternate 192.168.0.0/24 dev eth0
sudo ip route add table alternate 192.168.1.0/24 dev eth1
sudo ip route add table alternate default via 192.168.0.1

You will normally want the alternate table to look similar to the main table. The only differences are when routing should be different. You may want not want to include the second line above if you literally want all NFS, HTTP, etc. traffic to go via the default gateway on eth0 even if it's destined for the network on eth1. Next step is to add a rule for when to use this alternate routing table:

sudo ip rule add from 192.168.0.0/24 pref 10 table alternate

This rule says any traffic coming from an address on the 192.168.0 network will use the alternate routing table instead of the normal main table. The last step is to make sure all clients that must use eth0 bind to it. With wget, for example, set --bind-address=192.168.0.2, for NFS set the clientaddr=192.168.0.2 mount option. If using LibWWW with Perl, you can set the localaddr option in LWP::UserAgent to control the local interface it binds to. If you happen to have a client you can't control the binding and compiling source is not an option, you might be able to use an iptables rule to modify it's address, but this is more of a hack and may not work. You would need a SNAT rule set up in the PREROUTING chain of either the nat table or the mangle table. You will still need the modified routing tables given above for this to work.