Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Background: I haven't done anything with iptables in a few years... I have Fedora 16 running in a VM on VMWare, with my firewall (TomatoUSB) port forwarding to the VM.

VM is at 192.168.1.155. I know that the packets are making it to the VM...

Based on this illustration to see how the packets are supposed to go, I would expect the packets to come out of nat-PREROUTING and either go to mangle-INPUT or mangle-FORWARD unless the kernel is dropping them for some other reason.

So I turned on some logging:

iptables -t mangle -v -A PREROUTING -j LOG -p tcp --destination-port 80 --log-prefix 'mangle-PREROUTING '
iptables -t nat -v -A PREROUTING -j LOG -p tcp --destination-port 80 --log-prefix 'nat-PREROUTING '
iptables -t filter -v -I INPUT 1 -j LOG -p tcp --destination-port 80 --log-prefix 'filter-INPUT '
iptables -t filter -v -I FORWARD 1 -j LOG -p tcp --destination-port 80 --log-prefix 'filter-FORWARD '
iptables -t mangle -v -I INPUT 1 -j LOG -p tcp --destination-port 80 --log-prefix 'mangle-INPUT ' 
iptables -t mangle -v -I FORWARD 1 -j LOG -p tcp --destination-port 80 --log-prefix 'mangle-FORWARD '

and then I used an external test service and I can see the packets coming in past the PREROUTING chains, but then being ignored:

Apr 23 19:11:52 webmail64 kernel: [  351.116042] mangle-PREROUTING IN=eth1 OUT= MAC=00:0c:29:fa:36:c7:bc:ae:c5:c3:68:f9:08:00 SRC=66.249.67.195 DST=192.168.1.155 LEN=60 TOS=0x00 PREC=0x20 TTL=48 ID=20466 DF PROTO=TCP SPT=64135 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 
Apr 23 19:11:52 webmail64 kernel: [  351.121701] nat-PREROUTING IN=eth1 OUT= MAC=00:0c:29:fa:36:c7:bc:ae:c5:c3:68:f9:08:00 SRC=66.249.67.195 DST=192.168.1.155 LEN=60 TOS=0x00 PREC=0x20 TTL=48 ID=20466 DF PROTO=TCP SPT=64135 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 
Apr 23 19:11:55 webmail64 kernel: [  354.113372] mangle-PREROUTING IN=eth1 OUT= MAC=00:0c:29:fa:36:c7:bc:ae:c5:c3:68:f9:08:00 SRC=66.249.67.195 DST=192.168.1.155 LEN=60 TOS=0x00 PREC=0x20 TTL=48 ID=20467 DF PROTO=TCP SPT=64135 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 
Apr 23 19:11:55 webmail64 kernel: [  354.114834] nat-PREROUTING IN=eth1 OUT= MAC=00:0c:29:fa:36:c7:bc:ae:c5:c3:68:f9:08:00 SRC=66.249.67.195 DST=192.168.1.155 LEN=60 TOS=0x00 PREC=0x20 TTL=48 ID=20467 DF PROTO=TCP SPT=64135 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 
Apr 23 19:12:01 webmail64 kernel: [  360.109534] mangle-PREROUTING IN=eth1 OUT= MAC=00:0c:29:fa:36:c7:bc:ae:c5:c3:68:f9:08:00 SRC=66.249.67.195 DST=192.168.1.155 LEN=60 TOS=0x00 PREC=0x20 TTL=48 ID=20468 DF PROTO=TCP SPT=64135 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 
Apr 23 19:12:01 webmail64 kernel: [  360.111023] nat-PREROUTING IN=eth1 OUT= MAC=00:0c:29:fa:36:c7:bc:ae:c5:c3:68:f9:08:00 SRC=66.249.67.195 DST=192.168.1.155 LEN=60 TOS=0x00 PREC=0x20 TTL=48 ID=20468 DF PROTO=TCP SPT=64135 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 

You can see that TTL is fine. The VM's IP is 192.168.1.155 so it should go to INPUT next but it never makes it. If the packet came from inside my network, it is as expected:

Apr 23 19:20:03 webmail64 kernel: [  841.725402] mangle-PREROUTING IN=eth1 OUT= MAC=00:0c:29:fa:36:c7:00:1f:3b:cb:2e:99:08:00 SRC=192.168.1.69 DST=192.168.1.155 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=4562 DF PROTO=TCP SPT=61520 DPT=80 WINDOW=4042 RES=0x00 ACK FIN URGP=0 
Apr 23 19:20:03 webmail64 kernel: [  841.729647] mangle-INPUT IN=eth1 OUT= MAC=00:0c:29:fa:36:c7:00:1f:3b:cb:2e:99:08:00 SRC=192.168.1.69 DST=192.168.1.155 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=4562 DF PROTO=TCP SPT=61520 DPT=80 WINDOW=4042 RES=0x00 ACK FIN URGP=0 
Apr 23 19:20:03 webmail64 kernel: [  841.731056] filter-INPUT IN=eth1 OUT= MAC=00:0c:29:fa:36:c7:00:1f:3b:cb:2e:99:08:00 SRC=192.168.1.69 DST=192.168.1.155 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=4562 DF PROTO=TCP SPT=61520 DPT=80 WINDOW=4042 RES=0x00 ACK FIN URGP=0 
Apr 23 19:20:03 webmail64 kernel: [  841.732784] mangle-PREROUTING IN=eth1 OUT= MAC=00:0c:29:fa:36:c7:00:1f:3b:cb:2e:99:08:00 SRC=192.168.1.69 DST=192.168.1.155 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=4563 DF PROTO=TCP SPT=61520 DPT=80 WINDOW=4042 RES=0x00 ACK URGP=0 
Apr 23 19:20:03 webmail64 kernel: [  841.734257] mangle-INPUT IN=eth1 OUT= MAC=00:0c:29:fa:36:c7:00:1f:3b:cb:2e:99:08:00 SRC=192.168.1.69 DST=192.168.1.155 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=4563 DF PROTO=TCP SPT=61520 DPT=80 WINDOW=4042 RES=0x00 ACK URGP=0 
Apr 23 19:20:03 webmail64 kernel: [  841.735676] filter-INPUT IN=eth1 OUT= MAC=00:0c:29:fa:36:c7:00:1f:3b:cb:2e:99:08:00 SRC=192.168.1.69 DST=192.168.1.155 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=4563 DF PROTO=TCP SPT=61520 DPT=80 WINDOW=4042 RES=0x00 ACK URGP=0 

What have I tried?

  • Turned off SELinux
  • Totally turned off iptables
  • Ensured that the default policies are ACCEPT
    • Saw that the packet counts for the ACCEPT incremented
  • Turned on IP forwarding (/proc/sys/net/ipv4/ip_forward) just in case

My setup: * kernel = Linux webmail64 3.3.2-1.fc16.x86_64 #1 SMP Sat Apr 14 00:31:23 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux * iptables v1.4.12

Here are all the iptables in case it matters:

[root@webmail64 ~]# iptables-save 
# Generated by iptables-save v1.4.12 on Mon Apr 23 20:47:24 2012
*nat
:PREROUTING ACCEPT [916:127527]
:INPUT ACCEPT [1:60]
:OUTPUT ACCEPT [87:7857]
:POSTROUTING ACCEPT [87:7857]
-A PREROUTING -p tcp -m tcp --dport 80 -j LOG --log-prefix "nat-PREROUTING "
COMMIT
# Completed on Mon Apr 23 20:47:24 2012
# Generated by iptables-save v1.4.12 on Mon Apr 23 20:47:24 2012
*mangle
:PREROUTING ACCEPT [1402:193108]
:INPUT ACCEPT [1343:189856]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [303:67789]
:POSTROUTING ACCEPT [303:67789]
-A PREROUTING -p tcp -m tcp --dport 80 -j LOG --log-prefix "mangle-PREROUTING "
-A INPUT -p tcp -m tcp --dport 80 -j LOG --log-prefix "mangle-INPUT "
-A FORWARD -p tcp -m tcp --dport 80 -j LOG --log-prefix "mangle-FORWARD "
COMMIT
# Completed on Mon Apr 23 20:47:24 2012
# Generated by iptables-save v1.4.12 on Mon Apr 23 20:47:24 2012
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1075:220262]
-A INPUT -p tcp -m tcp --dport 80 -j LOG --log-prefix "filter-INPUT "
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -p tcp -m tcp --dport 80 -j LOG --log-prefix "filter-FORWARD "
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Mon Apr 23 20:47:24 2012

Where can I look next?

Update

I was asked to run tcpdump and it looks like I am never sending ACK packets?:

tcpdump -i eth1 -An -vvv \(net 50 or net 173\)
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
19:31:26.305048 IP (tos 0x20, ttl 53, id 26094, offset 0, flags [DF], proto TCP (6), length 60)
    50.22.90.226.48891 > 192.168.1.155.http: Flags [S], cksum 0xca12 (correct), seq 2918539684, win 5840, options [mss 1460,sackOK,TS val 1152517194 ecr 0,nop,wscale 7], length 0
E .<e.@.5..r2.Z........P..a....................
D..J........
19:31:26.521815 IP (tos 0x20, ttl 53, id 61033, offset 0, flags [DF], proto TCP (6), length 60)
    50.22.90.226.48892 > 192.168.1.155.http: Flags [S], cksum 0x82b4 (correct), seq 1826089481, win 5840, options [mss 1460,sackOK,TS val 1152517216 ecr 0,nop,wscale 7], length 0
E .<.i@.5...2.Z........Pl..     ...................
D..`........
19:31:29.300994 IP (tos 0x20, ttl 53, id 26095, offset 0, flags [DF], proto TCP (6), length 60)
    50.22.90.226.48891 > 192.168.1.155.http: Flags [S], cksum 0xc8e6 (correct), seq 2918539684, win 5840, options [mss 1460,sackOK,TS val 1152517494 ecr 0,nop,wscale 7], length 0
E .<e.@.5..q2.Z........P..a....................
D..v........
19:31:29.521214 IP (tos 0x20, ttl 53, id 61034, offset 0, flags [DF], proto TCP (6), length 60)
    50.22.90.226.48892 > 192.168.1.155.http: Flags [S], cksum 0x8188 (correct), seq 1826089481, win 5840, options [mss 1460,sackOK,TS val 1152517516 ecr 0,nop,wscale 7], length 0
E .<.j@.5...2.Z........Pl..     ...................
D...........
19:31:35.302578 IP (tos 0x20, ttl 53, id 26096, offset 0, flags [DF], proto TCP (6), length 60)
    50.22.90.226.48891 > 192.168.1.155.http: Flags [S], cksum 0xc68e (correct), seq 2918539684, win 5840, options [mss 1460,sackOK,TS val 1152518094 ecr 0,nop,wscale 7], length 0
E .<e.@.5..p2.Z........P..a....................
D...........
19:31:35.532347 IP (tos 0x20, ttl 53, id 61035, offset 0, flags [DF], proto TCP (6), length 60)
    50.22.90.226.48892 > 192.168.1.155.http: Flags [S], cksum 0x7f2f (correct), seq 1826089481, win 5840, options [mss 1460,sackOK,TS val 1152518117 ecr 0,nop,wscale 7], length 0
E .<.k@.5...2.Z........Pl..     ........./.........
D...........
share|improve this question

2 Answers 2

I know this is a bit late, however ... Does the host know it is responsible that IP? Here's a schema I put at the top of the rules files I create. You can find similar flow-charts elsewhere, but having it in ASCII (wouldn't want to call it art ;)) can be quite useful on the terminal.

For the most part I know this from heart, but hey, doesn't hurt to have a reference if forgetfulness strikes.

###############################################################################
###
###            PACKET FLOW THROUGH NETFILTER TABLES AND CHAINS
###
###
###                          {Packet in}
###                               |
###                               v
###                       +-----------------+
###                       |mangle/PREROUTING|
###                       +-----------------+
###                               |
###                               v
###                       +-----------------+
###                       |  nat/PREROUTING |
###                       +-----------------+
###                               |
###                               v
###                       *~~~~~~~~~~~~~~~~~*
###                       |  kernel routing |
###                       *~~~~~~~~~~~~~~~~~*
###                               |
###                               v
###           .------------{?for this host?}------------.
###      yes! |                                         | no!
###           v                                         v
###  +-----------------+                       +-----------------+
###  |   mangle/INPUT  |                       |  mangle/FORWARD |
###  +-----------------+                       +-----------------+
###           |                                         |
###           v                                         v
###  +-----------------+                       +-----------------+
###  |   filter/INPUT  |                       |  filter/FORWARD |
###  +-----------------+                       +-----------------+
###           |                                         |
###           v                                         |
### *~~~~~~~~~~~~~~~~~~~~*                              |
### | response & routing |                              |
### *~~~~~~~~~~~~~~~~~~~~*                              |
###           |                                         |
###           v                                         |
###  +-----------------+                                |
###  |  mangle/OUTPUT  |                                |
###  +-----------------+                                |
###           |                                         |
###           v                                         |
###  +-----------------+                                |
###  |    nat/OUTPUT   |                                |
###  +-----------------+                                |
###           |                                         |
###           v                                         |
###  +-----------------+                                |
###  |  filter/OUTPUT  |                                |
###  +-----------------+                                |
###           |                                         |
###           .-------------------+---------------------.
###                               |
###                               v
###                      +------------------+
###                      |mangle/POSTROUTING|
###                      +------------------+
###                               |
###                               v
###                      +------------------+
###                      |  nat/POSTROUTING |
###                      +------------------+
###                               |
###                               v
###                          {Packet out}
###
###############################################################################

Now what does that mean?

Routing can happen either using the routing table ("kernel routing" in the above flowchart) or using netfilter. Now if - and that's the most likely scenario in your case - you didn't set up the routing table accordingly, the kernel would not know where the packet has to go and eventually drop it. Btw: in such cases it's useful to create a custom chain that will LOG and then DROP or add the rules in that order. This way you'll see what rules were hit. Also useful is iptables-save -c which will prepend the packet and byte counters to each rule line similar to how it appends it for the chains (format [packets:bytes]).

For ports forwarded to a VM via DNAT I have the following recipe (will explain below):

#!/bin/bash
VMNET=192.168.1.0/24
MAINIP=66.249.67.195
CONTIP=192.168.1.2
VMPORT=80
INPORT=80
ACTION="-I"
iptables -t nat    $ACTION PREROUTING  -d $MAINIP -p tcp --dport $INPORT -j DNAT --to-destination $CONTIP:$VMPORT
iptables -t nat    $ACTION POSTROUTING -s $VMNET ! -d $VMNET -p tcp -j MASQUERADE --to-ports 1024-65535
iptables -t nat    $ACTION POSTROUTING -s $VMNET ! -d $VMNET -p udp -j MASQUERADE --to-ports 1024-65535
iptables -t nat    $ACTION POSTROUTING -s $VMNET ! -d $VMNET -j MASQUERADE
iptables -t filter $ACTION INPUT       -p tcp -d $MAINIP --dport $INPORT -j ACCEPT
iptables -t filter $ACTION FORWARD     -p tcp --dport $INPORT -d $VMNET -j ACCEPT
iptables -t filter $ACTION FORWARD     -p tcp --dport $INPORT -d $MAINIP -j ACCEPT

Mind that you may want to alter the order to fit the rules with your own. Also, if the OUTPUT chain doesn't have ACCEPT as its default policy, make sure that you add an output rule, although for all practical purposes this should be catered by having an RELATED,ESTABLISHED state rule. You can also refine the interfaces to match or match by wildcard. For example I gave the bridges of my virtual guests all the prefix _ (an underscore) and can therefore match -i _+ and -o _+. Similarly for several network cards (eth0, eth1) you'd match with -i eth+.

So what happens here is that:

  1. a DNAT rule is inserted which takes input on (host) port $INPORT for TCP and "routes" it to $CONTIP:$VMPORT, i.e. the IP of the container and the port in the container. Yep, they could differ. If they don't it's alright to leave out the destination part (i.e. just ``$CONTIP`).
  2. three rules to mask traffic that's from the subnet of the virtual guests but not going to another virtual guest.
  3. an INPUT rule to allow incoming packets on the public IP (no interface given, but could be!) to port $INPORT through. I think this rule isn't strictly needed, at least not if you tie it to the public interface.
  4. a rule to forward traffic to $INPORT for the virtual guest subnet ($VMNET)
  5. a rule to forward traffic to $INPORT on the public IP ($MAINIP)

Last but not least the value for sysctl (/proc/sys/net/ipv4/ip_forward) should be:

# cat /proc/sys/net/ipv4/ip_forward
1

to make your host a router.

If it's not use echo 1 to write to the above "file" in the procfs or better yet use sysctl -w net.ipv4.ip_forward=1 as root.

share|improve this answer

I still don't know why but I got it to work.

I had two interfaces, eth0 (192.168.99.x) and eth1 (192.168.1.x). They were there for legacy reasons, I was lazy when I wanted a new VM and copied my previous one. Anyway, I simply disabled eth0 and everything worked.

I checked some of the rp_filter settings in /etc/sysctl.conf but that didn't fix it (but made me think it was a weird IF issue). The settings are documented in /usr/share/doc/kernel-doc-x.x.x/Documentation/networking/ip-sysctl.txt in the kernel-doc package.

So it works now, and I will leave this here for the search engines to find and maybe this will help somebody else some day.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.