Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have an issue with private network traffic not being masqueraded in very specific circumstances.

The network is a group of VMware guests using the 10.1.0.0/18 network.

The problematic host is 10.1.4.20 255.255.192.0 and the only gateway it is configured to use is 10.1.63.254. The gateway server $vpnGatewayAddress should be masquerading all outbound traffic and forwarding it through $outerGatewayAddress, but for some reason, 10.1.4.20 ends up occasionally having $outerGatewayAddress in its routing cache, as you can see here :

ip -s route show cache 199.16.156.40
199.16.156.40 from 10.1.4.20 via $outerGatewayAddress dev eth0
    cache  used 149 age 17sec ipid 0x9e49
199.16.156.40 via $outerGatewayAddress dev eth0  src 10.1.4.20
    cache  used 119 age 11sec ipid 0x9e49

The problem occurs only with api.twitter.com

After flushing the cache, I am able to ping it again, and the cache looks sane :

ip route flush cache 199.16.156.40

ping api.twitter.com
PING api.twitter.com (199.16.156.40) 56(84) bytes of data.
64 bytes from 199.16.156.40: icmp_req=1 ttl=247 time=93.4 ms

ip -s route show cache 199.16.156.40
199.16.156.40 from 10.1.4.20 via 10.1.63.254 dev eth0
    cache  age 3sec
199.16.156.40 via 10.1.63.254 dev eth0  src 10.1.4.20
    cache  used 2 age 2sec

Network information for the app server (without lo) :

ip a

eth0      Link encap:Ethernet  HWaddr 00:50:56:a4:48:20
          inet addr:10.1.4.20  Bcast:10.1.63.255  Mask:255.255.192.0
          inet6 addr: fe80::250:56ff:fea4:4820/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1523222895 errors:0 dropped:407 overruns:0 frame:0
          TX packets:1444207934 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1524116772058 (1.5 TB)  TX bytes:565691877505 (565.6 GB)

Network information for the VPN gateway (without lo too) :

 eth0      Link encap:Ethernet  HWaddr 00:50:56:a4:56:e9
           inet addr:$vpnGatewayAddress  Bcast:$broadcastAddress  Mask:255.255.255.192
           inet6 addr: fe80::250:56ff:fea4:56e9/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:7030472688 errors:0 dropped:1802 overruns:0 frame:0
           TX packets:6959026084 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:7777330931859 (7.7 TB)  TX bytes:7482143729162 (7.4 TB)

 eth0:0    Link encap:Ethernet  HWaddr 00:50:56:a4:56:e9
           inet addr:10.1.63.254  Bcast:10.1.63.255  Mask:255.255.192.0
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

 eth0:1    Link encap:Ethernet  HWaddr 00:50:56:a4:56:e9
           inet addr:10.1.127.254  Bcast:10.1.127.255  Mask:255.255.192.0
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

 tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
           inet addr:10.8.1.1  P-t-P:10.8.1.2  Mask:255.255.255.255
           UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
           RX packets:477047415 errors:0 dropped:0 overruns:0 frame:0
           TX packets:833650386 errors:0 dropped:101834 overruns:0 carrier:0
           collisions:0 txqueuelen:100
           RX bytes:89948688258 (89.9 GB)  TX bytes:1050533566879 (1.0 TB)

eth0 leads to the outside world, and tun0 to an openvpn network of VMs on which sits the app server.

ip r for the VPN gateway :

default via $outerGatewayAddress dev eth0  metric 100
10.1.0.0/18 dev eth0  proto kernel  scope link  src 10.1.63.254
10.1.64.0/18 dev eth0  proto kernel  scope link  src 10.1.127.254
10.8.1.0/24 via 10.8.1.2 dev tun0
10.8.1.2 dev tun0  proto kernel  scope link  src 10.8.1.1
10.9.0.0/28 via 10.8.1.2 dev tun0
$addressEndingWithAZero/26 dev eth0  proto kernel  scope link  src $vpnGatewayAddress

ip r on the app server :

default via 10.1.63.254 dev eth0  metric 100
10.1.0.0/18 dev eth0  proto kernel  scope link  src 10.1.4.20

Firewall rules:

Chain PREROUTING (policy ACCEPT 380M packets, 400G bytes) 
pkts bytes target prot opt in out source destination 

Chain INPUT (policy ACCEPT 127M packets, 9401M bytes) 
pkts bytes target prot opt in out source destination 

Chain OUTPUT (policy ACCEPT 1876K packets, 137M bytes) 
pkts bytes target prot opt in out source destination 

Chain POSTROUTING (policy ACCEPT 223M packets, 389G bytes) 
pkts bytes target prot opt in out source destination 

32M 1921M MASQUERADE all -- * eth0 10.1.0.0/17 0.0.0.0/0

redirect configuration :

sysctl net.ipv4.conf.all.send_redirects
net.ipv4.conf.all.send_redirects = 1
sysctl net.ipv4.conf.eth0.send_redirects
net.ipv4.conf.eth0.send_redirects = 1
sysctl net.ipv4.conf.tun0.send_redirects
net.ipv4.conf.tun0.send_redirects = 1

Why does the masquerade fail sometimes ? How can I prevent that ?

share|improve this question
    
Just a quick question, why is the subnet mask in the PREROUTING rule a slash 17 when you have a slash 18 network? –  datUser Apr 14 at 12:14
    
I did not setup any of this and the person who did no longer works for us, so I don't know, sadly... –  greg0ire Apr 14 at 12:52
    
Is it possible that the VPN gateway is wrongly sending ICMP redirects? Check the net.ipv4.conf.all.send_redirects and net.ipv4.conf.eth0.send_redirects sysctl settings or alternatively the contents of the files /proc/sys/net/ipv4/conf/{all,eth0}/send_redirects. –  Michael Ehrenreich Apr 15 at 14:01
    
I updated my answer with the details about redirects... what does it mean ? –  greg0ire Apr 15 at 15:33
    
The net.ipv4.conf.*.send_redirects specify, whether ICMP redirects may be sent, overall, or on a specific interface. ICMP redirects may be used to tell a client to take a specific route. In this case, I suspect that the VPN gateway might be telling the app server to go via the VPN gateway's default gateway. You could try setting the two settings above to 0 in order to check whether that is the problem. This is also the only way I could imagine an IP two hops away ending up in the app servers routing table. –  Michael Ehrenreich Apr 15 at 16:02
show 7 more comments

1 Answer 1

up vote 1 down vote accepted
+200

As discussed in the comments, it seems like the problem was the VPN gateway wrongly sending ICMP redirects to the app server because setting the sysctl settings net.ipv4.conf.all.send_redirects and/or net.ipv4.conf.eth0.send_redirects to 0 appears to have solved the problem.

I don't know why the VPN gateway would tell the app server to go via the outer gateway which the app server can't possibly reach without going through the VPN server.

share|improve this answer
    
triple combo : +1 , accepted, bounty! Thanks a lot for your insight! –  greg0ire Apr 16 at 17:21
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.