Linux workstations intermittently lose internet connections

Question

About three months ago, a strange phenomenon started happening in our network.

Firstly, let me describe the network layout (only the details which I think I'll be allowed to divulge):

• We have a router which establishes a VPN connection over the internet to our client's network.
• Checkpoint is used for the VPN connection
• I don't know the exact model of the router. But our IT admins refer to it as the "Nokia firewall"
• Workstations set their gateway to the router's IP address and that gives them both internet and VPN access.

So there's the problem:

About four months ago, Linux workstations intermittently lose internet and VPN connections. This does not happen to Windows workstations. The problem appears to only plague the ones running on Linux.

We can see that packets gets sent to the gateway but no response is received. This lasts for about 3-5mins and then internet and VPN connectivity is reestablished. We fail to see any pattern on the occurrence. The said problem reoccurs at intervals ranging from a few seconds up to a few hours.

To rule-out the Linux workstations as being the problem, we tried switching a couple of them to a different gateway. A plain router (let's call it GateWay-B). The problem does not occur when a different gateway is used.

One thing I did was to setup the IP tables so that:

• All packets bound to workstations over the VPN, use the "Nokia/checkpoint" gateway
• All other packets use another gateway; GateWay-B.

With this, I no longer loose internet connection. However, the VPN connection remains intermittent.

Question:

Does anyone have an idea what the problem is? Is it possible for a gateway to determine the OS where an incoming packet came from?

Answer 1

This sounds like potentially some arp cache confusion.

One possibility is if the "Nokia firewall" is part of a High Availability (HA) pair, there could be some failover or load balancing events occurring. If there is an HA pair and one of them becomes the active firewall, the linux workstation may continue to send requests to the wrong firewall due to the incorrect arp cache entry.

You can easily test this next time you lose connectivity to the VPN site. Make sure the linux workstation has the iproute package installed. Execute ip neigh flush dev eth0 (substituting the correct interface). This will temporarily clear the arp cache until it repopulates, potentially with the hardware address of the firewall that is correctly forwarding traffic.

If you can discern which hardware address is forwarding traffic correctly, you can add that as a static arp mapping (though this could potentially break any HA or load balancing performed by the firewalls).

Ultimately, this should be pointed out to the group responsible for maintaining and configuring the firewalls so it can be resolved.