Kernel routing issue on Ubuntu 14.04 virtual machine

Question

We are having some weird network behaviors on some Linux VMs (cloud based, multiple providers, mostly Ubuntu 14.04 and 16.04). We have two distinct networks with a Strongswan gateway in-between.

Site A: Network - 10.104.16.0/20 VPN gateway and routing configured on the main router (no conf needed on virtual machines)

Site B: Network - 10.240.132.0/25 Strongswan gateway - 10.240.132.15 Routing configured per VM depending on the need (or not) to communicate with Site A

Kernel routing table on one of the VMs on Site B which need to communicate with Site A VMs:

# route -vn
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.240.132.1    0.0.0.0         UG    0      0        0 eth0
10.104.16.0     10.240.132.15   255.255.240.0   UG    0      0        0 eth0
10.240.132.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0

And now, the issue... When everything works good, the VM does ping VMs on Site A and this is what the traceroute command outputs:

# traceroute 10.104.19.4
traceroute to 10.104.19.4 (10.104.19.4), 30 hops max, 60 byte packets
 1  10.240.132.15 (10.240.132.15)  0.248 ms  0.228 ms  0.220 ms
 2  * * *
 3  10.104.19.4 (10.104.19.4)  15.048 ms  15.042 ms  15.028 ms

Then suddenly the VM could not ping the Site A resources and the traceroute output would look like this:

# traceroute 10.104.19.4
traceroute to 10.104.19.4 (10.104.19.4), 30 hops max, 60 byte packets
 1  10.104.19.4 (10.104.19.4)  0.552 ms  0.567 ms  0.616 ms
 2  * 10.104.19.4 (10.104.19.4)  0.659 ms  0.707 ms
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *^C

It looks completely random tho. When this eventually happens I would remove then add the route again with:

# route del -net 10.104.16.0 gw 10.240.132.15 netmask 255.255.240.0
# route add -net 10.104.16.0 gw 10.240.132.15 netmask 255.255.240.0

Of course that solves the problem for a while it doesn't last for long... Any idea of what could go wrong or of what I'm doing wrong?

Thank you guys ;)

Answer 1

Ok never mind... After realizing the connection would be lost every 5 minutes precisely, at 9:00AM, 9:05AM, 9:10AM... I had a look at the Strongswan logs and discovered that it would restart the service at those given times (process receiving a SIGKILL command).

We had a talk with the team leader and he said

Hum we might have a cron job on the strongswan server pinging a remote IP and rebooting the service if the remote IP cannot be found.

Yes indeed. And because that remote IP was long gone and nobody disabled or updated this undocumented job, it was rebooting the service since, well, since the beginning. Until we noticed the issue while replicating a Postgres database.