I already did a bunch of troubleshooting in the last three days regarding this issue. Here are the tests and the observations I made so far:
- Internet connection to my server is dropping constantly (see ping below)
- Server has static address and the address is out of the DCHP Range of the router
- Modem is a Modem/Router used in Bridged mode. (Reset to factory settings 3 times now)
- I tried 3 different routers = same results 50% packet loss
- I tried two different physical interfaces = same 50% packet loss
- I tried 3 different Ethernet cables = same 50% packet loss
- I stopped many of my processes on my server simultaneously (see the list below)
- I disabled my firewall
- My connections from the clients to the server on my LAN is always 100% packet received
- Connecting the server directly to the modem showed no packet loss while pinging 8.8.8.8
- Got nothing unusual in dmesg or journalctl when the Internet connection failures happen
- Connecting the Ethernet cable in my laptop instead of the server results in no packet loss from the laptop while pinging 8.8.8.8
- Booting a Arch Linux from a USB drive gives me no problem with the Internet whatsoever.
I figured at that point that this must be a software related issue. But now I'm stuck. Here's my ping results. It's basically the same every time I run it under the different environments I tried listed below.
➜ ~ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=14.5 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=119 time=14.4 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=119 time=13.9 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=119 time=14.9 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=119 time=16.1 ms
64 bytes from 8.8.8.8: icmp_seq=6 ttl=119 time=14.8 ms
64 bytes from 8.8.8.8: icmp_seq=7 ttl=119 time=15.2 ms
64 bytes from 8.8.8.8: icmp_seq=8 ttl=119 time=15.5 ms
64 bytes from 8.8.8.8: icmp_seq=9 ttl=119 time=2052 ms
64 bytes from 8.8.8.8: icmp_seq=10 ttl=119 time=1024 ms
64 bytes from 8.8.8.8: icmp_seq=11 ttl=119 time=16.7 ms
64 bytes from 8.8.8.8: icmp_seq=12 ttl=119 time=22.7 ms
64 bytes from 8.8.8.8: icmp_seq=13 ttl=119 time=19.5 ms
64 bytes from 8.8.8.8: icmp_seq=14 ttl=119 time=23.0 ms
64 bytes from 8.8.8.8: icmp_seq=15 ttl=119 time=12.6 ms
64 bytes from 8.8.8.8: icmp_seq=16 ttl=119 time=18.0 ms
64 bytes from 8.8.8.8: icmp_seq=17 ttl=119 time=26.0 ms
64 bytes from 8.8.8.8: icmp_seq=18 ttl=119 time=18.8 ms
64 bytes from 8.8.8.8: icmp_seq=31 ttl=119 time=24.8 ms
64 bytes from 8.8.8.8: icmp_seq=32 ttl=119 time=13.9 ms
64 bytes from 8.8.8.8: icmp_seq=33 ttl=119 time=19.0 ms
64 bytes from 8.8.8.8: icmp_seq=34 ttl=119 time=15.9 ms
64 bytes from 8.8.8.8: icmp_seq=35 ttl=119 time=15.6 ms
64 bytes from 8.8.8.8: icmp_seq=36 ttl=119 time=14.6 ms
64 bytes from 8.8.8.8: icmp_seq=37 ttl=119 time=12.8 ms
64 bytes from 8.8.8.8: icmp_seq=38 ttl=119 time=14.7 ms
64 bytes from 8.8.8.8: icmp_seq=39 ttl=119 time=14.7 ms
64 bytes from 8.8.8.8: icmp_seq=40 ttl=119 time=16.0 ms
64 bytes from 8.8.8.8: icmp_seq=41 ttl=119 time=15.2 ms
64 bytes from 8.8.8.8: icmp_seq=42 ttl=119 time=15.8 ms
64 bytes from 8.8.8.8: icmp_seq=43 ttl=119 time=33.0 ms
64 bytes from 8.8.8.8: icmp_seq=44 ttl=119 time=21.1 ms
64 bytes from 8.8.8.8: icmp_seq=45 ttl=119 time=19.4 ms
64 bytes from 8.8.8.8: icmp_seq=46 ttl=119 time=15.0 ms
64 bytes from 8.8.8.8: icmp_seq=47 ttl=119 time=30.3 ms
64 bytes from 8.8.8.8: icmp_seq=48 ttl=119 time=15.8 ms
64 bytes from 8.8.8.8: icmp_seq=49 ttl=119 time=14.0 ms
64 bytes from 8.8.8.8: icmp_seq=50 ttl=119 time=12.9 ms
64 bytes from 8.8.8.8: icmp_seq=51 ttl=119 time=16.1 ms
64 bytes from 8.8.8.8: icmp_seq=52 ttl=119 time=14.3 ms
64 bytes from 8.8.8.8: icmp_seq=53 ttl=119 time=15.8 ms
64 bytes from 8.8.8.8: icmp_seq=54 ttl=119 time=14.4 ms
64 bytes from 8.8.8.8: icmp_seq=55 ttl=119 time=15.1 ms
64 bytes from 8.8.8.8: icmp_seq=56 ttl=119 time=20.4 ms
64 bytes from 8.8.8.8: icmp_seq=57 ttl=119 time=13.9 ms
64 bytes from 8.8.8.8: icmp_seq=58 ttl=119 time=15.5 ms
64 bytes from 8.8.8.8: icmp_seq=59 ttl=119 time=16.1 ms
64 bytes from 8.8.8.8: icmp_seq=60 ttl=119 time=14.5 ms
64 bytes from 8.8.8.8: icmp_seq=61 ttl=119 time=14.5 ms
64 bytes from 8.8.8.8: icmp_seq=62 ttl=119 time=13.5 ms
64 bytes from 8.8.8.8: icmp_seq=63 ttl=119 time=14.9 ms
64 bytes from 8.8.8.8: icmp_seq=64 ttl=119 time=16.1 ms
64 bytes from 8.8.8.8: icmp_seq=65 ttl=119 time=16.3 ms
64 bytes from 8.8.8.8: icmp_seq=66 ttl=119 time=14.0 ms
64 bytes from 8.8.8.8: icmp_seq=85 ttl=119 time=2080 ms
64 bytes from 8.8.8.8: icmp_seq=86 ttl=119 time=1056 ms
64 bytes from 8.8.8.8: icmp_seq=87 ttl=119 time=32.5 ms
64 bytes from 8.8.8.8: icmp_seq=88 ttl=119 time=22.6 ms
64 bytes from 8.8.8.8: icmp_seq=89 ttl=119 time=15.3 ms
64 bytes from 8.8.8.8: icmp_seq=90 ttl=119 time=12.9 ms
64 bytes from 8.8.8.8: icmp_seq=91 ttl=119 time=14.7 ms
64 bytes from 8.8.8.8: icmp_seq=92 ttl=119 time=15.2 ms
64 bytes from 8.8.8.8: icmp_seq=93 ttl=119 time=15.7 ms
64 bytes from 8.8.8.8: icmp_seq=94 ttl=119 time=17.5 ms
64 bytes from 8.8.8.8: icmp_seq=95 ttl=119 time=15.1 ms
64 bytes from 8.8.8.8: icmp_seq=117 ttl=119 time=16.6 ms
64 bytes from 8.8.8.8: icmp_seq=118 ttl=119 time=13.0 ms
64 bytes from 8.8.8.8: icmp_seq=119 ttl=119 time=13.6 ms
64 bytes from 8.8.8.8: icmp_seq=120 ttl=119 time=11.8 ms
64 bytes from 8.8.8.8: icmp_seq=121 ttl=119 time=13.6 ms
64 bytes from 8.8.8.8: icmp_seq=122 ttl=119 time=13.6 ms
64 bytes from 8.8.8.8: icmp_seq=123 ttl=119 time=13.5 ms
64 bytes from 8.8.8.8: icmp_seq=124 ttl=119 time=27.5 ms
64 bytes from 8.8.8.8: icmp_seq=149 ttl=119 time=960 ms
64 bytes from 8.8.8.8: icmp_seq=150 ttl=119 time=14.6 ms
64 bytes from 8.8.8.8: icmp_seq=151 ttl=119 time=15.9 ms
64 bytes from 8.8.8.8: icmp_seq=152 ttl=119 time=15.6 ms
64 bytes from 8.8.8.8: icmp_seq=153 ttl=119 time=26.9 ms
64 bytes from 8.8.8.8: icmp_seq=154 ttl=119 time=13.6 ms
64 bytes from 8.8.8.8: icmp_seq=155 ttl=119 time=15.5 ms
64 bytes from 8.8.8.8: icmp_seq=156 ttl=119 time=15.2 ms
64 bytes from 8.8.8.8: icmp_seq=157 ttl=119 time=16.0 ms
64 bytes from 8.8.8.8: icmp_seq=164 ttl=119 time=14.1 ms
64 bytes from 8.8.8.8: icmp_seq=165 ttl=119 time=15.2 ms
64 bytes from 8.8.8.8: icmp_seq=166 ttl=119 time=21.3 ms
64 bytes from 8.8.8.8: icmp_seq=167 ttl=119 time=14.7 ms
64 bytes from 8.8.8.8: icmp_seq=168 ttl=119 time=15.4 ms
64 bytes from 8.8.8.8: icmp_seq=169 ttl=119 time=16.1 ms
64 bytes from 8.8.8.8: icmp_seq=170 ttl=119 time=14.3 ms
64 bytes from 8.8.8.8: icmp_seq=171 ttl=119 time=15.9 ms
64 bytes from 8.8.8.8: icmp_seq=172 ttl=119 time=20.3 ms
64 bytes from 8.8.8.8: icmp_seq=173 ttl=119 time=16.8 ms
64 bytes from 8.8.8.8: icmp_seq=174 ttl=119 time=16.1 ms
64 bytes from 8.8.8.8: icmp_seq=175 ttl=119 time=14.0 ms
64 bytes from 8.8.8.8: icmp_seq=176 ttl=119 time=17.3 ms
64 bytes from 8.8.8.8: icmp_seq=177 ttl=119 time=16.7 ms
64 bytes from 8.8.8.8: icmp_seq=178 ttl=119 time=21.3 ms
64 bytes from 8.8.8.8: icmp_seq=179 ttl=119 time=14.4 ms
64 bytes from 8.8.8.8: icmp_seq=180 ttl=119 time=17.6 ms
64 bytes from 8.8.8.8: icmp_seq=181 ttl=119 time=13.7 ms
64 bytes from 8.8.8.8: icmp_seq=182 ttl=119 time=23.8 ms
64 bytes from 8.8.8.8: icmp_seq=183 ttl=119 time=15.4 ms
64 bytes from 8.8.8.8: icmp_seq=184 ttl=119 time=14.7 ms
64 bytes from 8.8.8.8: icmp_seq=185 ttl=119 time=20.2 ms
64 bytes from 8.8.8.8: icmp_seq=186 ttl=119 time=14.5 ms
64 bytes from 8.8.8.8: icmp_seq=187 ttl=119 time=12.8 ms
64 bytes from 8.8.8.8: icmp_seq=188 ttl=119 time=15.4 ms
64 bytes from 8.8.8.8: icmp_seq=189 ttl=119 time=14.8 ms
64 bytes from 8.8.8.8: icmp_seq=190 ttl=119 time=16.1 ms
64 bytes from 8.8.8.8: icmp_seq=191 ttl=119 time=14.1 ms
64 bytes from 8.8.8.8: icmp_seq=195 ttl=119 time=320 ms
64 bytes from 8.8.8.8: icmp_seq=196 ttl=119 time=13.5 ms
64 bytes from 8.8.8.8: icmp_seq=197 ttl=119 time=14.5 ms
64 bytes from 8.8.8.8: icmp_seq=198 ttl=119 time=15.5 ms
64 bytes from 8.8.8.8: icmp_seq=199 ttl=119 time=15.0 ms
64 bytes from 8.8.8.8: icmp_seq=200 ttl=119 time=13.1 ms
64 bytes from 8.8.8.8: icmp_seq=201 ttl=119 time=17.9 ms
64 bytes from 8.8.8.8: icmp_seq=202 ttl=119 time=15.6 ms
64 bytes from 8.8.8.8: icmp_seq=203 ttl=119 time=13.2 ms
64 bytes from 8.8.8.8: icmp_seq=225 ttl=119 time=1056 ms
64 bytes from 8.8.8.8: icmp_seq=226 ttl=119 time=32.5 ms
64 bytes from 8.8.8.8: icmp_seq=227 ttl=119 time=15.5 ms
64 bytes from 8.8.8.8: icmp_seq=228 ttl=119 time=13.6 ms
64 bytes from 8.8.8.8: icmp_seq=229 ttl=119 time=15.7 ms
64 bytes from 8.8.8.8: icmp_seq=230 ttl=119 time=21.7 ms
64 bytes from 8.8.8.8: icmp_seq=231 ttl=119 time=16.9 ms
64 bytes from 8.8.8.8: icmp_seq=232 ttl=119 time=13.5 ms
64 bytes from 8.8.8.8: icmp_seq=233 ttl=119 time=14.5 ms
^C
--- 8.8.8.8 ping statistics ---
248 packets transmitted, 128 received, 48.3871% packet loss, time 1216ms
rtt min/avg/max/mdev = 11.806/82.526/2080.231/306.259 ms, pipe 3
Two traceroute executed back to back
➜ ~ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 * * *
2 My WAN IP.mc.videotron.ca (My WAN IP) 19.047 ms 10.699 ms 19.035 ms
3 10.170.183.146 (10.170.183.146) 20.538 ms 21.036 ms 21.203 ms
4 72.14.205.46 (72.14.205.46) 20.269 ms 20.188 ms 20.189 ms
5 108.170.251.17 (108.170.251.17) 20.355 ms 108.170.251.33 (108.170.251.33) 20.149 ms 108.170.251.17 (108.170.251.17) 20.134 ms
6 108.170.231.63 (108.170.231.63) 21.359 ms 108.170.231.61 (108.170.231.61) 16.538 ms 108.170.231.63 (108.170.231.63) 16.186 ms
7 dns.google (8.8.8.8) 16.761 ms 15.483 ms 15.866 ms
➜ ~ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 * * *
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * dns.google (8.8.8.8)
/etc/network/interfaces
➜ ~ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
#source /etc/network/interfaces.d/*
# The loopback network interface
#auto lo
#iface lo inet loopback
# The primary network interface
#allow-hotplug enx00e04c6802a2
auto lo br0
iface lo inet loopback
auto enp5s0
iface enp5s0 inet manual
iface br0 inet static
address 192.168.3.104
netmask 255.255.255.0
gateway 192.168.3.1
bridge_ports enp5s0
bridge_stp off
bridge_fd 0
bridge_maxwait 0
post-up iptables-restore < /etc/iptables.up.rules
Thanks for your help
EDIT 1
➜ ~ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP mode DEFAULT group default qlen 1000
link/ether 70:85:c2:da:2a:87 brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 70:85:c2:da:2a:87 brd ff:ff:ff:ff:ff:ff
4: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/none
➜ ~ ip -br address
lo UNKNOWN 127.0.0.1/8 ::1/128
enp5s0 UP
br0 UP 192.168.3.104/24 fe80::7285:c2ff:feda:2a87/64
wg0 UNKNOWN 192.168.3.1/24
➜ ~ ip route;
default via 192.168.3.1 dev br0 onlink
103.83.20.3 dev wg0 scope link
103.83.20.4 dev wg0 scope link
169.254.0.0/16 dev br0 scope link metric 1000
192.168.3.0/24 dev br0 proto kernel scope link src 192.168.3.104
192.168.3.0/24 dev wg0 proto kernel scope link src 192.168.3.1
➜ ~ ip neigh
192.168.3.126 dev br0 lladdr 04:6c:59:f6:a0:3b REACHABLE
192.168.3.164 dev br0 lladdr be:95:3a:fc:bf:03 STALE
[PUBLIC IP] dev br0 lladdr c4:04:15:10:b1:bd STALE
192.168.3.127 dev br0 lladdr 4c:3b:df:7f:c6:03 STALE
192.168.3.1 dev br0 lladdr c4:04:15:10:b1:bd REACHABLE
192.168.3.125 dev br0 lladdr 0c:c4:13:18:a4:c5 STALE
192.168.3.144 dev br0 lladdr b8:27:eb:28:4a:98 REACHABLE
➜ ~ ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
EDIT 2 @A.B is right. I forgot to mention I have been running a VPN Server (Wireguard) and it was indeed the problem.
ip link; ip -br address; ip route; ip neigh; ip rule
+iptables-save -c
. Don't omit parts you'd think aren't needed, or explain why and what you omitted.