2

On my Ubuntu 14.04.2 server IPv4 goes offline several times per hour (one to four times I've seen, but at no particular minute per hour or so).

My hoster insists that the problem is on the server-side and the fact that a Debian-based rescue system doesn't show the same symptoms makes me think they're right. However, the rescue system doesn't configure a global IPv6 address on any interface, like the installed Ubuntu system does.

Routinely between one to four times an hour the (IPv4-based) SSH connection will drop due to too many timed out packets.

When monitoring the server from another remote server ICMPv4 pings will either time out or the router will respond that the destination host isn't available (I routinely see both!). At the same time the ICMPv6 pings are totally unaffected.

Also, when I use IPv6 to connect from that other remote host via SSH, that connection doesn't stall nor does the system appear to freeze or so (as I had suspected initially).

The system and kernel logs indicate no issues either and it makes no difference whether I disable all firewall rules or leave the firewall turned on. I also had it running with logging enabled for all dropped packets to see whether I could correlate something there.

No cron jobs are running at those offline times and it also doesn't happen at the same minute or so, indicating some regular cron job.

I also narrowed another aspect of this down. When I ping (ICMPv4) from the host that shows the symptoms, loopback is not affected, eth0 is. This would suggest to me that it's not about IPv4 in general, but specific to the interface that corresponds to the one network card in the system.

How can I proceed my troubleshooting from here? What would be the next step(s), given what I have done so far? Is there perhaps even a known bug that would correspond to the symptoms I see?

NB: I have worked on diagnosing this for well over a month. So asking here, to me is kind of a last resort. Please request more details as needed and I will add them.


What I have done so far:

  • ping vs. ping6
  • mtr from and to the server, my hoster doesn't deem the few lost packets anything irregular
  • SSH connection via IPv4 and IPv6 respectively
  • tail-ed /var/log/kern.log, /var/log/syslog and /var/log/auth.log to see whether anything would show up during the offline period
  • flushed all firewall rules for IPv4 and IPv6 respectively
    • also simply enabled logging for dropping of packets
  • removed several packages I suspected of being potential culprits

Here are the list of manually installed packages:

# echo $(apt-mark showmanual)
acl adduser aggregate apparmor apparmor-profiles apparmor-utils apt apt-cacher-ng apt-file apt-rdepends apt-utils base-files base-passwd bash bash-completion bash-static bridge-utils bsdutils btrfs-tools busybox-initramfs busybox-static bzip2 bzr ca-certificates cgmanager cgroup-bin cifs-utils colordiff coreutils cpio crda cron cron-apt cryptmount cryptsetup dash debconf debianutils debootstrap debsums dh-python dialog diffutils dnsutils dpkg dpkg-dev duplicity e2fslibs e2fsprogs ed etckeeper fakechroot fakeroot file findutils gcc-4.8-base gcc-4.9-base gdisk-noicu git git-svn gnupg gnutls-bin gpgv grep gzip haveged heirloom-mailx hostname htop ifupdown init-system-helpers initramfs-tools initramfs-tools-bin initscripts insserv iproute2 ipset iptables iputils-ping klibc-utils kmod kpartx less libacl1 libapt-inst1.5 libapt-pkg4.12 libattr1 libaudit-common libaudit1 libblkid1 libbz2-1.0 libc-bin libc6 libcap2 libcgmanager0 libck-connector0 libcomerr2 libdb5.3 libdbus-1-3 libdebconfclient0 libdrm2 libedit2 libevent-2.0-5 libexpat1 libffi6 libgcc1 libgdbm3 libgssapi-krb5-2 libjson-c2 libjson0 libk5crypto3 libkeyutils1 libklibc libkmod2 libkrb5-3 libkrb5support0 liblzma5 libmount1 libmpdec2 libncurses5 libncursesw5 libnih-dbus1 libnih1 libnl-3-200 libnl-genl-3-200 libpam-modules libpam-modules-bin libpam-mount libpam-runtime libpam-systemd libpam0g libpci3 libpcre3 libplymouth2 libpng12-0 libprocps3 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libpython3-stdlib libpython3.4-minimal libpython3.4-stdlib libreadline6 libselinux1 libsemanage-common libsemanage1 libsepol1 libslang2 libsqlite3-0 libss2 libssl1.0.0 libstdc++6 libtinfo5 libudev1 libui-dialog-perl libusb-0.1-4 libusb-1.0-0 libustr-1.0-1 libuuid1 libwrap0 linux-firmware linux-image-3.13.0-24-generic linux-image-extra-3.13.0-24-generic linux-image-generic localepurge locales logcheck logcheck-database login logrotate lsb-base lsb-release lshw lsof lxc lxc-templates make makedev man-db manpages manpages-dev mawk mc md5deep mdadm mercurial mime-support mlocate module-init-tools molly-guard mount mountall mtr-tiny multiarch-support ncurses-base ncurses-bin ndisc6 net-tools netcat-openbsd netsniff-ng nmap openntpd openssh-client openssh-server openssh-sftp-server p7zip-full p7zip-rar passwd pax pciutils perl perl-base perl-modules plymouth postfix procps psmisc pv python python-apt-common python-mako python-mechanize python-minimal python2.7 python2.7-minimal python3 python3-apt python3-minimal python3.4 python3.4-minimal readline-common reprepro resolvconf rsyslog sed sensible-utils sharutils smartmontools subversion sudo sysv-rc sysvinit-utils tar tcpdump tcptraceroute tmux traceroute tree tzdata ubuntu-keyring ucf udev uidmap unattended-upgrades unbound-host unrar unzip upstart usbutils util-linux vim-nox vnstat wget whois wireless-regdb xz-utils zerofree zip zlib1g zsh-doc zsh-static

(Some of these come from the debootstrap process, of course.)


The requested information:

$ uname -a|sed 's/'$(hostname -f)'/foobar/g'
Linux foobar 3.13.0-46-generic #79-Ubuntu SMP Tue Mar 10 20:06:50 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I updated to a newer kernel (package linux-image-generic-lts-utopic):

$ uname -a|sed 's/'$(hostname -f)'/foobar/g'
Linux foobar 3.16.0-33-generic #44~14.04.1-Ubuntu SMP Fri Mar 13 10:33:29 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

The sysctl -a output has been anonymized and put here.

The command was (minus one sed to replace the name of an interface to _bridge):

sudo sysctl -a|sed 's/'$(hostname -f)'/foobar/g;s/'$(hostname -s)'/foobar/g'|grep -Ev '^net\.ipv[46]\.(neigh|conf)\._[s]'|grep -v nf_log

There are overall three interfaces like _bridge all configured for IPv4 and IPv6 and only differing in IP addresses. However, they aren't currently in use. They are slated to be used for one LXC guest each.

# lspci -s 06:00.0 -vv
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 02)
        Subsystem: Micro-Star International Co., Ltd. [MSI] X58 Pro-E
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 256 bytes
        Interrupt: pin A routed to IRQ 42
        Region 0: I/O ports at e800 [size=256]
        Region 2: Memory at fbeff000 (64-bit, non-prefetchable) [size=4K]
        Region 4: Memory at f6ff0000 (64-bit, prefetchable) [size=64K]
        [virtual] Expansion ROM at fbe00000 [disabled] [size=128K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee00000  Data: 40c1
        Capabilities: [70] Express (v1) Endpoint, MSI 01
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        Capabilities: [b0] MSI-X: Enable- Count=2 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00000800
        Capabilities: [d0] Vital Product Data
                Unknown small resource type 05, will not decode more.
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [140 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=01
                        Status: NegoPending- InProgress-
        Capabilities: [160 v1] Device Serial Number 01-00-00-00-68-4c-e0-00
        Kernel driver in use: r8169
# modinfo r8169
filename:       /lib/modules/3.16.0-33-generic/kernel/drivers/net/ethernet/realtek/r8169.ko
firmware:       rtl_nic/rtl8168g-3.fw
firmware:       rtl_nic/rtl8168g-2.fw
firmware:       rtl_nic/rtl8106e-2.fw
firmware:       rtl_nic/rtl8106e-1.fw
firmware:       rtl_nic/rtl8411-2.fw
firmware:       rtl_nic/rtl8411-1.fw
firmware:       rtl_nic/rtl8402-1.fw
firmware:       rtl_nic/rtl8168f-2.fw
firmware:       rtl_nic/rtl8168f-1.fw
firmware:       rtl_nic/rtl8105e-1.fw
firmware:       rtl_nic/rtl8168e-3.fw
firmware:       rtl_nic/rtl8168e-2.fw
firmware:       rtl_nic/rtl8168e-1.fw
firmware:       rtl_nic/rtl8168d-2.fw
firmware:       rtl_nic/rtl8168d-1.fw
version:        2.3LK-NAPI
license:        GPL
description:    RealTek RTL-8169 Gigabit Ethernet driver
author:         Realtek and the Linux r8169 crew <netdev@vger.kernel.org>
srcversion:     D0E1934D763B6927E0CB4A4
alias:          pci:v00000001d00008168sv*sd00002410bc*sc*i*
alias:          pci:v00001737d00001032sv*sd00000024bc*sc*i*
alias:          pci:v000016ECd00000116sv*sd*bc*sc*i*
alias:          pci:v00001259d0000C107sv*sd*bc*sc*i*
alias:          pci:v00001186d00004302sv*sd*bc*sc*i*
alias:          pci:v00001186d00004300sv*sd*bc*sc*i*
alias:          pci:v00001186d00004300sv00001186sd00004B10bc*sc*i*
alias:          pci:v000010ECd00008169sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008168sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008167sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008136sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008129sv*sd*bc*sc*i*
depends:        mii
intree:         Y
vermagic:       3.16.0-33-generic SMP mod_unload modversions
signer:         Magrathea: Glacier signing key
sig_key:        25:26:EE:FE:32:C9:58:B4:CD:85:CA:5F:BF:EB:ED:A1:75:D1:B2:18
sig_hashalgo:   sha512
parm:           use_dac:Enable PCI DAC. Unsafe on 32 bit PCI slot. (int)
parm:           debug:Debug verbosity level (0=none, ..., 16=all) (int)
-1

You have a network stack issue for sure, so I do recommend you long but proven roadmap to fix this. I used it myself in a similar cases, it happened even on a real hardware. Install Ncurses and build essential dev packets along with compiler. First, Make a git snapshot of Linux kernel, and do this in the git snapshot directory :

git checkout-index -a -f --prefix=/usr/src/linux-build/ <--- trailing slash is MUST-HAVE!
cd /usr/src/linux-build
cp /boot/config-\`uname -r\` .config
make menuconfig

Check all the IP options you need and DISABLE IPv6. After that build and install your kernel. Second, in your /etc/sysctl.conf :

net.ipv6.conf.default.autoconf=0
net.ipv6.conf.default.accept_dad=0
net.ipv6.conf.default.accept_ra=0
net.ipv6.conf.default.accept_ra_defrtr=0
net.ipv6.conf.default.accept_ra_rtr_pref=0
net.ipv6.conf.default.accept_ra_pinfo=0
net.ipv6.conf.default.accept_source_route=0
net.ipv6.conf.default.accept_redirects=0
net.ipv6.conf.default.forwarding=0
net.ipv6.conf.all.autoconf=0
net.ipv6.conf.all.accept_dad=0
net.ipv6.conf.all.accept_ra=0
net.ipv6.conf.all.accept_ra_defrtr=0
net.ipv6.conf.all.accept_ra_rtr_pref=0
net.ipv6.conf.all.accept_ra_pinfo=0
net.ipv6.conf.all.accept_source_route=0
net.ipv6.conf.all.accept_redirects=0
net.ipv6.conf.all.forwarding=0
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1

Reboot then, and check this in your /etc/ssh/sshd_config :

KeyRegenerationInterval 3600
ServerKeyBits 768
Compression yes <---- PAY ATTENTION TO THIS : see below

Compression directive MUST be set to "yes" or "no", by default it's "delayed" - that's a problem-starter value.

  • 1
    Thanks for your time, but can you explain to me how disabling IPv6 is going to solve anything? YMMV, but I actually want to use IPv6 and IPv4 concurrently. Your suggestion is like suggesting to cut off my left leg just because my right leg itches occasionally. – 0xC0000022L Mar 17 '15 at 17:50
  • Stack co-existense is not bad, but auto-configuration of IPv6 seems to struck IPv4 part sometimes. It's a RARE case, but I had encountered it. And the solution was to disable unconfigured IPv6 even if I haven't used it at all. If you plan to use it, just make a static address and router setup and explicitly disable any autoconfig on it. You've said that you haven't used IPv6, in this case disabling will be a solution. – Alexey Vesnin Mar 17 '15 at 21:51
  • I have not made such a statement. In fact I have static addresses configured for IPv6 and auto-configuration disabled. Whe connecting via IPv6 and SSH (something that is in my question) everything works fine. The issue is with IPv4 having "outages" several times an hour. Can you detail how it affects IPv4? Also, why does this yield offline time occasionally but not something like IPv4 failing altogether (e.g. interface dropping address)? – 0xC0000022L Mar 17 '15 at 22:27
  • please post your sysctl -a list and uname -a - it's interesting then... – Alexey Vesnin Mar 18 '15 at 15:43
  • put it up on pastebin, but slightly redacted. See my edit on the question. – 0xC0000022L Mar 18 '15 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.