3

This issue appeared after the CentOS installation. Server becomes unavailable completely neither through ssh nor through smb protocol. It happens randomly and there is no exact regularity. It can happen at any moment of time and doesn't depend on server loading or kernel version (tried different for the last month). So after the my first investigation I found out that it's related to network. When I could catch this "blackout" it turned out that I couldn't ping server neither from office net to server nor from server to any machine in the office but I could ping localhost. Moreover network green LED at the back of motherboard just was lighting constantly without blinking and orange LED didn't light up at all. So when I did systemctl restart network locally on the server everything became fine.

Last time it happened the 14-th of Oct 2016 at 16:05 (by MSK TZ)

Checked dmesq

[2016-11-13 01:00:32]  md: using 128k window, over a total of 487313408k.
    [2016-11-13 02:54:57]  md: md127: data-check done.
    [2016-11-13 04:52:32]  md: md1: data-check done.
    [2016-11-14 16:36:57]  usb 4-2: new low-speed USB device number 2 using uhci_hcd
    [2016-11-14 16:36:58]  usb 4-2: New USB device found, idVendor=1a2c, idProduct=0c23
    [2016-11-14 16:36:58]  usb 4-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
    [2016-11-14 16:36:58]  usb 4-2: Product: USB Keyboard
    [2016-11-14 16:36:58]  usb 4-2: Manufacturer: USB
    [2016-11-14 16:36:58]  input: USB USB Keyboard as /devices/pci0000:00/0000:00:1a.1/usb4/4-2/4-2:1.0/input/input15
    [2016-11-14 16:36:58]  hid-generic 0003:1A2C:0C23.0001: input,hidraw0: USB HID v1.10 Keyboard [USB USB Keyboard] on usb-0000:00:1a.1-2/input0
    [2016-11-14 16:36:58]  input: USB USB Keyboard as /devices/pci0000:00/0000:00:1a.1/usb4/4-2/4-2:1.1/input/input16
    [2016-11-14 16:36:58]  hid-generic 0003:1A2C:0C23.0002: input,hidraw1: USB HID v1.10 Device [USB USB Keyboard] on usb-0000:00:1a.1-2/input1
    [root@london stats]#

As we can see - nothing

Checked /var/log/messages

Nov 14 15:01:01 london systemd: Removed slice user-0.slice.
Nov 14 15:01:01 london systemd: Stopping user-0.slice.
Nov 14 16:01:01 london systemd: Created slice user-0.slice.
Nov 14 16:01:01 london systemd: Starting user-0.slice.
Nov 14 16:01:01 london systemd: Started Session 171 of user root.
Nov 14 16:01:01 london systemd: Starting Session 171 of user root.
Nov 14 16:01:01 london systemd: Removed slice user-0.slice.
Nov 14 16:01:01 london systemd: Stopping user-0.slice.
Nov 14 16:05:24 london smbd[6249]: [2016/11/14 16:05:24.098079,  0] ../source3/smbd/oplock.c:700(oplock_timeout_handler)
Nov 14 16:05:24 london smbd[6249]:  Oplock break failed for file Z_Drive/BackUp/...... DC.lnk -- replying anyway
Nov 14 16:05:54 london smbd[6249]: [2016/11/14 16:05:54.133612,  0] ../source3/smbd/oplock.c:700(oplock_timeout_handler)
Nov 14 16:05:54 london smbd[6249]:  Oplock break failed for file Z_Drive/BackUp/..... X8.lnk -- replying anyway
.
.
.
.
Nov 14 16:36:57 london kernel: usb 4-2: new low-speed USB device number 2 using uhci_hcd
Nov 14 16:36:57 london kernel: usb 4-2: New USB device found, idVendor=1a2c, idProduct=0c23
Nov 14 16:36:57 london kernel: usb 4-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Nov 14 16:36:57 london kernel: usb 4-2: Product: USB Keyboard
Nov 14 16:36:57 london kernel: usb 4-2: Manufacturer: USB

As we can see there is nothing here too. There are a lot of failed messages from smbd demon about broken connection. But it's not appropriate for us I assume.

Systemctl told us that network is OK too. There are no entries for the 14 of Oct.

   ● network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network)
   Active: active (exited) since Tue 2016-11-08 01:53:35 MSK; 6 days ago
     Docs: man:systemd-sysv-generator(8)
  Process: 925 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=0/SUCCESS)

Nov 08 01:53:32 london.local systemd[1]: Starting LSB: Bring up/down networking...
Nov 08 01:53:32 london.local network[925]: Bringing up loopback interface:  Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
Nov 08 01:53:33 london.local network[925]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
Nov 08 01:53:33 london.local network[925]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
Nov 08 01:53:33 london.local network[925]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
Nov 08 01:53:33 london.local network[925]: [  OK  ]
Nov 08 01:53:35 london.local network[925]: Bringing up interface enp2s0:  [  OK  ]
Nov 08 01:53:35 london.local systemd[1]: Started LSB: Bring up/down networking.

Here additional data for investigation

Kernel info

3.10.0-327.36.3.el7.x86_64

OS info

CentOS Linux release 7.2.1511 (Core)

Network info

dmesg | grep -i 'Ethernet'

[root@london /]# dmesg | grep -i 'Ethernet'
[    1.327135] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    1.327619] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[root@london /]#

modinfo r8169

[root@london /]# modinfo r8169
filename:       /lib/modules/3.10.0-327.36.3.el7.x86_64/kernel/drivers/net/ethernet/realtek/r8169.ko
firmware:       rtl_nic/rtl8107e-2.fw
firmware:       rtl_nic/rtl8107e-1.fw
firmware:       rtl_nic/rtl8168h-2.fw
firmware:       rtl_nic/rtl8168h-1.fw
firmware:       rtl_nic/rtl8168g-3.fw
firmware:       rtl_nic/rtl8168g-2.fw
firmware:       rtl_nic/rtl8106e-2.fw
firmware:       rtl_nic/rtl8106e-1.fw
firmware:       rtl_nic/rtl8411-2.fw
firmware:       rtl_nic/rtl8411-1.fw
firmware:       rtl_nic/rtl8402-1.fw
firmware:       rtl_nic/rtl8168f-2.fw
firmware:       rtl_nic/rtl8168f-1.fw
firmware:       rtl_nic/rtl8105e-1.fw
firmware:       rtl_nic/rtl8168e-3.fw
firmware:       rtl_nic/rtl8168e-2.fw
firmware:       rtl_nic/rtl8168e-1.fw
firmware:       rtl_nic/rtl8168d-2.fw
firmware:       rtl_nic/rtl8168d-1.fw
version:        2.3LK-NAPI
license:        GPL
description:    RealTek RTL-8169 Gigabit Ethernet driver
author:         Realtek and the Linux r8169 crew <netdev@vger.kernel.org>
rhelversion:    7.2
srcversion:     17C11305BF371984CB1A920
alias:          pci:v00000001d00008168sv*sd00002410bc*sc*i*
alias:          pci:v00001737d00001032sv*sd00000024bc*sc*i*
alias:          pci:v000016ECd00000116sv*sd*bc*sc*i*
alias:          pci:v00001259d0000C107sv*sd*bc*sc*i*
alias:          pci:v00001186d00004302sv*sd*bc*sc*i*
alias:          pci:v00001186d00004300sv*sd*bc*sc*i*
alias:          pci:v00001186d00004300sv00001186sd00004B10bc*sc*i*
alias:          pci:v000010ECd00008169sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008168sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008167sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008136sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008129sv*sd*bc*sc*i*
depends:        mii
intree:         Y
vermagic:       3.10.0-327.36.3.el7.x86_64 SMP mod_unload modversions
signer:         CentOS Linux kernel signing key
sig_key:        DD:8A:C3:DE:FE:A6:07:A6:13:C9:58:35:D0:37:19:40:E7:75:43:CF
sig_hashalgo:   sha256
parm:           use_dac:Enable PCI DAC. Unsafe on 32 bit PCI slot. (int)
parm:           debug:Debug verbosity level (0=none, ..., 16=all) (int)
[root@london /]#

lspci | grep net

[root@london /]# lspci | grep net
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8110SC/8169SC Gigabit Ethernet (rev 10)
02:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8110SC/8169SC Gigabit Ethernet (rev 10)
[root@london /]#

lshw -class net

[root@london av]# lshw -class net
  *-network:0 UNCLAIMED
       description: Ethernet controller
       product: RTL-8110SC/8169SC Gigabit Ethernet
       vendor: Realtek Semiconductor Co., Ltd.
       physical id: 0
       bus info: pci@0000:02:00.0
       version: 10
       width: 32 bits
       clock: 66MHz
       capabilities: pm cap_list
       configuration: latency=32 maxlatency=64 mingnt=32
       resources: ioport:ce00(size=256) memory:fddff000-fddff0ff memory:fdd00000-fdd1ffff
  *-network:1 UNCLAIMED
       description: Ethernet controller
       product: RTL-8110SC/8169SC Gigabit Ethernet
       vendor: Realtek Semiconductor Co., Ltd.
       physical id: 1
       bus info: pci@0000:02:01.0
       version: 10
       width: 32 bits
       clock: 66MHz
       capabilities: pm cap_list
       configuration: latency=64 maxlatency=64 mingnt=32
       resources: ioport:cc00(size=256) memory:fddfe000-fddfe0ff memory:fdd20000-fdd3ffff
  *-network:2
       description: Ethernet interface
       product: SMC2-1211TX
       vendor: Accton Technology Corporation
       physical id: 3
       bus info: pci@0000:02:03.0
       logical name: enp2s3
       version: 10
       serial: 00:10:b5:c0:43:bb
       size: 100Mbit/s
       capacity: 100Mbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=8139too driverversion=0.9.28 duplex=full ip=192.168.1.4 latency=32 link=yes maxlatency=64 mingnt=32 multicast=yes port=MII speed=100Mbit/s
       resources: irq:22 ioport:ca00(size=256) memory:fddfc000-fddfc0ff
[root@london av]#

ethtool enp2s0

[root@london /]# ethtool enp2s0
Settings for enp2s0:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Speed: 100Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000033 (51)
                               drv probe ifdown ifup
        Link detected: yes
[root@london /]#

Interface config

root@london stats]# cat /etc/sysconfig/network-scripts/ifcfg-enp2s0
TYPE="Ethernet"
BOOTPROTO="none"
IPADDR="192.168.1.4"
NETMASK="255.255.255.0"
GATEWAY="192.168.1.101"
DNS1="192.168.1.101"
DNS2="8.8.8.8"
IPV4_FAILURE_FATAL="no"
NAME="enp2s0"
UUID="d9ca6b87-aa3c-4581-bfa8-4fcbebd78e06"
DEVICE="enp2s0"
ONBOOT="yes"
NM_CONTROLLED="no"


[root@london stats]#

The server role is samba shares No IPTables, SELinux is off.

So the questions are:

  1. What is the root cause of this issue ?
  2. What way should I dig in to resolve this issue ?
  3. Whether I should follow to these steps to resolve issue or not

In case that additional info is needed please tell me. All advises are welcome. Please help. Thanks in advance.

  • I think it's a driver or hardware bug. You can write a miniscript to check network and restart it when needed. – Ipor Sircer Nov 14 '16 at 21:04
  • It looks losely like a dupplicate IP, how does host get it's IP ? dhcp ? static ? – Archemar Nov 16 '16 at 15:10
  • BOOTPROTO="none" IPADDR="192.168.1.4" means that do not use DHCP for this interface and use static IP. Specified IP is out of DHCP range – Raphael Nov 16 '16 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.