Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have some servers with multiple network interfaces, setup with bonding and some VLANs. Whenever I reboot the server, one of the bonded network interfaces is not reachable from other servers, nor can any traffic leave that interface. The status from ifconfig on that interface does indicate that the link is up, however. Simply restarting the network at this point will restore everything to normal.

The fact that everything works as expected after I restart the network makes me think that my configuration is correct, but it is something in the startup order that isn't working right on reboot, but gets straightened out when restarting the network.

I have 7 identical servers with the same setup (other than IP addresses differing), and it happens on all of them, every time they are rebooted.

A bit more details on the setup:

  • Servers: HP ProLiant DL380
  • 6 network interfaces, setup as 3 bonded interfaces named: bondm, bondr, bondt.
  • 4 interfaces are built-in, the remaining 2 are in an add-on PCI card
  • bondm is configured with 2 VLANs
  • bondm is used as the default route
  • bondm is setup to use eth0 and eth2
  • bondm is the interface that is failed on reboot

Update: I've restested this with the exact same configuration and kickstart files, but with SL 6.2 vs 6.3. Everything is fine with 6.2, but I get this behavior with 6.3. Is it due to the different kernels?

Here are some of the relevant config files from /etc/sysconfig/network-scripts:

$ cat ifcfg-eth0 ifcfg-eth2 ifcfg-bondm ifcfg-bondm.132 ifcfg-bondm.832 
DEVICE=eth0
BOOTPROTO=none
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Ethernet
HWADDR=44:1E:A1:03:71:C4
SLAVE=yes
MASTER=bondm
ETHTOOL_OPTS="-s eth0 speed 1000 duplex full"

DEVICE=eth2
HWADDR=44:1E:A1:03:71:C8
NM_CONTROLLED=no
ONBOOT=yes
SLAVE=yes
MASTER=bondm
ETHTOOL_OPTS="-s eth2 speed 1000 duplex full"

DEVICE=bondm
BOOTPROTO=none
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Ethernet
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
BONDING_OPTS="mode=active-backup miimon=100"

DEVICE=bondm.132
BOOTPROTO=none
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Ethernet
IPADDR=192.168.13.19
PREFIX=28
GATEWAY=192.168.13.17
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
BONDING_OPTS="mode=active-backup miimon=100"
VLAN=yes

DEVICE=bondm.832
BOOTPROTO=none
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Ethernet
IPADDR=10.123.94.69
PREFIX=28
DEFROUTE=no
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
BONDING_OPTS="mode=active-backup miimon=100"
VLAN=yes
share|improve this question
    
Could you show config files /etc/sysconfig/network-scripts/ifcfg-*. Is there any clues in messages during boot? –  dchirikov Jan 24 '13 at 18:09
    
I've added the config files. There is nothing obvious in /var/log/messages. Is there anywhere else I should look or commands I should run when its in this state. –  wolfcastle Jan 24 '13 at 19:14
    
Do you have iptables script setup? I saw some scripts before that actually reset nic on boot. –  John Siu Jan 29 '13 at 16:05
    
@JohnSiu, yes we use iptables. Are you referring to something in the /etc/sysconfig/iptables config file, or the service script in /etc/rc.d/init.d ? What should I look for (or post here?) –  wolfcastle Jan 29 '13 at 17:35
    
(1) There are different ways to do it, I am not sure how your site is setup. But the files that contains all the iptables lines. (2) Do a grep bondm /var/log/syslog and see if all bonded interfaces are actually setup at some point but later dropped. –  John Siu Jan 29 '13 at 21:52

1 Answer 1

up vote 3 down vote accepted
+50

modprobe.d

Per the instructions on this RHEL6 site did you create the file /etc/modprobe.d/bonding.conf and add your bondm device to that file?

alias bondm bonding

missing TYPE for 2nd NIC

Also I'm not if it matters but your eth2 device is missing this line:

TYPE=Ethernet

Disabled NetworkManager?

Have you tried disabling the NetworkManager service? Try that and see if the problem persists, reboot to confirm.

% chkconfig off NetworkManager

UDEV

Are you making use of udev on these boxes? I've run into problems where udev has populated a file here, /etc/udev/rules.d/70-persistent-net.rules. This file has had redundant entries for NICs on boxes and I've had to manually edit this file. Mine looks like this:

# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# net device () (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="54:52:00:ff:ff:f5", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

UDEV assigns devices based on MAC addresses, you can force it to assign based on the position the NIC is taking in the PCI bus.

You can use this command to determine your NICs' PCI info:

% for i in /sys/class/net/*;do printf "device: %6s - %s\n" `basename $i` `readlink -f $i`;done
device:    br0 - /sys/devices/virtual/net/br0
device:   eth0 - /sys/devices/pci0000:00/0000:00:1c.5/0000:09:00.0/net/eth0
device:   eth1 - /sys/devices/pci0000:00/0000:00:2d.5/0000:03:00.0/net/eth1
device:     lo - /sys/devices/virtual/net/lo

Based on this output you'd need to populated your own udev rules file:

% cat > /etc/udev/rules.d/70-persistent-net.rules << EOF
ACTION=="add", SUBSYSTEM=="net", BUS=="pci", KERNELS=="0000:00:1c.5", \
    NAME="eth0"
ACTION=="add", SUBSYSTEM=="net", BUS=="pci", KERNELS=="0000:00:2d.5", \
    NAME="eth1"
EOF

NOTE: Also make sure that you remove/disable any pre-existing udev rules file that may already be attempting to setup your NICs.

Bug with CentOS 6.3

I came across this bug on the CentOS issue tracker. The release notes for 6.3 list it as well.

Excerpt from Centos 6.3 Release Notes:

There seems to be an issue when using 802.1q VLANing on bonded (802.3ad) interfaces and certain NICs. See this upstream bugzilla entry and this CentOS bugzilla entry for details. The CentOS-Plus Kernel released with 6.3 contains a patch to fix this issue. Starting with kernel 2.6.32-279.2.1 this issue is fixed.

This issue sounds suspicously like the one you've been dealing with. What kernel are you running? (uname -a).

share|improve this answer
    
Yes, the alias line is present in the /etc/modprobe.d/bonding.conf. I hadn't noticed the difference in the eth2 file. I also doubt it has any impact, but they likely should be the same. I will try to make them the same and retest today or tomorrow. –  wolfcastle Jan 31 '13 at 14:59
    
NetworkManager is definitely off. Hopefully I'll get to test this out today. The odd thing is the ifcfg-ethx for the other interfaces that DO work are the same as eth2 (missing TYPE and BOOTPROTO lines) –  wolfcastle Feb 4 '13 at 15:06
    
I did mention I'm using 6 interfaces (eth0-eth5). I've only posted eth0/2 config because they are the interfaces used in the bonding that is failing. eth1 belongs to a different bond configuration. –  wolfcastle Feb 4 '13 at 15:55
2  
@slm nice udev info and tracking down potential related bug!!1 –  John Siu Feb 7 '13 at 2:39
1  
For the record, I was encountering the CentOS 6.3 bug. After moving to 6.4 when it was released (which included the fix @sim mentioned), the issue went away. –  wolfcastle Nov 8 '13 at 16:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.