Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

The RHEL 5 documentation is good about bonding pairs of ports in various ways.

The Oracle team where I work want to get more complex than that. They want an active bonded pair, with a third interface ready as a passive backup.

Has anyone tried bonding at this level of complexity?

I thought about trying to layer bonding configurations. Could I configure bondeth0 as an active pair combining eth0 and eth2, and then create a bondeth1 with bondeth0 as active and eth1 as passive?

So,

  • eth0 + eth2 = bondeth0, configured as an active pair.
  • bondeth0 + eth1 = bondeth1, configured for active/passive.

Has anyone ever tried that?

EDIT: Here's a more comprehensive description. These are two hosts in an Oracle 11g RAC/RDBMS cluster. eth0 is the unrouted management connection between the two hosts. eth1 and eth3 go from each host to its corresponding switch. eth2 goes from the host to the opposite host's switch. This way, no single networking failure takes down a host. Here's a rough drawing:

 __________ eth1           ____________
 | Host 1 |================| Switch 1 |
 |________| eth3           |__________|
eth2 |   |                      |
     |   |                      |
     |    \-eth0--------eth0-   |
     |                       \  |
     |                       |  | eth2
 ____________         eth3 __|_______
 | Switch 2 |==============| Host 2 |
 |__________|         eth1 |________|

I don't need to worry about eth0; it's working just fine. The RHEL Documentation covers the basics of bonding well. Many examples of bonding are available. I've even found a great description of the six bonding modes.

What I need to know is how to combine two modes across three ports.

share|improve this question
    
You appear to be configuring this for enterprise purposes, this question is probably more appropriate for ServerFault. You seem to be doing your research and are capable of describing your problem well, so you'd be more than welcome over there. –  suprjami Nov 17 at 22:14

1 Answer 1

up vote 1 down vote accepted

You cannot do this, as you cannot bond bonds.

What you should do is configure three ports in a Mode 4 bond to each switch, and use the ad_select=count bonding option.

Each switch will negotiate a separate Aggregator ID, and only one Aggregator can be used at a time, so all traffic to and from a given Linux system will go via one switch.

This will give you an "uneven" bond, where one Aggregator has two cables, and one Aggregator has one cable, giving the concept of a "primary" and "secondary" switch. The primary Aggregator will always be selected, but if the primary goes down then the secondary kicks in automatically.

You also have a requirement of dedicated Oracle RAC control traffic. The last port (eth3) will be used for this. (Note: I don't know if Oracle support using crossover for this, I'm just going off your requirements)

Network Diagram

Note that Po on the switch denotes PortChannel, or an LACP bond.

.-------------.    .-------------.
|   Switch 1  |    |   Switch 2  |
|-------------|    |-------------|
| Po1  | Po2  |    | Po1  | Po2  |
'-=--=-+-=--=-'    '-=--=-+-=--=-'
  |  |   |  |        |      |   
  |  |   |  '-----,  |      |   
  |  |   '-----,  |  |      |   
  |  |         |  |  |      |   
  |  |   ,-----------'      |   
  |  |   |  ,-------------- | -,
  |  |   |  |  |  |         |  |
  |  |   |  |  |  '-----,   |  |
  |  |   |  |  '-----,  |   |  |
  |  |   |  |        |  |   |  |
 .=--=---=--=.      .=--=---=--=.
 |e0|e1|e2|e3|      |e0|e1|e2|e3|
 +--+--+--+--+      +--+--+--+--+
 |   bond0   |      |   bond0   |
 |-----------|      |-----------|
 |  Host 1   |      |  Host 2   |
 '-----------'      '-----------'

Or to think about it another way:

Host 1 eth0 = bond0 ---- Switch1 Po1
Host 1 eth1 = bond0 ---- Switch1 Po1
Host 1 eth2 = bond0 ---- Switch2 Po1
Host 1 eth3 ------------ Host 2 eth3

Host 2 eth0 = bond0 ---- Switch1 Po2
Host 2 eth1 = bond0 ---- Switch1 Po2
Host 2 eth2 = bond0 ---- Switch2 Po2
Host 2 eth3 ------------ Host 1 eth3

Configure each bond with BONDING_OPTS="miimon=100 mode=4 ad_select=count"

If all your external traffic goes through a gateway, add xmit_hash_policy=layer2+3 to the bonding options to load balance, and configure Layer 3 load balancing on the switch's PortChannel interface as well.

share|improve this answer
    
Thanks for the ideas! So, eth1 and eth3 get a Mode 4 bond, and eth2 gets a separate single-interface-bond, also using Mode 4? –  dafydd Dec 5 at 3:38
    
No. Each system has all 4 interfaces in bond0 which runs in Mode 4. eth0 and eth1 go to one switch, and that switch has an LACP bond configured on those two ports. eth2 and eth3 go to the other switch, and that switch has an LACP bond configured on those two ports. I can edit my answer to include a diagram if you like? –  suprjami Dec 5 at 10:17
    
I added a diagram. –  suprjami Dec 5 at 10:40
    
Thank you. The eth0 "network" isn't for general network traffic. Oracle RAC requires a private, unrouted network within a database cluster for it to perform its Database Node management. It's how Oracle RAC can tell which node is performing a particular database query, etc. (It's much more complex than that, but I'm not a DBA; I would have to look up the DBA-level complexities.) –  dafydd Dec 6 at 19:14
    
Ah sure, sounds like heartbeat or other cluster control traffic. You could add another interface for this, or just take one of the existing interfaces out of a bond. If you did that so bond0 had three interfaces, then you'd still have automatic failover if a switch went down, and the PortChannel with two cables would always be the "preferred" traffic path, so you'd have a "primary" and a "secondary" switch. –  suprjami Dec 7 at 0:39

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.