I want to create a mesh network with 10GBit Ethernet links between servers and a switch stack capable of distributed trunking.

In this scenario for a number n>2 (but typically 2<n<6) imagine:

  • 1 Switch stack with at least n physical members, capable of distributed trunking on 10GB links.

  • n hosts, each with

    • n+1 10GB interfaces, of those using
      • n-1 for connecting directly to all other n-1 hosts to create the physical full mesh between the servers. Example pictures below
      • 2 connections in one bond0 to the switch stack, to different members.
  • k number of VLANs

GraphViz PNG rendering(1) of fully meshed server node constellation with n=5:

/*_5_Server_Nodes_Network_Mesh_°°°*/ graph n5 { node[shape=plaintext,height=0,width=0,margin=0] edge[len=0.6] a-- b-- c-- d-- e-- a-- c-- e-- b-- d-- a }5 Node Server Mesh

Note: To keep the graphs small, I chose n=3 and k=2 for the rest of this question, because 3 Nodes in a triangle formation are fully meshed, and the same solution should apply.

/*_3_Server_Nodes_Network_Mesh_°°°*/ graph n3 { node[shape=plaintext,height=0,width=0,margin=0] edge[len=0.4] a -- b -- c -- a } 3 Node Mesh Network

Lets say, the cards used for these connection are eth0&eth1 ({0..(n-2)})

I also need to be establish k zones, VLANs(blue). Say VLANs M, and N:

/*_3_Server_Mesh_with_VLAN__°°°°°°*/ graph n3 { node[shape=plaintext,height=0,width=0,margin=0] edge[len=0.4] a -- b -- c -- a edge[color=blue] a--{aM aN}; b--{bM bN}; c--{cM cN}; }enter image description here

2 more of the 10GBit Ethernet Ports (here eth2&eth3) on each hosts are bonded(brown) for upstream connection to outside world:

/*_3_Server_Mesh_w/_VLAN_and_bond0__*/ graph n3 { node[shape=plaintext,height=0,width=0,margin=0] edge[len=0.3]; a -- b -- c -- a edge[color=blue] a--{aM aN}; b--{bM bN}; c--{cM cN}; edge[color=brown,len=0.4,penwidth=2] a -- au; b -- bu; c -- cu;} 3_Server_Mesh_w/_VLAN_and_bond0

Both ports of the bond0 interface to the different members of the upstream switch stack, that itself uses a ring of 10GBit connections(green) for stacking: /*_3_Servers_w/VLAN+Stack*/ graph n3 { node[shape=plaintext,margin=0, height=0,width=0] edge[len=0.3] a -- b -- c -- a edge[color=blue,len=0.3] a--{aM aX}; b--{bM bN}; c--{cM cN}; edge[color=brown,len=0.4] a -- au; b -- bu; c -- cu; edge[color=brown,len=1.1] {cu[shape=point] au} -- S1 {au[shape=point] bu} -- S2 {bu[shape=point] cu} -- S3 edge[color=green,len=2.9] S1 -- S2 -- S3 --S1 } 3_Servers_w/_VLAN_and_Stack

And all the users would would be connected to (networks on) the "outside" of S1..S$n

Logically, the Switch stack S1..S$n behaves like one big Switching Unit. So, the servers a,b,..,n see the following structure, with everything else connected to S

/*_3_Server_Mesh_w/_VLAN_and_Stack__*/ graph n3 { node[shape=plaintext,height=0,width=0,margin=0] edge[len=1]; a -- b -- c -- a edge[color=brown,len=0.3,penwidth=2] S -- {a b c} edge[color=blue,len=0.3] a--{aM aN}; b--{bM bN}; c--{cM cN}; } enter image description here

Now we connect some Virtual Machines(red) to these VLANs:

/*_3_Server_Mesh_with_VLAN_clients__*/ graph n3 { node[shape=plaintext,height=0,width=0,margin=0] edge[len=1]; a -- b -- c -- a edge[color=brown,len=0.3,penwidth=2] S -- {a b c} edge[color=blue,len=0.3,penwidth=1] a--{aM aN}; b--{bM bN}; c--{cM cN}; edge[color=red,len=0.3,penwidth=1] v1--{aM aN}; {v2 v7}--bM; v3--{cM cN}; v4--aN; {v5 v6 v8}--bN; v9--cM; } 3_Server_Mesh_with_VLAN_clients

This is the sort of functionality often seen in context with SDN, SPB, TRILL, VxLAN, all of which seem relevant after checking them out. Of those Shortest Path Bridging(802.1aq) sounds like the best fit. But where are the implementations for linux, and how do I use them on Debian to build a fully meshed brouter?

So what could I install/configure on each node to achieve the desired constellation?

(1) Graphviz renderings were created with neato -Tpng -O file.dot && browser file.dot.png

  • Can't you simply install BGP on each node so that they can share their routing information? – Julie Pelletier Aug 27 '16 at 16:22
  • That's one unusual network. Unless you have highly special requirements (which you did not include in the question) I strongly suggest you talk to a network engineer. – AlexP Dec 23 '16 at 20:35
  • @AlexP: Unusual? The above is just a formal description of a fully meshed network illustrated for the simplest case, a triangle formation. This mesh uses 10G connections for meshing, and another 2 per host to connect to "a big external switch". This is a fairly easy setup. I have setup such networks. This question is about avoiding the shortcomings of STP, as mentioned. The official successor is SPB. A competing standard is TRILL. I want to get one of them to work. But I would also accept something with the same gains as sought with SPB/TRILL. – Alex Stragies Dec 23 '16 at 21:25
  • ~2.5h before the end of the bounty somebody started a thread on the B.A.T.M.A.N. mailing list asking for precisely this scenario :) – Alex Stragies Dec 30 '16 at 19:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.