0

I have a situation where I wish to connect two motherboards running Fedora with a PCIe x1 to PCIe x1 cable. The two motherboards are side by side, 1 cm away from each other. The purpose of the connection is to allow an application on Computer 1 to communicate with a different application on Computer 2, as if they were connected by Ethernet.

Computer 1 is connected to the network by Ethernet, and Computer 2 needs to be accessible over the network through Computer 1.

I realise that if I connect the computers together with an Ethernet cable, and bridge the two ports on Computer 1, then I would have a solution. But I have very good reasons to want to connect the two computers using motherboard headers rather than back panel ports.

I also realise that I could insert PCIe to Ethernet devices in each PCIe port and run a short Ethernet cable between them. But I do not want this complexity.

I am wanting to know if it is possible to achieve the solution with a simple port to port cable using the PCIe x1 onboard ports.

An alternative is to use COM ports, but my reading suggests I can get better performance using PCIe??

1

Indeed you would get better performance with PCIe as opposed to COM ports... with an exceptionally fast COM port you might get a transfer speed of 0.92 Mbps, compared to 7880 Mbps you might achieve with a single PCIe v3.x link.

Unfortunately a simple port to port cable is not a possible solution here.

The PCIe ports in the card slots are specifically downstream ports, which must communicate with corresponding upstream ports in the PCIe cards themselves. You are trying to connect a downstream port to another downstream port, which is not allowed by the PCIe standard.

Even if you could solve that issue, it would mean that each of the two computers would have unlimited direct access to the system bus of the other computer. Essentially, those would no longer be treatable as two separate computers, but would be more like a single NUMA system with a pretty serious bottleneck between its halves (if you used just a single PCIe x1 connection). Without any way to synchronize the access between the halves of the system, you would just trade off hardware complexity to gain a very tricky programming problem. As both halves would be regular motherboards designed to run independently, retrofitting even the minimum necessary hardware synchronization might not even be possible.

But the concept you are trying to achieve is not new: it is known as a non-transparent bridge. In the Linux kernel, an infrastructure to support such things has existed since kernel version 3.9. There are hardware implementations of NTB supported right now (you'll note that those mostly tend to be parts of fairly serious server-grade hardware). There is even a ntb_netdev driver that makes a NTB link look like a very fast Ethernet device.

So... to realize your idea, you would have to research a number of existing NTB chips, pick one suitable for your needs, design a PCB for it (handling high-frequency signals!), build it, develop the necessary patches for Linux kernel to support your specific implementation, and finally debug the whole thing.

While a truly wizardly hardware engineer might call this "simple", for ordinary people that aren't planning to fill multiple datacenters with their solution, a short Ethernet cable would be much, much simpler.

| improve this answer | |
  • Thank you for clarifying that telcoM. It will save me from a lot of wasted time. I am most grateful. – Mark Nov 17 '19 at 22:43
  • @Mark, you don't have enough reputation points to up-vote telcoM's answer (which would be one way to say "thank") but since this is your question, you can also accept this answer by clicking on the checkmark next to it. That tells the Stack Exchange system that this Answer solved your problem. Thanks, and welcome to U&L! – Jeff Schaller Nov 24 '19 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.