17

I have two computer, both with linux, which are currently connected to a router/switch via a 1000Mb/s ethernet port (installed in mb).

This allows me to transfer data at max 120MB/s between two hosts.

As I need to increase this speed, would it be possible to install two 2.5Gb ethernet card in the two PCIe slots of both motherboard (total of 4 new ethernet card), connect them directly with 2 cables (not passing via router/switch), and make both cards to be seen as one logical network interface with a theoretical speed of 5Gb ?

How would I do that?

Also, I see single ethernet cards (PCIx) with two RJ45 port; can these 2 ports be used at full speed at the same time?

6
  • 2
    I would suggest you read en.wikipedia.org/wiki/Link_aggregation first and come back here with much more precise questions. As for your last question it would certainly depend on the device itself. In any case, both ports would certainly trigger the same irq and traffic share the same bus. While having two different devices would, depending on your mb allow 2 different irq as well as 2 different buses.
    – MC68020
    Dec 19, 2023 at 13:31
  • 7
    "1000MB/s ethernet port" - this would be approximately 10 Gb/s. I assume you meant "1000 Mb/s" or "1 Gb/s". Letter case is important - there are 8 b to each B, for example. And then you say that, "This allows me to transfer data at max 120Mb/s" - but this doesn't make sense. Even if we correct your original statement to be "1000 Mb/s" you should be able to transfer data at around 900 Mb/s, not the slow 120 Mb/s you describe. Dec 19, 2023 at 15:59
  • 7
    Next time (or even this time) please edit your question. We shouldn't have to guess what you intend. Dec 19, 2023 at 20:02
  • 8
    To further articulate @ChrisDavies's point, it'd be one thing if you consistently used upper case or lower case but you actually go so far as to switch cases so that what you've typed is always the opposite of what you apparently meant. Dec 20, 2023 at 3:30
  • 1
    Are you seeing 120Mb/s between hosts or is this a calculated maximum? The maximum speed of any given operation (like, a file transfer) will be the maximum speed of the slowest part of the process, including ability to read/write to storage, stream through protocols (maybe NFS or SMB for a file transfer), bus bandwidth, and (including) what you ask about, network link connectivity. The 1Gbps link may not be the thing that is slowing down your operation. Dec 20, 2023 at 15:08

5 Answers 5

25

Yes, it is possible.

You, most likely, would need to install "link bonding driver". A driver which would balance and synchronize packets sent over two (or more) links. It is not a part of default setup in most distributions, but available in repositories. For Ubuntu family, for example, you can get a nice instruction here: https://help.ubuntu.com/community/UbuntuBonding

You can also google words "link aggregation" or "link teaming". While this approach to connection speed and sustainability is used a lot on server side, we still do not have a single term for it...

6
  • 1
    Hint: For throughput you want Mode4/802.3ad (requires capable, configured switch) or Mode0/RoundRobin (any switch). Whether you actually get more throughput depends on the capabilities of the switch ; slightly less cheap ones only switch at line speed, middling ones will switch much faster but often aggregate ports into groups which only switch at line speed (i.e. if in doubt, plug cables from each host into opposite ends of the switch). Or, as you suggest direct connect them.
    – symcbean
    Dec 19, 2023 at 14:44
  • ....just to clarify, I believe 802.3ad won't work without a switch - but you could always test it yourself and report back here.
    – symcbean
    Dec 19, 2023 at 14:49
  • 2
    @symcbean The OP's setup was a direct physical connection, no switch. Dec 20, 2023 at 12:09
  • 2
    @symcbean Being connected to a switch is not a requirement for using 802.3ad (LACP). It requires that the connected peer also implements 802.3ad. That could be active or passive mode (see Linux driver parameter lacp_active). If two Linux PCs have their Bonding driver configured to use 802.3ad mode and at least one of them has lacp_active=on, you should be good to go. If both use passive mode (or no LACP at all), the ports might not come up. The same applies when connecting a PC or server to a managed switch (or other managed network gear). Unmanaged switches might behave differently.
    – riha
    Dec 20, 2023 at 13:54
  • 1
    I just noticed my previous comment is partially incorrect: If neither peer uses LACP, then obviously LACP will not prevent the ports from coming up. Corrected statement: If both peers use passive LACP, or one uses LACP (active or passive) while the other does not use LACP, then ports might not come up.
    – riha
    Dec 20, 2023 at 15:15
10

As I need to increase this speed, would it be possible to install two 2.5GB ethernet card in the two PCIe slots of both mb (total of 4 new ethernet card), connect them directly with 2 cables (not passing via router/switch), and make both cards to be seen as one logical network interface with a theoretical speed of 5GB ?

Yes, as the other answers have covered, this should be doable.

That is, for the very specific case of a point-to-point link between two systems you can do this and achieve (almost) what you want. Both nodes need to be configured identically, and both need to be using bonding mode 0 (balance-rr), because that is the only way that you can achieve true double-throughput in a point-to-point link between two systems with the Linux bonding driver (every other mode will not balance things evenly, or not balance at all).

You will probably not see the exact equivalent of a 5GbE link though, because there’s some extra overhead from the bonding driver (it should get close however).

This only works for a point-to-point link between two systems with an identical bonding configuration on both ends though. For anything else you probably want bonding mode 6 (balance-alb), but your switch needs to be well designed, and that will balance individual connections across the two links (IOW, each connection can only use up to the equivalent of one link, but only about half the connections will be competing for that bandwidth).

All that said, you may be better off just buying a pair of 10 GbE adapters. You would need slightly more expensive cabling, but the cost-performance ratio is better than four 2.5 GbE or two 5 GbE links, because 2.5 GbE and 5 GbE as commonly implemented are just 10 GbE with a slower transmission rate (and thus a lot of the internals are identical). This approach has the distinct advantage that it would give you a huge boost in performance even compared to your planned setup, and it should just work (provided you get a card that is well supported by Linux) with no extra configuration, and it should actually cost somewhat less than twice as much as using two pairs of 2.5 GbE cards would.

Also, I see single ethernet cards (PCIx) with two rj45 port; can these 2 ports be used at full speed at the same time?

The answer to this one is a definite maybe. Using both ports concurrently should always work. You just may not be able to use their full bandwidth (though you will still usually get the latency benefits of their advertised bandwidth).

The reasons are a bit too complicated for this answer, but they have to do with some quirks PCI-e link negotiation and some rather prevalent questionable design choices made by manufacturers of motherboards.

1
  • 1
    It does depend on what slots he has available. If he has 2.0 x4 or better slots free than 10G is the obvious choice, if he only has x1 slots then things get a lot messier.
    – plugwash
    Dec 21, 2023 at 5:27
5

Synology can do specifically what you are referring to, not so much with multiple network cards but with a single multiple [quad] port network card. For example a https://www.synology.com/en-us/products/RS3618xs#specs

https://kb.synology.com/en-ca/DSM/help/DSM/AdminCenter/connection_network_linkaggr?version=7

Unfortunately this setup that I have done on Synology has been through their graphical DSM interface. How it translates into their debian style of linux underneath I do not know. But per your question yes, it is possible and it is has been in practice for a while.

Here is an instructional for bonding:

Network bonding is a method to combine or aggregate network interfaces to provide a logical interface with higher throughput or redundancy.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/configuring-network-bonding_configuring-and-managing-networking

does not matter if it is just one network card with multiple ports, or multiple cards each having one port... every port will show up as some interface name (eth0, eth1, eth2.. eth-n) and it is at that level where you bond.

2

As noted in other answers, this can be done with link aggregation / bonding. I'll not repeat those details, but here are some tips / limitations to work around:

  • Both of the devices on either end of the network cable must support it and have it enabled with the same LACP options. (This applies to both direct to host or to a switch.)
  • Note that if you go through a switch, you can increase the bandwidth of one machine going to multiple machines, and the other machines don't also need to do bonding to take advantage. (But the non-bonded machines will be limited by the speed of a single link on their end.)
  • If you are going through a switch rather than direct, the backbone of the switch must have enough bandwidth to support your aggregate bandwidth. Some cheaper switches only have enough backbone bandwidth for a single full bandwidth connection, but then they probably don't support LACP anyway. Even midrange switches likely can not handle aggregate full bandwidth on every port simultaneously. (Not an issue if you connect direct obviously.)
  • If you use separate cards, the bandwidth can be limited by IRQ conflicts between the cards as well as motherboard bandwidth.
  • If you use multiple ports on the same card, you will also be limited by the number of lanes the card has (but this might be better than IRQ conflicts, and it might not be an issue anyway depending on PCI speed of your motherboard).
  • Due to the nature of the way bonding works and depending on LACP settings, you may not get increased bandwidth from a single connection. Multiple TCP streams to the same or separate machines may be necessary to take full advantage.
  • After setting up the connection, you may find playing with iperf options helpful in testing your potentially increased bandwidth.
1

Yes. You can do that by means of mptcp protocol that is enabled by default in many Linux distros. You have to work on ip tables to configure your connection to use mptcp instead of the default tcp. You can find instructions here https://github.com/multipath-tcp/mptcp

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .