3

I'm trying to benchmark the network connection between 2 ubuntu servers on which I don't have root privileges.

I did manage to download and start iperf-2.0.9 on both boxes.

My problem is I can't open ports (through the firewall).

So the client can't connect to the server.

Is there any way I can make this work over a ssh connection between the two? forward a port or something?

Any solution is greatly appreciated!

Note: I did scp several files back and forth... But that's not even close to what iperf can give me

  • 1
    The point of using iperf is lost using it in a ssh tunnel for several reasons, even if you manage to do it. – Rui F Ribeiro Jun 22 '16 at 12:52
5

iperf allows the user to specify the ports in three places- one where the server listens, one where the client connects, and one where the client spawns a mini server for the -d/--dualtest option. We need all three for this.

While it might be possible to do it with fewer, I found it was easier to specify all the ports so I could better keep track of them. In this setup, I will assume a setup that looks something like this:

-----------            -------          -------
| Control |  SSH #1,2  | Box |  SSH #3  | Box |
|   Box   | ---------> | #1  | -------> | #2  |
-----------            -------          -------

The "control box" might also have direct access to SSH on box #2, but we don't need it. For this, Box #2 will be the iperf server listening on 7001 and box 2 will be a client listening back on port 7002. These can be any accessible ports, I chose those two at random.

First, connect to box #1. Next, you need to connect through to Box #2. In this nested session, you will need to create two port tunnels: one forward and one reverse. The ssh options that do this are -L7001:localhost:7001 for the forward and -R7002:localhost:7002 for the reverse. Because iperf expects the ports to be located on a remote host, each tunnel has to be symmetrical (the same port number at both ends of the tunnel). Then, start the iperf server listening at port 7001 (iperf -s -p 7001).

It might look something like this:

me@control$ ssh box1.example.com
box1$ ssh -L7001:localhost:7001 -R7002:localhost:7002 box2.example.com
box2$ iperf -s -p 7001
------------------------------------------------------------
Server listening on TCP port 7001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------

After that's started, open a second session to Box #1. Here, start an iperf client to localhost at port 7001, with the listen-back port on 7002 (the default listen-back port is 5001 like the server). This means that the client will attempt to connect with an iperf server on localhost:7001, which SSH grabs and sends off to box #2. It then starts a "mini" iperf server listening on 7002. Once the connection from the client to the server is initiated, the iperf client tells the iperf server to connect back on port 7002. The server notes that the inbound connection is coming from 127.0.0.1 (or ::1 depending on the configuration), so it starts a "mini" client that will connect to 127.0.0.1:7002. Because we also have the reverse forward in place, ssh snaps up this connection too and sends it off to box 1.

Your second session might look something like this:

(side note for this example: I set the time to 30s for a different test; the default will suffice)

me@control$ ssh box1.example.com
box1$ iperf -c localhost -p 7001 -L 7002 -d -t 30
------------------------------------------------------------
Server listening on TCP port 7002
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to localhost, TCP port 7001
TCP window size: 4.00 MByte (default)
------------------------------------------------------------
[  3] local 127.0.0.1 port 37014 connected with 127.0.0.1 port 7001
[  5] local 127.0.0.1 port 7002 connected with 127.0.0.1 port 51806
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  1.26 GBytes   361 Mbits/sec
[  5]  0.0-30.2 sec  1.23 GBytes   349 Mbits/sec

When the client is finished testing, your server window might look something like this:

...
box2$ iperf -s -p 7001
------------------------------------------------------------
Server listening on TCP port 7001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 127.0.0.1 port 7001 connected with 127.0.0.1 port 41997
------------------------------------------------------------
Client connecting to 127.0.0.1, TCP port 7002
TCP window size: 4.00 MByte (default)
------------------------------------------------------------
[  6] local 127.0.0.1 port 46864 connected with 127.0.0.1 port 7002
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-30.0 sec  1.23 GBytes   351 Mbits/sec
[  4]  0.0-30.2 sec  1.26 GBytes   359 Mbits/sec

WARNING: SSH WILL distort the perceived connection speeds. Running iperf without SSH between the same two boxes yielded this (the boxes are in the same roles):

Client:

box1$ iperf -c box2.example.com -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to box2.example.com, TCP port 5001
TCP window size:  306 KByte (default)
------------------------------------------------------------
[  3] local 172.20.0.1 port 45722 connected with 172.20.0.2 port 5001
[  5] local 172.20.0.1 port 5001 connected with 172.20.0.2 port 60909
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.01 GBytes   866 Mbits/sec
[  5]  0.0-10.0 sec   823 MBytes   689 Mbits/sec

Server:

box2$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 172.20.0.2 port 5001 connected with 172.20.0.1 port 45722
------------------------------------------------------------
Client connecting to 172.20.0.1, TCP port 5001
TCP window size:  306 KByte (default)
------------------------------------------------------------
[  6] local 172.20.0.2 port 60909 connected with 172.20.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec   823 MBytes   690 Mbits/sec
[  4]  0.0-10.0 sec  1.01 GBytes   864 Mbits/sec

I tried messing with the TCP window settings, the length of the buffer, TCP_NODELAY, and using multiple SSH sessions, but the overhead was still present. I also tried HPN-SSH, but I actually got better performance over regular SSH, so I think there's a setting I missed when I was setting up HPN. Running the iperf connections in simplex rather than duplex (option -r/--tradeoff (Do a bidirectional test individually)) got results closer to the link speed, but still with significant SSH overhead.

That all being said, if you need to create bridge between these two machines and measure the capacity of that bridge, this solution is perfect. If you are trying to measure the raw throughput between these machines, the numbers these tests provide will be less (and probably much less) than the link speed.

  • With an ssh tunnel over a 30kB/s connection I get no overhead at all with default settings everywhere. Maybe it's CPU limited for higher speeds. – Nobody Jan 28 '17 at 12:25
  • I'd believe that it's a CPU issue, but I don't think it's intentionally limited. I remember an issue I had once with a gigabit wifi router that couldn't push more than 200Mb/s when DD-WRT was installed. If I have some time this semester, I might revisit it to see if I can get it up to full bandwidth. – computergeek125 Jan 28 '17 at 17:42
  • I get significantly lower SSH overhead by using the options: Ciphers arcfour128 Compression no - which I would put in my .ssh/config entry for the server, for convenience. Note that the arcfour encryption-methods are disabled by default in recent SSHD versions, as they may be less secure. – esskov Nov 1 '17 at 20:12
  • Fair enough. I didn't mess too much with the ciphers I used, and if I remember correctly, one of these boxes was VM running on an unknown ESXi host in a building that was actually across the river and then some (which I didn't know at the time), so I didn't account for inter-building traffic bottlenecks. Obviously I didn't get time that semester to fix it XD – computergeek125 Nov 2 '17 at 0:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.