5

I have got a 300mbit symmetrical fiberoptic line, and I have got to transfer a 51MBYTE tar file from HOST A (fiberoptic 300mbit) to HOST B (digitalocean machine with more than gigabit bandwith).

On both side I have got nice speedtests results (300mbit on A, 700 on B) but when I scp from A to B I have got this:

assets.tar            100%   51MB 220.3KB/s   03:55

only 220kbit of maximium speed.

But if I do from HOST B to A I have got a really nice result:

assets.tar            100%   51MB   8.4MB/s   00:06    ***REALLY GOOD SPEED***

What can be the issue?

3
  • What if you try with some speed limit like scp -l 200000 ? Check if this helps.
    – Vombat
    Jun 9, 2016 at 18:39
  • Is A your home box? in this case 300 mbit/s is download speed only. To save some money many internet providers limit upload speed.
    – Putnik
    Jun 10, 2016 at 8:32
  • You are muddling your Bytes ("B") and bits ("b"). For example, "when I scp from A to B i have got [...] only 220kbit of maximium speed". The 220KB/s is 2.2Mbit/s. Jun 11, 2016 at 7:35

3 Answers 3

3

If using openssh or some derivative, check your sshd & ssh IPQoS defaults.

IPQoS default was lowdelay throughput before version V_7_8_P1, then changed to af21 cs1 (first is TOS tag for interactive sessions, second for bulk transfers).

On our WAN routes (even within our provider's LAN), CS1 is prioritized very low, leading to very poor transfer speeds for scp, sftp and rsync via ssh. Changing IPQoS back to the previous default solved this issue for us.

For details & reasons of that change, see the openssh commit info.

Don't rely on your openssh package version, some distributions apparently decided to keep the old defaults. Affected distributions for us were all Redhat based distributions (i.e. CentOS, AlmaLinux, RockyLinux) beginning with release 8.

Check your sshd config by sshd -T | grep -i IPQoS.

To test, try scp -o IPQoS=throughput ….

To persistently revert to the previous defaults:

  1. Add to /etc/ssh/sshd_config:

    IPQoS lowdelay throughput
    
  2. Add a default or specific host rule to /etc/ssh/ssh_config, e.g.:

    Host *
      IPQoS lowdelay throughput
    
  3. Activate the new config by systemctl reload sshd.

YMMV and depend on your actual network topology & policies, af21 is also supposed to be better for interactive sessions, so I recommend testing different TOS tags.

1

SCP is very simple tool to simple copy files back and forth. It was not designed to super-fast speeds and it has really small buffers on both sides.

If you aim for performance, you should use sftp or rsync.

About the speed measures, lets draw some diagram:

[host A]   --- ??? mbit  ---    [host B]
        \                      /
         \ 300 mbit           / 700 mbit
          \                  /
           [speedtest server]

The data between two hosts does not have to go through the speedtest server you measured the speed against (and they probably don't go) so these measures are irrelevant for your case. If you want to measure speed between these two hosts, you really need to measure traffic between these two and not anything else. There might be some line that is assymetric or different way limited.

5
  • ISTR that today's OpenSSH uses sftp under the covers for the scp command.
    – slim
    Jun 11, 2016 at 6:06
  • @slim, no, it does not. Check the source.
    – Jakuje
    Jun 11, 2016 at 7:11
  • I'm certain I've seen systems where /usr/bin/scp is a symlink to /usr/bin/sftp (and sftp handles parmams differently if argv[0] is scp). However the source you've linked doesn't confirm that, and nor does the Raspbian system that is the only Linux I can reach right now.
    – slim
    Jun 11, 2016 at 7:32
  • I have never seen such system. They have different synopsis, so you can't symlink them without confusing a lot of users. Also they use different way to handle the server side (subsystem x just command).
    – Jakuje
    Jun 11, 2016 at 7:36
  • 1
    Not sure what you mean there. ls and cat have different synopsis, but on some systems both are symlinks to busybox and most users don't even notice.
    – slim
    Jun 11, 2016 at 7:54
1

I would start by measuring the speed of the dumbest TCP stream possible. A plain FTP (not SFTP or FTPS) would do it. If for some reason FTP doesn't work (firewalls can be an issue), try netcat.

FTP just literally throws bytes at a socket. As long as it's using the full TCP packet size, and we're talking about a single file, you can't use TCP more efficiently. So this will give you a benchmark of what can be achieved between the two hosts.

(There are some UDP-based protocols that can go faster over WAN by avoiding the wait for TCP ACK packets, but none of these is a commonly used standard).

Note, however, that FTP does not compress, so it may take longer for certain kinds of data. This is good for our aim of measuring raw TCP throughput.

If FTP is slow/asymmetric too, then it may just be that there's an asymmetric link along the route between these machines. You could do some further diagnosis by running a Wireshark sniffer on both ends, and checking the trace for lost packets etc.

If FTP is fast and symmetrical, then you have some other problem. Without delving in deep, it's hard to guess, and there's a lot of possibilities. For example, one machine's SSH might be configured to compress while the other is not.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .