5

I've got a very weird SSH behavior and it's making me a bit nervous. Basically the progress meter is totally bogus: so bogus that it's nearly useless. And I fear that there may be something else going on (more on this below).

Basically I only have a 60 KB/s upload link or so, yet every single scp I initiate starts saying it's doing 2 MB/s.

Then it invariably tries to "fix" that number, slowly converging toward the real value.

Then, for "big" files (a few MB or more), it invariably stalls at 100% for many seconds (and eventually it succeeds and I get back to the prompt).

The output looks like this:

...
test.tgz 16% 2112KB   2.1MB/s   00:04 ETA
test.tgz 17% 2208KB   1.7MB/s   00:06 ETA
test.tgz 18% 2320KB   1.2MB/s   00:08 ETA
test.tgz 19% 2448KB   1.1MB/s   00:08 ETA
test.tgz 20% 2576KB 942.2KB/s   00:10 ETA
test.tgz 21% 2704KB 697.3KB/s   00:14 ETA
test.tgz 22% 2832KB 576.3KB/s   00:16 ETA
test.tgz 23% 2960KB 478.3KB/s   00:20 ETA
test.tgz 24% 3088KB 399.0KB/s   00:23 ETA
test.tgz 25% 3216KB 334.7KB/s   00:27 ETA
test.tgz 26% 3344KB 282.6KB/s   00:32 ETA
test.tgz 27% 3472KB 240.4KB/s   00:37 ETA
test.tgz 28% 3600KB 185.6KB/s   00:48 ETA
test.tgz 29% 3728KB 161.9KB/s   00:54 ETA
test.tgz 30% 3856KB 142.7KB/s   01:01 ETA
...

(it's on a single line: I did paste multiple lines here, which I got using scp -vvv to better explain my issue).

At the end, when the output is invariably stuck at 100%, there's actually quite a lot of data missing: I've checked this on the other end and the file is definitely not there in its entirety yet.

All the measurements in percentage are basically pointless: when it says that 80% of a 12 MB file are there, there are only about 65% on the server.

What can explain this?

I'm posting here because I was wondering how these numbers would look like if a man-in-the-middle attack was happening close to my system (maybe on a compromised router or something).

I'm using Linux and SSH / SCP since many years and I don't remember that these numbers were that off.

EDIT

Download works as expected: if I scp from the remote host to my computer, then the %, ETA and KB/s are all correct.

Improve this question
5
  • Does it work OK for smaller files? Does the transfer finish if you let it run long enough? Do you get the same behavior when downloading? The high speeds at the beginning are normal, I have always seen that when using scp.
    – terdon ♦
    Nov 14, 2013 at 23:54
  • @terdon: yup, the transfer finishes if I let it run long enough (I can see the file "growing" on the remote host). The smaller the file, the less the system is "stuck" at 100%, but it's still stuck. For tiny files it's near instantaneous so I can't really tell. So it "works", but the numbers (both the %, the ETA and the speed) are all just totally bogus. It's conforting to read your comment that high speed at the beginning are normal. Download works normally: starts at the correct speed right away, with correct estimates, etc.
    – Cedric Martin
    Nov 15, 2013 at 0:00
  • Yeah, I've often wondered about that. I have always thought that it is an issue of the scp protocol but don't really know. In any case, what you describe is normal for me. My transfers always starts fast then go down to the kind of speed I would expect. I also always get a pause after 100% but before the transfer finishes. Not too long though, just a few seconds, more for larger files. Mind you, my ETA tends to be OK, it changes according to the speed and I also have the same thing when downloading but I have a crappy connection.
    – terdon ♦
    Nov 15, 2013 at 0:04
  • @terdon: oh I see... It's weird because I'm pretty sure I didn't have the problem (at least not that bad) with other systems. I've also read posts about people having this issue with only certain hosts (despite all the hosts being on the same network). It's a bit weird :-/ But then seen that you (and others) do see the same, I'm a bit less worried. I'd still like to know why it's that way and if it can be "fixed" somehow.
    – Cedric Martin
    Nov 15, 2013 at 0:06
  • I just ran some tests and can confirm that i) I have the same thing only when uploading (downloading is fine, sorry, my bad) ii) same problem uploading to a local machine on my home LAN or to a remote server and iii) copying between two remote servers (which are on the same remote LAN) works fine. So, my guess is that it boils down to the quality of our network cards but I really don't know. The servers I am copying between are both relatively high end, professional machines so I am guessing their network card is better somehow.
    – terdon ♦
    Nov 15, 2013 at 0:38

2 Answers 2

Reset to default
11

This behaviour is easily explained through output buffer size and TCP window settings.

First, when receiving data, you either have the bits or you don't. Your local scp knows how much it is expecting and how much it has received so far, so it can give you an accurate assessment of the progress and estimated time remaining.

When you are sending data, you don't have any information about how much of that data has actually reached the receiver yet. Your local machine will have an output buffer that holds data after it has been "sent" by the application (scp) and before it is actually transmitted on the network. Additionally, TCP allows a certain amount of data to be "in flight" between the sender and the receiver.

When sending data, scp only sees how much it has handed off to the OS for eventual transmission. Filling up the output buffers happens really quickly, so that's why scp measures a high transmission rate initially. As the transfer progresses, this value converges toward the real transmission rate. After you have handed all the data off to your OS, it still has to reach the other side and that's why it appears to be "stuck" at 100% for several seconds at the end.

Modern OSes and TCP networks have increased the TCP window size (see TCP window scale option) to account for "long fat networks" with both high bandwidth and high latency. That is why you may be seeing this behaviour more often than you have in the past.

Improve this answer
5
  • That makes a lot of sense, +1. I take it then that the long pause after reaching 100% is the time it takes to transmit the last data in the buffer? Also, why don't I see this behavior when copying between high-end servers?
    – terdon ♦
    Nov 15, 2013 at 1:19
  • 1
    @terdon: The behaviour is still there when copying between any two servers, it just might be less noticeable depending on the performance characteristics of the network between them.
    – Greg Hewgill
    Nov 15, 2013 at 1:51
  • @GregHewgill: +1, thanks for that explanation. So you mean that when scp is "stuck" at 100% it's because 100% of the data has already been sent to the OS? And then it begs another question: how does scp know that it's not done yet? I mean, if scp believes that 100% of the transfer has left, how comes it knows that the file ain't transfered fully yet? And if scp knows that it's not fully transferred, why can't it display the "correct" percentage !? (as a sidenote in a way it'd be "better" if it showed 99% instead of 100%: it would be a little less confusing)
    – Cedric Martin
    Nov 15, 2013 at 2:48
  • @CedricMartin: scp asks the OS to report back when 100% of the data has been successfully received by the other end. If scp completed earlier, then there could be a transmission problem that wouldn't be detected by scp (because it already finished). scp knows that it's not done yet, but the OS doesn't make the information about how much has been received by the other end available to applications (that is, that information is not part of the socket interface).
    – Greg Hewgill
    Nov 15, 2013 at 2:57
  • @GregHewgill ioctl with SIOCOUTQ should get the amount of buffered data. At least according to the docs in tcp(7), haven't actually tried it. But scp can't do that, since ssh is actually handling the socket.
    – derobert
    Mar 30, 2015 at 14:35
0

I recently came across a very similar issue where WireShark was showing TCP ZeroWindow errors any time I would try to push data over ssh beyond a certain rate. I finally traced this back to something in my route screwing up with IP quality of service handling. Adding this config to my sshd_config and ssh_config fixed it:

IPQoS=af21 cs1

These settings will be the default in a newer SSH version than I currently have: https://www.openssh.com/txt/release-7.8

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .