2

I've noticed when I SSH into a remote machine over a slow link, SSH seems to "stick" after a relatively large amount of data is transferred.

For example, typing text over the link is fine, but running ls /etc will freeze the connection for a few minutes.

I would understand if the delay was because I was maxing out the connection speed and it returned to normal once the data had been transferred, but the connection freezes for far longer than you'd expect for the amount of data actually involved.

The weird thing is that if I open two SSH connections to and from the same machines, when one of them has frozen, the other one still works fine. So I can't be maxing out the connection speed, otherwise they would both freeze at the same time.

There is no traffic shaping active on either box or (as far as I can tell) the routers in between, so it shouldn't be anything dropping packets to keep the average transfer speed within a certain range.

Can anyone suggest anything that could cause this kind of behaviour, or anything further to check?

The same thing happens with scp and sshfs, with scp reporting a huge transfer rate (many MB/sec, then the speed slowly falls back to stalled for a few minutes, then if I'm lucky it'll repeat until the file finishes transferring.) sshfs works but often (not always) after a file is saved the mount point is non-responsive for a few minutes, temporarily blocking any program that tries to access it.

EDIT: I tried using iperf and get some interesting stats:

Local:  0.0- 0.9 sec   256 KBytes  2.25 Mbits/sec
Remote: 0.0- 7.0 sec   256 KBytes   302 Kbits/sec

Local:  0.0-15.9 sec  1.00 MBytes   529 Kbits/sec
Remote: 0.0-16.0 sec  1.00 MBytes   524 Kbits/sec

It looks like below a certain amout of data, the local end can send a lot faster than the remote can receive. I guess this isn't an SSH problem after all. I will investigate some TCP settings that might adjust this, but if anyone knows any for sure please advise!

  • 1
    Wild guess: Do a top or ps axu, and see if any ssh-related process is stuck in D state. – dirkt Dec 21 '17 at 8:09
  • @dirkt: No zombie processes, and I should clarify that if I leave it sit for long enough everything comes good again, for a while... – Malvineous Dec 21 '17 at 8:13
  • I meant D (uninterruptible I/O), not Z (zombie). Maybe I should explain the reasoning: Your disk I/O may be slower than your network speed, so when the buffers are filled, you have to wait until they are written out. Additionally, I and other people experience mysterious delays of up to several minutes with processes stuck in D - I thought yours might be another case. – dirkt Dec 21 '17 at 8:27
  • Oh my apologies - haven't tested this but I'm sure it can't be disk I/O, because when I'm on site and using the console there's no problem. The data is on an SSD, and even something that uses no disk like cat /proc/cpuinfo will cause the severe lag when run remotely over SSH. – Malvineous Dec 21 '17 at 8:34
  • 1
    The mysterious thing is that a process is stuck in D even though there's no disk I/O. So test it. Assuming that is the case, try dropping caches with echo 3 | sudo tee /proc/sys/vm/drop_caches - that helps if it happens here. – dirkt Dec 21 '17 at 8:52
6

Your SSH connection is exceeding the MTU size somewhere between client and server, and Path Maximum Transmission Unit Discovery is not working. (This is one of several reasons that blanket prevention of ICMP traffic in the name of security is a bad idea.)

Further reading

  • Looks like this might be the case, as tracepath times out unless I lower the MTU on my NIC. However even with the lowered MTU SSH still freezes intermittently, so it looks like there's something else in the path that is dropping ICMP traffic and breaking MTUPD. – Malvineous Dec 22 '17 at 0:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.