ssh very slow connection

Question

I have several remote systems, and one of them, a linode running debian, is very slow to ssh into - it takes approximately 20-25 seconds every time. This seems to have happened relatively recently. I have tried setting GSSAPIAuthentication to no or to yes as suggested in several answers to similar questions, and it doesn't make a difference. It also doesn't make any difference if I login using the fqdn or the ip address. I have the same delay sshing from either my local linux box or my local Macintosh. I have no such delay sshing from the linode to the local linux box. I have another remote system using the same version of Debian and I can ssh into it in 2 seconds. The only difference between the /etc/ssh/sshd_config files on the two Debian boxes is that the fast one doesn't allow passwords and also specifies a list of allowed ciphers.

If I login using ssh -vvv root@linode, the delay happens at the part marked with >>>>>>

debug2: key: /root/.ssh/id_ecdsa ((nil))
debug2: key: /root/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50

>>>>>>

debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug3: send_pubkey_test

(This is only a partial log - full log available on request)

I can't find anything about the login in /var/log/auth.log or /var/log/syslog during the delay time - afterwards I just get

Jul 27 13:46:43 linode sshd[23049]: Accepted publickey for root from 199.241.27.237 port 51464 ssh2: RSA 89:08:ef:44:48:a4:84:b7:0a:de:14:65:1b:d9:86:f8
Jul 27 13:46:43 linode sshd[23049]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jul 27 13:46:43 linode systemd-logind[3235]: New session 10361 of user root.

Answer 1

If creating the connection is slow, but it is normal speed after being created, you will most likely have a problem that the server is doing a reverse DNS lookup for the client and that, for some reason, it fails.

In general, when debugging this, you can also try to login from two terminals. With the first login look at the sshd log on the server, while you are trying to login from the second. That gives you more information about what the server is doing (or waiting for).

You can try to find proof for this for the cause being reverse DNS lookup by setting one, or both, of the following in /etc/ssh/sshd_config:

UseDNS no
UsePAM no

and see if that speeds up creating the connection. If it does you can often leave things that way until solved (if you care about that).

If this is a reverse DNS lookup problem, this depends on the DNS server the machine that you login to is using. According to Wikipedia not all IP addresses have a reverse entry, as this is not an actual standards requirement. But more likely this is some configuration issue.

Answer 2

On Debian / Ubuntu systems the trick is to kick out the "avahi-daemon" from the system and the problem is gone.

apt-get -y purge avahi*