2

I have this strange behavior occurring with SSH on one particular machine. If I SSH into it, it'll work for a few seconds and then just 'hang'. I can create a new SSH in verbose ssh -v and it'll just hang (no shell prompt gets shown) with the following last output:

debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0

What might be the cause of this?

Full log:

ssh -v chris@192.168.0.37        
OpenSSH_7.9p1, OpenSSL 1.0.2r  26 Feb 2019
debug1: Reading configuration data /home/chris/.ssh/config
debug1: /home/chris/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 5: Applying options for *
debug1: auto-mux: Trying existing master
debug1: Control socket "/home/chris/.ssh/sockets/chris@192.168.0.37-22" does not exist
debug1: Connecting to 192.168.0.37 [192.168.0.37] port 22.
debug1: Connection established.
debug1: identity file /home/chris/.ssh/id_rsa type 0
debug1: identity file /home/chris/.ssh/id_rsa-cert type -1
debug1: identity file /home/chris/.ssh/id_dsa type -1
debug1: identity file /home/chris/.ssh/id_dsa-cert type -1
debug1: identity file /home/chris/.ssh/id_ecdsa type -1
debug1: identity file /home/chris/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/chris/.ssh/id_ed25519 type -1
debug1: identity file /home/chris/.ssh/id_ed25519-cert type -1
debug1: identity file /home/chris/.ssh/id_xmss type -1
debug1: identity file /home/chris/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9
debug1: match: OpenSSH_7.9 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 192.168.0.37:22 as 'chris'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-ed25519 SHA256:Q3IPnF5PorgEAJwAF1EBlFrD4XmttrmsSBgVQKvgaUM
debug1: Host '192.168.0.37' is known and matches the ED25519 host key.
debug1: Found key in /home/chris/.ssh/known_hosts:54
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /home/chris/.ssh/id_rsa RSA SHA256:jZ4wVcWhhtqNdX/SwnrbG7TRfTE9cmm9Ar1PLyCJwuc
debug1: Will attempt key: /home/chris/.ssh/id_dsa 
debug1: Will attempt key: /home/chris/.ssh/id_ecdsa 
debug1: Will attempt key: /home/chris/.ssh/id_ed25519 
debug1: Will attempt key: /home/chris/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key: /home/chris/.ssh/id_rsa RSA SHA256:jZ4wVcWhhtqNdX/SwnrbG7TRfTE9cmm9Ar1PLyCJwuc
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Trying private key: /home/chris/.ssh/id_dsa
debug1: Trying private key: /home/chris/.ssh/id_ecdsa
debug1: Trying private key: /home/chris/.ssh/id_ed25519
debug1: Trying private key: /home/chris/.ssh/id_xmss
debug1: Next authentication method: keyboard-interactive
Password: 
debug1: Authentications that can continue: publickey,password,keyboard-interactive
Password: 
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to 192.168.0.37 ([192.168.0.37]:22).
debug1: setting up multiplex master socket
debug1: channel 0: new [/home/chris/.ssh/sockets/chris@192.168.0.37-22]
debug1: control_persist_detach: backgrounding master process
debug1: forking to background
debug1: Entering interactive session.
debug1: pledge: id
debug1: multiplexing control connection
debug1: channel 1: new [mux-control]
debug1: channel 2: new [client-session]
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0

It seems the issue is also encountered here, but no concrete solution is mentioned:

https://forum.manjaro.org/t/ssh-connection-hangs-after-logging-in/4847/28 https://stackoverflow.com/questions/53410559/ssh-stuck-to-client-input-global-request-rtype-hostkeys-00openssh-com-want-rep

Strangely it seems to be a network issue - but everything works fine otherwise... Which seems odd.

Each time the SSH shell / connection hangs I'm seeing TCP Retransmission in the network log.

Here is the tshark output from a other machine I'm using to connect TO the machine in question:

sudo tshark -f "tcp port 22" -i any   
Running as user "root" and group "root". This could be dangerous.
tshark: Lua: Error during loading:
 ...94ln7cy52ca-wireshark-cli-2.6.6/share/wireshark/init.lua:32: dofile has been disabled due to running Wireshark as superuser. See https://wiki.wireshark.org/CaptureSetup/CapturePrivileges for help in running Wireshark as an unprivileged user.
Capturing on 'any'
    1 0.000000000  192.168.0.6 → 192.168.0.37 SSH 128 Client: Encrypted packet (len=60)
    2 4.393384377  192.168.0.6 → 192.168.0.37 TCP 76 33764 → 22 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=4094028584 TSecr=0 WS=128
    3 4.451072834 192.168.0.37 → 192.168.0.6  TCP 76 22 → 33764 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=2514759161 TSecr=4094028584 WS=128
    4 4.451117228  192.168.0.6 → 192.168.0.37 TCP 68 33764 → 22 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=4094028641 TSecr=2514759161
    5 4.451358744  192.168.0.6 → 192.168.0.37 SSH 89 Client: Protocol (SSH-2.0-OpenSSH_7.9)
    6 4.459998058 192.168.0.37 → 192.168.0.6  TCP 68 22 → 33764 [ACK] Seq=1 Ack=22 Win=65152 Len=0 TSval=2514759170 TSecr=4094028641
    7 4.475179826 192.168.0.37 → 192.168.0.6  SSHv2 89 Server: Protocol (SSH-2.0-OpenSSH_7.9)
    8 4.475220883  192.168.0.6 → 192.168.0.37 TCP 68 33764 → 22 [ACK] Seq=22 Ack=22 Win=64256 Len=0 TSval=4094028665 TSecr=2514759186
    9 4.475398990  192.168.0.6 → 192.168.0.37 SSHv2 1468 Client: Key Exchange Init
   10 4.486180419 192.168.0.37 → 192.168.0.6  SSHv2 780 Server: Key Exchange Init
   11 4.486193334  192.168.0.6 → 192.168.0.37 TCP 68 33764 → 22 [ACK] Seq=1422 Ack=734 Win=64128 Len=0 TSval=4094028676 TSecr=2514759194
   12 4.488140621 192.168.0.37 → 192.168.0.6  TCP 68 22 → 33764 [ACK] Seq=734 Ack=1422 Win=64128 Len=0 TSval=2514759196 TSecr=4094028666
   13 4.488149618  192.168.0.6 → 192.168.0.37 SSHv2 116 Client: Elliptic Curve Diffie-Hellman Key Exchange Init
   14 4.495305110 192.168.0.37 → 192.168.0.6  TCP 68 22 → 33764 [ACK] Seq=734 Ack=1470 Win=64128 Len=0 TSval=2514759206 TSecr=4094028678
   15 4.591438906 192.168.0.37 → 192.168.0.6  SSHv2 448 Server: Elliptic Curve Diffie-Hellman Key Exchange Reply, New Keys, Encrypted packet (len=172)
   16 4.591486757  192.168.0.6 → 192.168.0.37 TCP 68 33764 → 22 [ACK] Seq=1470 Ack=1114 Win=64128 Len=0 TSval=4094028782 TSecr=2514759301
   17 4.598789938  192.168.0.6 → 192.168.0.37 SSHv2 84 Client: New Keys
   18 4.605412113 192.168.0.37 → 192.168.0.6  TCP 68 22 → 33764 [ACK] Seq=1114 Ack=1486 Win=64128 Len=0 TSval=2514759316 TSecr=4094028789
   19 4.605456186  192.168.0.6 → 192.168.0.37 SSHv2 112 Client: Encrypted packet (len=44)
   20 4.611308177 192.168.0.37 → 192.168.0.6  TCP 68 22 → 33764 [ACK] Seq=1114 Ack=1530 Win=64128 Len=0 TSval=2514759322 TSecr=4094028796
   21 4.611946582 192.168.0.37 → 192.168.0.6  SSHv2 112 Server: Encrypted packet (len=44)
   22 4.611986097  192.168.0.6 → 192.168.0.37 SSHv2 136 Client: Encrypted packet (len=68)
   23 4.617575973 192.168.0.37 → 192.168.0.6  TCP 68 22 → 33764 [ACK] Seq=1158 Ack=1598 Win=64128 Len=0 TSval=2514759328 TSecr=4094028802
   24 4.625638737 192.168.0.37 → 192.168.0.6  SSHv2 144 Server: Encrypted packet (len=76)
   25 4.625769093  192.168.0.6 → 192.168.0.37 SSHv2 696 Client: Encrypted packet (len=628)
   26 4.646837769 192.168.0.37 → 192.168.0.6  SSHv2 144 Server: Encrypted packet (len=76)
   27 4.646955832  192.168.0.6 → 192.168.0.37 SSHv2 160 Client: Encrypted packet (len=92)
   28 4.663613579 192.168.0.37 → 192.168.0.6  SSHv2 128 Server: Encrypted packet (len=60)
   29 4.704685867  192.168.0.6 → 192.168.0.37 TCP 68 33764 → 22 [ACK] Seq=2318 Ack=1370 Win=64128 Len=0 TSval=4094028895 TSecr=2514759374
   30 6.314404352  192.168.0.6 → 192.168.0.37 SSHv2 152 Client: Encrypted packet (len=84)
   31 6.408675360 192.168.0.37 → 192.168.0.6  SSHv2 112 Server: Encrypted packet (len=44)
   32 6.408721657  192.168.0.6 → 192.168.0.37 TCP 68 33764 → 22 [ACK] Seq=2402 Ack=1414 Win=64128 Len=0 TSval=4094030599 TSecr=2514761119
   33 6.408792972  192.168.0.6 → 192.168.0.37 SSHv2 152 Client: Encrypted packet (len=84)
   34 6.417718667 192.168.0.37 → 192.168.0.6  SSHv2 96 Server: Encrypted packet (len=28)
   35 6.417766608  192.168.0.6 → 192.168.0.37 TCP 68 33764 → 22 [ACK] Seq=2486 Ack=1442 Win=64128 Len=0 TSval=4094030608 TSecr=2514761128
   36 6.417913946  192.168.0.6 → 192.168.0.37 SSHv2 180 Client: Encrypted packet (len=112)
   37 6.433295532 192.168.0.37 → 192.168.0.6  SSHv2 720 Server: Encrypted packet (len=652)
   38 6.433347530  192.168.0.6 → 192.168.0.37 TCP 68 33764 → 22 [ACK] Seq=2598 Ack=2094 Win=64128 Len=0 TSval=4094030624 TSecr=2514761143
   39 6.440019259 192.168.0.37 → 192.168.0.6  SSHv2 112 Server: Encrypted packet (len=44)
   40 6.440039427  192.168.0.6 → 192.168.0.37 TCP 68 33764 → 22 [ACK] Seq=2598 Ack=2138 Win=64128 Len=0 TSval=4094030630 TSecr=2514761150
   41 6.440138534  192.168.0.6 → 192.168.0.37 SSHv2 460 Client: Encrypted packet (len=392)
   42 6.674368565  192.168.0.6 → 192.168.0.37 TCP 460 [TCP Retransmission] 33764 → 22 [PSH, ACK] Seq=2598 Ack=2138 Win=64128 Len=392 TSval=4094030865 TSecr=2514761150
   43 6.907359347  192.168.0.6 → 192.168.0.37 TCP 460 [TCP Retransmission] 33764 → 22 [PSH, ACK] Seq=2598 Ack=2138 Win=64128 Len=392 TSval=4094031098 TSecr=2514761150
   44 7.370357846  192.168.0.6 → 192.168.0.37 TCP 460 [TCP Retransmission] 33764 → 22 [PSH, ACK] Seq=2598 Ack=2138 Win=64128 Len=392 TSval=4094031561 TSecr=2514761150
   45 8.330683518  192.168.0.6 → 192.168.0.37 TCP 460 [TCP Retransmission] 33764 → 22 [PSH, ACK] Seq=2598 Ack=2138 Win=64128 Len=392 TSval=4094032521 TSecr=2514761150
   46 10.186372155  192.168.0.6 → 192.168.0.37 TCP 460 [TCP Retransmission] 33764 → 22 [PSH, ACK] Seq=2598 Ack=2138 Win=64128 Len=392 TSval=4094034377 TSecr=2514761150
   47 13.898356243  192.168.0.6 → 192.168.0.37 TCP 460 [TCP Retransmission] 33764 → 22 [PSH, ACK] Seq=2598 Ack=2138 Win=64128 Len=392 TSval=4094038089 TSecr=2514761150
   48 19.274398519  192.168.0.6 → 192.168.0.37 TCP 520 [TCP Retransmission] 33742 → 22 [FIN, PSH, ACK] Seq=4294966905 Ack=1 Win=501 Len=452 TSval=4094043465 TSecr=2514723303
   49 21.322527112  192.168.0.6 → 192.168.0.37 TCP 460 [TCP Retransmission] 33764 → 22 [PSH, ACK] Seq=2598 Ack=2138 Win=64128 Len=392 TSval=4094045513 TSecr=2514761150
Improve this question
  • That looks like a successful SSH connection, just that the server you connected to does not seem to open a shell. Try running with extra verbosity, ssh -vvv, and see if the extra debug points to anything interesting. – GracefulRestart Jul 3 '19 at 3:09
  • Can you also change LogLevel in sshd_config on the remote machine to see what happens on the server side? This might help to find the problem. – dirkt Jul 3 '19 at 9:10
  • similar issue here stackoverflow.com/questions/53410559/… as well as maybe shell not properly set on the other end? maybe clock issue? – Bart Jul 3 '19 at 14:46
1

I had the same issue after modifying some networking configuration on my ssh server. In my case, the TCP retransmissions indicated that the client wasn't getting any response from the server, and was trying to resend the same packet. At the same time, the server had received that packet, and a network trace on the server seemed to indicate responses were being sent.

In my case, the underlying problem was an invalid CIDR prefix for the network interface on the server -- a /32 instead of a /24. Given this, there was no way for the packets from the server to route back to the client.

This issue was reproducible with netcat: on the server with the usual sshd daemon turned off:

nc -l 22

On the client:

nc sshserver 22

and then after typing a few linesof text into the client netcat, noting that the first line sent from the client "makes it" to the server, but no subsequent lines. A network trace shows the exact same behavior: the client continues to see "TCP Retransmission" of the first packet containing the first line of data, and the subsequent data is "stuck".

Fixing the network configuration fixed ssh as well.

Improve this answer
0

Having had ssh hang on a particular machine with a similar error many years ago, it seemed like a good idea to mention here that such things can also be caused by having something in the host server's user session login scripts that emits text, such as an echo command or program called by .bashrc that outputs to stderr under certain circumstances.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.