I can't log into a clean Centos8 install on my VPS from the terminal using public key authentication.
When logging in it seems to authenticate OK, but then it times-out then returns a "Connection reset by [remote host ip address]" (not peer).
I've checked out all the logs. But nothing is really forthcoming.
Things I've checked out:
- /var/log/messages
- /var/log/audit/audit.log
- /var/log/secure
- Permissions for ~/.ssh and ~/.ssh/id_rsa* files
- /etc/ssh/sshd_config
I've tried logging in from different machines in my LAN without success:
- Ubuntu 18.04
- CentOS 7 machine
- WSL
It's worthwhile to note I have no problems logging into a clean CentOS7 install from the same VPS.
Is there a change in OpenSSH 8 that I'm not aware of?
I've attached verbose output for logging into a CentOS7 and CentOS8 system:
- CentOS 7 SSH OK: https://pastebin.com/uGc1T8tk
- CentOS 8 SSH Fail: https://pastebin.com/XadQQEBK
As you can see authentication succeeds in both:
debug1: Authentication succeeded (publickey)
The main difference I can see in CentOS 8 is that it's now using OpenSSH 8 instead of OpenSSH 7
Is there a change in OpenSSH 8 that I'm not aware of?
Thanks all in advance.
Edit 1
@thehostingadmin I don't know how to copy n paste from the web GUI but here's the output from the command you provided: https://i.stack.imgur.com/9lJec.jpg
Edit 2
# sysctl -w net.ipv4.tcp_tw_recycle=0
sysctl: cannot stat /proc/sys/net/ipv4/tcp_tw_recycle: No such file or directory
# sysctl -w net.ipv4.tcp_tw_reuse=0
net.ipv4.tcp_tw_reuse = 0
Result: Still no dice
Edit 3
Success/Fail Matrix
| OS | SSH Client | Network | Auth | Success |
|--------------------|---------------|--------------|------|------------|
| Ubuntu 18.04 (WSL) | OpenSSH_7.6p1 | LAN (NAT x2) | OK | E01 | TO/H |
| Debian 10 (RPi) | OpenSSH_7.9p1 | LAN (NAT x2) | OK | TO/H |
| Ubuntu 18.04 | OpenSSH_7.6p1 | LAN (NAT x1) | OK | E02 |
| CentOS 7.8 | OpenSSH_7.4p1 | LAN (NAT x1) | OK | TO/H |
| CentOS 7.8 | OpenSSH_7.4p1 | VPS | OK | OK |
E01
: Connection reset by [remote host ip address] port 22E02
: packet_write_wait: Connection to [remote host ip address] port 22: Broken pipeTO/H
: Timeout/Hang
Edit 4 Made the following changes:
# /etc/ssh/sshd_config
TCPKeepAlive yes
ClientAliveInterval 20
ClientAliveCountMax 5
No measurable difference.
Edit 5
When ssh client throws error E01
, I can see the following on the server:
# journalctl -u systemd-logind -fn0
Aug 27 14:51:18 host systemd-logind[701]: New session 136 of user root.
debug1: client_input_channel_req: channel 0 rtype [email protected] reply 1
Aug 27 14:51:38 host systemd-logind[701]: Session 136 logged out. Waiting for processes to exit.
Aug 27 14:51:38 host systemd-logind[701]: Removed session 136.
A successful login looks like this:
Aug 27 15:04:42 host systemd-logind[701]: New session 143 of user root.
Edit 6
I turned LogLevel
to DEBUG3
and found the following in /var/log/secure
# Login Fail
Aug 27 15:59:07 host sshd[9387]: debug3: monitor_read: checking request 124
Aug 27 15:59:26 host sshd[9389]: Read error from remote host <wan ip> port 2611: Connection reset by peer
Aug 27 15:59:26 host sshd[9389]: debug1: do_cleanup
# Login Success
Aug 27 15:57:48 host sshd[9360]: debug3: monitor_read: checking request 124
Edit 7
Apologies to @fra-san. Adding more verbose logging (-vvv
) did provide more detail
# SSH fail
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: send packet: type 1
Connection reset by <remote host ip> port 22
# SSH Success
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Activate the web console with: systemctl enable --now cockpit.socket
Edit 8
Switched out my home router (custom firmware) with ISP provided router. Still no joy.
Edit 9
Wading into wireshark packet tracing.
- Success vs Fail: https://i.stack.imgur.com/fkL0z.jpg
- Graph: https://i.stack.imgur.com/SWrog.jpg
Note: After raising my issue with my hosting provider (works for them) the OpenSSH server on the server has switched to OpenSSH 7.8 - but the issue persist.
Edit 10
Wireshark. Client vs Server: https://i.stack.imgur.com/XRZRW.jpg
Notes
- Frame 29 from server not received by client
- Hosting provider changed
openssh
back to v8
ssh -vvv
) to your question may provide some hints.ssh
to your CentOS 8 system by using a different authentication method?