Can't SSH into openssh server version 7.8 and greater

Question

I can't log into a clean Centos8 install on my VPS from the terminal using public key authentication.

When logging in it seems to authenticate OK, but then it times-out then returns a "Connection reset by [remote host ip address]" (not peer).

I've checked out all the logs. But nothing is really forthcoming.

Things I've checked out:

• /var/log/messages
• /var/log/audit/audit.log
• /var/log/secure
• Permissions for ~/.ssh and ~/.ssh/id_rsa* files
• /etc/ssh/sshd_config

I've tried logging in from different machines in my LAN without success:

• Ubuntu 18.04
• CentOS 7 machine
• WSL

It's worthwhile to note I have no problems logging into a clean CentOS7 install from the same VPS.

Is there a change in OpenSSH 8 that I'm not aware of?

I've attached verbose output for logging into a CentOS7 and CentOS8 system:

• CentOS 7 SSH OK: https://pastebin.com/uGc1T8tk
• CentOS 8 SSH Fail: https://pastebin.com/XadQQEBK

As you can see authentication succeeds in both:

debug1: Authentication succeeded (publickey)

The main difference I can see in CentOS 8 is that it's now using OpenSSH 8 instead of OpenSSH 7

Is there a change in OpenSSH 8 that I'm not aware of?

Thanks all in advance.

---

Edit 1

@thehostingadmin I don't know how to copy n paste from the web GUI but here's the output from the command you provided: https://i.stack.imgur.com/9lJec.jpg

Edit 2

# sysctl -w net.ipv4.tcp_tw_recycle=0
sysctl: cannot stat /proc/sys/net/ipv4/tcp_tw_recycle: No such file or directory

# sysctl -w net.ipv4.tcp_tw_reuse=0
net.ipv4.tcp_tw_reuse = 0

Result: Still no dice

Edit 3

Success/Fail Matrix

|         OS         |  SSH Client   |   Network    | Auth |  Success   |
|--------------------|---------------|--------------|------|------------|
| Ubuntu 18.04 (WSL) | OpenSSH_7.6p1 | LAN (NAT x2) | OK   | E01 | TO/H |
| Debian 10 (RPi)    | OpenSSH_7.9p1 | LAN (NAT x2) | OK   | TO/H       |
| Ubuntu 18.04       | OpenSSH_7.6p1 | LAN (NAT x1) | OK   | E02        |
| CentOS 7.8         | OpenSSH_7.4p1 | LAN (NAT x1) | OK   | TO/H       |
| CentOS 7.8         | OpenSSH_7.4p1 | VPS          | OK   | OK         |

• E01: Connection reset by [remote host ip address] port 22
• E02: packet_write_wait: Connection to [remote host ip address] port 22: Broken pipe
• TO/H: Timeout/Hang

Edit 4 Made the following changes:

# /etc/ssh/sshd_config
TCPKeepAlive yes
ClientAliveInterval 20
ClientAliveCountMax 5

No measurable difference.

Edit 5

When ssh client throws error E01, I can see the following on the server:

# journalctl -u systemd-logind -fn0
Aug 27 14:51:18 host systemd-logind[701]: New session 136 of user root.
debug1: client_input_channel_req: channel 0 rtype [email protected] reply 1
Aug 27 14:51:38 host systemd-logind[701]: Session 136 logged out. Waiting for processes to exit.
Aug 27 14:51:38 host systemd-logind[701]: Removed session 136.

A successful login looks like this:

Aug 27 15:04:42 host systemd-logind[701]: New session 143 of user root.

Edit 6

I turned LogLevel to DEBUG3 and found the following in /var/log/secure

# Login Fail
Aug 27 15:59:07 host sshd[9387]: debug3: monitor_read: checking request 124
Aug 27 15:59:26 host sshd[9389]: Read error from remote host <wan ip> port 2611: Connection reset by peer
Aug 27 15:59:26 host sshd[9389]: debug1: do_cleanup

# Login Success
Aug 27 15:57:48 host sshd[9360]: debug3: monitor_read: checking request 124

Edit 7

Apologies to @fra-san. Adding more verbose logging (-vvv) did provide more detail

# SSH fail
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: send packet: type 1
Connection reset by <remote host ip> port 22

# SSH Success
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Activate the web console with: systemctl enable --now cockpit.socket

Edit 8

Switched out my home router (custom firmware) with ISP provided router. Still no joy.

Edit 9

Wading into wireshark packet tracing.

• Success vs Fail: https://i.stack.imgur.com/fkL0z.jpg
• Graph: https://i.stack.imgur.com/SWrog.jpg

Note: After raising my issue with my hosting provider (works for them) the OpenSSH server on the server has switched to OpenSSH 7.8 - but the issue persist.

Edit 10

Wireshark. Client vs Server: https://i.stack.imgur.com/XRZRW.jpg

Notes

• Frame 29 from server not received by client
• Hosting provider changed openssh back to v8

Answer 1

There's no way to know without enabling DEBUG logging and then tailing sshd with journalctl on the VPS.

To do that you'll need to ask your VPS provider how to access the server via Console. Different providers have different methods.

Add the following configuration anywhere in /etc/ssh/sshd_config on the VPS:

LogLevel DEBUG

Restart sshd on the VPS:

sudo systemctl restart sshd

Tail the sshd log on the VPS:

journalctl -u sshd -fn0

Now attempt to ssh from your local machine so that you can see what sshd says is wrong.

EDIT: Update based on results of comments:

Considering the fact that you're having difficulties connecting from hosts that NATd, check to see if the following kernel configs are enabled on your Cent 8 VPS, and if so disable them:

sysctl -w net.ipv4.tcp_tw_recycle=0

sysctl -w net.ipv4.tcp_tw_reuse=0

If this works use the /etc/sysctl.conf to persist those changes through a reboot.

Answer 2

I have some good news.

I raised an issue with my ISP. They've issued me with a new IP. I don't know what else they did but it has resolved the issue.

I have also raised an issue with redhat developers working on openssh.

They said this is a known issue especially in the VMWare community:

https://communities.vmware.com/thread/590825

The issue is related to a change in OpenSSH_7.8 relating to the QoS flag.

http://www.openssh.com/txt/release-7.8

The workaround is to use the following in your ~/.ssh/config file

Host *
    IPQoS=throughput

I will not mark this as the answer until I can confirm it working on my end.

Answer 3

This made the difference for me:

restorecon -Rv ~/.ssh 

From here: https://wiki.centos.org/HowTos/Network/SecuringSSH