1

I can't log into a clean Centos8 install on my VPS from the terminal using public key authentication.

When logging in it seems to authenticate OK, but then it times-out then returns a "Connection reset by [remote host ip address]" (not peer).

I've checked out all the logs. But nothing is really forthcoming.

Things I've checked out:

  • /var/log/messages
  • /var/log/audit/audit.log
  • /var/log/secure
  • Permissions for ~/.ssh and ~/.ssh/id_rsa* files
  • /etc/ssh/sshd_config

I've tried logging in from different machines in my LAN without success:

  • Ubuntu 18.04
  • CentOS 7 machine
  • WSL

It's worthwhile to note I have no problems logging into a clean CentOS7 install from the same VPS.

Is there a change in OpenSSH 8 that I'm not aware of?

I've attached verbose output for logging into a CentOS7 and CentOS8 system:

As you can see authentication succeeds in both:

debug1: Authentication succeeded (publickey)

The main difference I can see in CentOS 8 is that it's now using OpenSSH 8 instead of OpenSSH 7

Is there a change in OpenSSH 8 that I'm not aware of?

Thanks all in advance.


Edit 1

@thehostingadmin I don't know how to copy n paste from the web GUI but here's the output from the command you provided: https://imgur.com/9WQJGuN

Edit 2

# sysctl -w net.ipv4.tcp_tw_recycle=0
sysctl: cannot stat /proc/sys/net/ipv4/tcp_tw_recycle: No such file or directory

# sysctl -w net.ipv4.tcp_tw_reuse=0
net.ipv4.tcp_tw_reuse = 0

Result: Still no dice

Edit 3

Success/Fail Matrix

|         OS         |  SSH Client   |   Network    | Auth |  Success   |
|--------------------|---------------|--------------|------|------------|
| Ubuntu 18.04 (WSL) | OpenSSH_7.6p1 | LAN (NAT x2) | OK   | E01 | TO/H |
| Debian 10 (RPi)    | OpenSSH_7.9p1 | LAN (NAT x2) | OK   | TO/H       |
| Ubuntu 18.04       | OpenSSH_7.6p1 | LAN (NAT x1) | OK   | E02        |
| CentOS 7.8         | OpenSSH_7.4p1 | LAN (NAT x1) | OK   | TO/H       |
| CentOS 7.8         | OpenSSH_7.4p1 | VPS          | OK   | OK         |
  • E01: Connection reset by [remote host ip address] port 22
  • E02: packet_write_wait: Connection to [remote host ip address] port 22: Broken pipe
  • TO/H: Timeout/Hang

Edit 4 Made the following changes:

# /etc/ssh/sshd_config
TCPKeepAlive yes
ClientAliveInterval 20
ClientAliveCountMax 5

No measurable difference.

Edit 5

When ssh client throws error E01, I can see the following on the server:

# journalctl -u systemd-logind -fn0
Aug 27 14:51:18 host systemd-logind[701]: New session 136 of user root.
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Aug 27 14:51:38 host systemd-logind[701]: Session 136 logged out. Waiting for processes to exit.
Aug 27 14:51:38 host systemd-logind[701]: Removed session 136.

A successful login looks like this:

Aug 27 15:04:42 host systemd-logind[701]: New session 143 of user root.

Edit 6

I turned LogLevel to DEBUG3 and found the following in /var/log/secure

# Login Fail
Aug 27 15:59:07 host sshd[9387]: debug3: monitor_read: checking request 124
Aug 27 15:59:26 host sshd[9389]: Read error from remote host <wan ip> port 2611: Connection reset by peer
Aug 27 15:59:26 host sshd[9389]: debug1: do_cleanup

# Login Success
Aug 27 15:57:48 host sshd[9360]: debug3: monitor_read: checking request 124

Edit 7

Apologies to @fra-san. Adding more verbose logging (-vvv) did provide more detail

# SSH fail
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: send packet: type 1
Connection reset by <remote host ip> port 22

# SSH Success
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Activate the web console with: systemctl enable --now cockpit.socket

Edit 8

Switched out my home router (custom firmware) with ISP provided router. Still no joy.

Edit 9

Wading into wireshark packet tracing.

Note: After raising my issue with my hosting provider (works for them) the OpenSSH server on the server has switched to OpenSSH 7.8 - but the issue persist.

Edit 10

Wireshark. Client vs Server: https://imgur.com/fIVbEqq

Notes

  • Frame 29 from server not received by client
  • Hosting provider changed openssh back to v8
11
  • 1
    This isn't really likely to help, but attaching more verbose debugging information (obtained using ssh -vvv) to your question may provide some hints. – fra-san Aug 26 '20 at 7:37
  • Also, trying to rule out some kinds of issues: are you able to ssh to your CentOS 8 system by using a different authentication method? – fra-san Aug 26 '20 at 7:49
  • Adding more verbose debug information didn't provide any more detail into the issue. – denormalizer Aug 26 '20 at 8:15
  • I can do password login only by using my VPS provider's web based console. Password login in any other way results in the password being requested again and again – denormalizer Aug 26 '20 at 8:18
  • Well that's quite peculiar. I'm feeling suspicious of your network connection. Are you able to attempt the SSH connection from a different host/network? – thahostingadmin Aug 26 '20 at 8:24
0

There's no way to know without enabling DEBUG logging and then tailing sshd with journalctl on the VPS.

To do that you'll need to ask your VPS provider how to access the server via Console. Different providers have different methods.

Add the following configuration anywhere in /etc/ssh/sshd_config on the VPS:

LogLevel DEBUG

Restart sshd on the VPS:

sudo systemctl restart sshd

Tail the sshd log on the VPS:

journalctl -u sshd -fn0

Now attempt to ssh from your local machine so that you can see what sshd says is wrong.

EDIT: Update based on results of comments:

Considering the fact that you're having difficulties connecting from hosts that NATd, check to see if the following kernel configs are enabled on your Cent 8 VPS, and if so disable them:

sysctl -w net.ipv4.tcp_tw_recycle=0

sysctl -w net.ipv4.tcp_tw_reuse=0

If this works use the /etc/sysctl.conf to persist those changes through a reboot.

3
  • Thanks for your feedback. I've updated my question with the output – denormalizer Aug 26 '20 at 8:20
  • @denormalizer I added a potential solution based on a common issue I've seen. Could you give it a shot and let us know if it works? – thahostingadmin Aug 26 '20 at 12:11
  • Thanks for following up. Please see Edit 2 (and Edit 3) – denormalizer Aug 26 '20 at 13:28
0

I have some good news.

I raised an issue with my ISP. They've issued me with a new IP. I don't know what else they did but it has resolved the issue.

I have also raised an issue with redhat developers working on openssh.

They said this is a known issue especially in the VMWare community:

https://communities.vmware.com/thread/590825

The issue is related to a change in OpenSSH_7.8 relating to the QoS flag.

http://www.openssh.com/txt/release-7.8

The workaround is to use the following in your ~/.ssh/config file

Host *
    IPQoS=throughput

I will not mark this as the answer until I can confirm it working on my end.

1
  • It seems my upstream router (ISP) had problems with the QoS changes made in openssh server 7.8. I was moved to a different router which fixed the issue for me. – denormalizer Nov 4 '20 at 0:00
0

This made the difference for me:

restorecon -Rv ~/.ssh 

From here: https://wiki.centos.org/HowTos/Network/SecuringSSH

1
  • Permissions can sometimes be the source of these issues. But not in my case. – denormalizer Nov 4 '20 at 0:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.