Suddenly unable to ssh into CentOS 8 server with error "Server refused our key"

Question

I have a CentOS 8 server set up with a few users with SSH keys, and I haven't had any problems logging in with any of the users as of a few days ago. (Password authentication is disabled, so is root login.)

I go to log in today using PuTTY and WinSCP and get a chilling:

Server refused our key

This happens for ALL users. sshd seems to be functioning obviously since it gets up to the point of refusing a key. When I dig through the logs I see:

Server offered these authentication methods: publickey,gssapi-keyex,gssapi-with-mic

This indicates that sshd seems to be operating normally.

I tried logging in using keys loaded in Pageant, and I also tried bypassing Pageant and loading the keys directly in PuTTY and WinSCP, the key is still refused. And yes, it's the correct key, and it hasn't changed.

I happen to have a backup application running so I am able to see what changes to the files may have taken place on the server. I can confirm that the .ssh/authorized_keys files have not been modified. Nor has sshd_config, which I downloaded and looked at.

The last time I logged in was a few days ago and I did a dnf update, but I don't see how that can affect any of this.

Any ideas what may be causing this?

Answer 1

The problem was the /home permissions. I had a sneaking suspicion but refused to entertain the thought because I didn't see how the problem could have happened. As I jog my memory I vaguely remember mistyping a setfacl command when working on ACL entries for a directory. I accidentally did a setfacl -Rn [...] /home, which messed up the permissions that sshd requires for $home/.ssh/authorized_keys.

This meant all users were locked out. With PasswordAuthentication set to No, there was no way to get into the server. So how did I solve it? Thanks to Acronis backup. I simply restored sshd_config from several months ago back when it had PasswordAuthentication set to Yes and added reboot as a postcommand. I then logged in using password after the server rebooted and did setfacl -b in the right places to remove ACLs and everything is back to normal.

Kind of a lucky break, but I specifically configured the backup for several disaster scenarios, one of them being needing files from a long time ago.

P.S.: Friendly advice, double check your backup strategy NOW and invest in a solid solution. Without enterprise-grade backup, I would have been bogged down for days trying to move the server's hard drive to a new server and revive my web applications, or trying to boot into rescue mode or some other tedious/iffy method.