There's gazillion possible reasons — as you can see in this thread. Another one:
- extra
=
character in authorized_keys
.
Somehow, my ed25519 .pub
file had contained an extra b64 padding like this:
───────┬───────────────────────────────────────────────────────────────────────────────────────────────
│ File: .ssh/acme/acme-aws.key.pub
───────┼───────────────────────────────────────────────────────────────────────────────────────────────
1 │ ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHVz0kRGRITW0/GqAJ89Xa/UlrZpkbQHkUE7WVG55VBH= [email protected]
───────┴───────────────────────────────────────────────────────────────────────────────────────────────
Naturally, I pasted this into authorized_keys
as-is. This used to work no problem.
Bizarrely though, this time the server kept failing to match it.
sshd[825773]: debug2: userauth_pubkey: valid user ubuntu querying public key ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHVz0kRGRITW0/GqAJ89Xa/UlrZpkbQHkUE7WVG55VBH [preauth]
sshd[825773]: debug1: userauth_pubkey: publickey test pkalg ssh-ed25519 pkblob ED25519 SHA256:6I/LPlQEYs7pDIr8zjy/q2LtzzXRUC1E8V5RB8fQf5k [preauth]
sshd[825773]: debug1: temporarily_use_uid: 1000/1000 (e=0/0)
sshd[825773]: debug1: trying public key file /home/ubuntu/.ssh/authorized_keys
sshd[825773]: debug1: fd 5 clearing O_NONBLOCK
sshd[825773]: debug2: /home/ubuntu/.ssh/authorized_keys:1: check options: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDDNanyt0YmYPGOyHV9D/+rHgdL9QQWof0P1UQ6Yz8dT/4/JXbvSGzaJKbRsuRGpRE4n2k8Bcy2CC7PqKps/UZD+LELWbfT+LW+SQ3db5NZ2FL6U15jdzhf9tMnEMy/ci3tXPi3Uy7rQgrIAiT/5gJpDJBP6kZLtSvdAtRjuvWIhrXsxXox1aXxLzY23XmAtFEgbRyDrtrWZokEfZqbokkU5NogFu675IByxQKujlgCYSslS920gG7Xh1bfX2zjlUGkSrtajdOcEa7O2lEG5U+AZyXFCGDbM76QFkVLJhZr+mYMnfimUuu92tN588K+SHopYkgksfmcWFfECrcHB9ww4Ew+i6nbvIJGIcpCgpBintJHq6gDCWqOXmseij8edEkSOxtHb+JfT//EFkr
sshd[825773]: debug2: /home/ubuntu/.ssh/authorized_keys:1: advance: 'AAAAB3NzaC1yc2EAAAADAQABAAACAQDDNanyt0YmYPGOyHV9D/+rHgdL9QQWof0P1UQ6Yz8dT/4/JXbvSGzaJKbRsuRGpRE4n2k8Bcy2CC7PqKps/UZD+LELWbfT+LW+SQ3db5NZ2FL6U15jdzhf9tMnEMy/ci3tXPi3Uy7rQgrIAiT/5gJpDJBP6kZLtSvdAtRjuvWIhrXsxXox1aXxLzY23XmAtFEgbRyDrtrWZokEfZqbokkU5NogFu675IByxQKujlgCYSslS920gG7Xh1bfX2zjlUGkSrtajdOcEa7O2lEG5U+AZyXFCGDbM76QFkVLJhZr+mYMnfimUuu92tN588K+SHopYkgksfmcWFfECrcHB9ww4Ew+i6nbvIJGIcpCgpBintJHq6gDCWqOXmseij8edEkSOxtHb+JfT//EFkr2LpnVE+Qg7TcSf
sshd[825773]: debug2: /home/ubuntu/.ssh/authorized_keys:2: check options: 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHVz0kRGRITW0/GqAJ89Xa/UlrZpkbQHkUE7WVG55VBH= [email protected]\n'
sshd[825773]: debug2: /home/ubuntu/.ssh/authorized_keys:2: advance: 'AAAAC3NzaC1lZDI1NTE5AAAAIHVz0kRGRITW0/GqAJ89Xa/UlrZpkbQHkUE7WVG55VBH= [email protected]\n'
Read the log, paying attention:
- the server does receive the key I'd configured (which implies the client did send it):
"valid user ubuntu querying public key ...55VBH [preauth]"
- proceeds to scan authorized_keys; skips an ssh-rsa key; skips the ed25519 key too;
- the skipped key ends on
...55VBH=
. Due to the extra trailing =
, this fails to match.
Removing the =
padding in authorized_keys to match perfectly what ssh-keygen -y -f the.key
said — resolved the issue for me.
In general, so much can go wrong (and normal, too) resulting in this log — you'd be exercising luck trying to find advice with an expectation to see a-la "run sudo magic
and it'll get fixed".
What's guaranteed to help — is enabling debug logs and inspecting them carefully. Yes, even 1 character difference matters.
On the client: ssh -v
/ ssh -vv
/ ssh -vvv
will produce progressively more detailed debug logs.
On the server: edit /etc/ssh/sshd_config
remembering to sudo systemctl restart ssh
afterwards:
# …
LogLevel DEBUG
#LogLevel DEBUG2
#LogLevel DEBUG3
The server logs will be in journalctl -u ssh.service -f
.
/usr/local/sbin/sshd -D -e
, possible workaround: serverfault.com/a/211176/var/log/secure
withLogLevel DEBUG3
in/etc/ssh/sshd_config