0

We have two servers that we just upgraded to CentOS 7 and ever since then, none of our keys work to ssh into them for any of our users, it always goes to passwords.
This doesn't happen with our other servers and it also didn't happen with CentOS 6 on these same machines. I've checked the permissions of the .ssh directory and the authorized file folder, which are the most common, then I found when googling this problem, but all the files in .ssh are owned by the correct users.

No other answer in google or StackExchange has helped, since they all boil down to a typo or a permissions issue. The output looks very similar/identical to the one provided in this question (Why is SSH key authentication failing for this user? (CENTOS 7)), but the solution that worked for them did not work for us, since I don't get a permissions error when using ssh-copy-id.

Anyone have any ideas or suggestions?

EDIT: More pertinent info:

  • The updated servers are running OpenSSH_7.4 whereas our other servers that people are trying to ssh from are running OpenSSH_5.3
  • Here's the ssh -vvv output (edited): OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug2: ssh_connect: needpriv 0 debug1: Connecting to {servername} [159.28.23.7] port 22. debug1: Connection established. debug1: identity file /{home directory}/{username}/.ssh/identity type -1 debug1: identity file /{home directory}/{username}/.ssh/identity-cert type -1 debug3: Not a RSA1 key file /{home directory}/{username}/.ssh/id_rsa. debug2: key_type_from_name: unknown key type '-----BEGIN' debug3: key_read: missing keytype debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug2: key_type_from_name: unknown key type '-----END' debug3: key_read: missing keytype debug1: identity file /{home directory}/{username}/.ssh/id_rsa type 1 debug1: identity file /{home directory}/{username}/.ssh/id_rsa-cert type -1 debug1: identity file /{home directory}/{username}/.ssh/id_dsa type -1 debug1: identity file /{home directory}/{username}/.ssh/id_dsa-cert type -1 debug1: identity file /{home directory}/{username}/.ssh/id_ecdsa type -1 debug1: identity file /{home directory}/{username}/.ssh/id_ecdsa-cert type -1 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4 debug1: match: OpenSSH_7.4 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_5.3 debug2: fd 3 setting O_NONBLOCK debug1: SSH2_MSG_KEXINIT sent debug3: Wrote 864 bytes for a total of 885 debug1: SSH2_MSG_KEXINIT received debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96 debug2: kex_parse_kexinit: hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96 debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: kex_parse_kexinit: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: kex_parse_kexinit: none,zlib@openssh.com debug2: kex_parse_kexinit: none,zlib@openssh.com debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: mac_setup: found hmac-sha1 debug1: kex: server->client aes128-ctr hmac-sha1 none debug2: mac_setup: found hmac-sha1 debug1: kex: client->server aes128-ctr hmac-sha1 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<2048<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug3: Wrote 24 bytes for a total of 909 debug2: dh_gen_key: priv key bits set: 152/320 debug2: bits set: 1019/2048 debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug3: Wrote 272 bytes for a total of 1181 debug3: check_host_in_hostfile: host {servername} filename /{home directory}/{username}/.ssh/known_hosts debug3: check_host_in_hostfile: host {servername} filename /{home directory}/{username}/.ssh/known_hosts debug3: check_host_in_hostfile: match line 1 debug3: check_host_in_hostfile: host 159.28.23.7 filename /{home directory}/{username}/.ssh/known_hosts debug3: check_host_in_hostfile: host 159.28.23.7 filename /{home directory}/{username}/.ssh/known_hosts debug3: check_host_in_hostfile: match line 1 debug1: Host '{servername}' is known and matches the RSA host key. debug1: Found key in /{home directory}/{username}/.ssh/known_hosts:1 debug2: bits set: 1035/2048 debug1: ssh_rsa_verify: signature correct debug2: kex_derive_keys debug2: set_newkeys: mode 1 debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: Wrote 16 bytes for a total of 1197 debug2: set_newkeys: mode 0 debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug3: Wrote 52 bytes for a total of 1249 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug2: key: /{home directory}/{username}/.ssh/identity ((nil)) debug2: key: /{home directory}/{username}/.ssh/id_rsa (0x7f769e632270) debug2: key: /{home directory}/{username}/.ssh/id_dsa ((nil)) debug2: key: /{home directory}/{username}/.ssh/id_ecdsa ((nil)) debug3: Wrote 84 bytes for a total of 1333 debug1: Authentications that can continue: publickey,password debug3: start over, passed a different list publickey,password debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Trying private key: /{home directory}/{username}/.ssh/identity debug3: no such identity: /{home directory}/{username}/.ssh/identity debug1: Offering public key: /{home directory}/{username}/.ssh/id_rsa debug3: send_pubkey_test debug2: we sent a publickey packet, wait for reply debug3: Wrote 628 bytes for a total of 1961 debug1: Authentications that can continue: publickey,password debug1: Trying private key: /{home directory}/{username}/.ssh/id_dsa debug3: no such identity: /{home directory}/{username}/.ssh/id_dsa debug1: Trying private key: /{home directory}/{username}/.ssh/id_ecdsa debug3: no such identity: /{home directory}/{username}/.ssh/id_ecdsa debug2: we did not send a packet, disable method debug3: authmethod_lookup password debug3: remaining preferred: ,password debug3: authmethod_is_enabled password debug1: Next authentication method: password

EDIT 2: Also worth noting, it doesn't seem to happen with local users, only ones with their user directories in a location shared between all our servers.

  • Restore the previous /etc/ssh/sshd_config file and check logs. – Ipor Sircer Sep 16 '18 at 23:06
  • When you run ssh-keyscan <hostname> what is the output? is it a key or null? – user88036 Sep 16 '18 at 23:09
  • 1
    SSH algorithms are evolving with time....The question could benefit of having more technical details and an ssh debug output. – Rui F Ribeiro Sep 16 '18 at 23:13
  • @Goro The output is {hostname} SSH-2.0-OpenSSH_7.4 and then the server's rsa key – Laurence Ruberl Sep 17 '18 at 0:07
  • Did you try to delete the folder .ssh, and then create it again, after then generate new rsa keys? You can run the following commands: 1) rm -rf ~/.ssh 2) mkdir ~/.ssh 3) cd ~/.ssh 4) ssh-keygen -t rsa "choose no passphrase when asked and accept the default filename of id_rsa" 5) scp id_rsa.pub <user>@<yourhost>:.ssh/authorized_keys Provide your password when asked . – user88036 Sep 17 '18 at 0:13
0

The solution ended up being mostly unrelated PAM itself. The problem was actually an interaction with SELinux, which we don't normally use and which we think was installed and activated by default. Setting it to Permissive mode fixed the issue immediately and users can use their keys again.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.