ssh not allowing passwordless login

Question

I use passwordless logins all the time and have set it up on numerous systems and have debugged a lot of issues with it. However, I'm in a situation that I can't diagnose the root cause. I have an Ubuntu 18.04 server and Linux Mint 18.1 client. I can use the same client to get passwordless login via RSA keys to other servers, so I don't suspect any issue on the client. And the server is not allowing passwordless login to this Linux Mint client or another client (RHEL 7.3), so it's most likely a server configuration issue. Since my username on the server is ubuntu, I checked the permissions of /home/ubuntu, /home/ubuntu/.ssh, and /home/ubuntu/.ssh/authorized_keys and all are as expected. I copied my ~/.ssh/id_rsa.pub key over to the server's ~/.ssh/authorized_keys file as I also do. When I try to ssh, I get the following logs on the server, and I don't see any errors reported there that would prohibit the RSA key login.

Server DEBUG3 Logs

debug3: fd 5 is not O_NONBLOCK
debug1: Forked child 31422.
debug3: oom_adjust_restore
debug1: Set /proc/self/oom_score_adj to 0
debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
debug3: send_rexec_state: entering fd = 8 config len 371
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: inetd sockets after dupping: 3, 3
Connection from 10.20.30.3 port 42760 on 10.20.30.2 port 22
debug1: Client protocol version 2.0; client software version SSH
debug1: no match: SSH
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug2: fd 3 setting O_NONBLOCK
debug3: ssh_sandbox_init: preparing seccomp filter sandbox
debug2: Network child is on pid 31423
debug3: preauth child monitor started
debug3: privsep user:group 109:65534 [preauth]
debug1: permanently_set_uid: 109/65534 [preauth]
debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth]
debug3: ssh_sandbox_child: attaching seccomp filter program [preauth]
debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug3: send packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug3: receive packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug2: local server KEXINIT proposal [preauth]
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1 [preauth]
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] [preauth]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] [preauth]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: compression ctos: none,[email protected] [preauth]
debug2: compression stoc: none,[email protected] [preauth]
debug2: languages ctos:  [preauth]
debug2: languages stoc:  [preauth]
debug2: first_kex_follows 0  [preauth]
debug2: reserved 0  [preauth]
debug2: peer client KEXINIT proposal [preauth]
debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group1-sha1,diffie-hellman-group14-sha1 [preauth]
debug2: host key algorithms: ssh-rsa [preauth]
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,3des-cbc,arcfour [preauth]
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,3des-cbc,arcfour [preauth]
debug2: MACs ctos: hmac-md5,hmac-md5-96,hmac-sha1,hmac-sha1-96,hmac-sha2-256,hmac-sha2-512 [preauth]
debug2: MACs stoc: hmac-md5,hmac-md5-96,hmac-sha1,hmac-sha1-96,hmac-sha2-256,hmac-sha2-512 [preauth]
debug2: compression ctos: none [preauth]
debug2: compression stoc: none [preauth]
debug2: languages ctos:  [preauth]
debug2: languages stoc:  [preauth]
debug2: first_kex_follows 0  [preauth]
debug2: reserved 0  [preauth]
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256 [preauth]
debug1: kex: host key algorithm: ssh-rsa [preauth]
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_DH_GEX_REQUEST [preauth]
debug3: receive packet: type 34 [preauth]
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received [preauth]
debug3: mm_request_send entering: type 0 [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 0
debug3: mm_answer_moduli: got parameters: 2048 7680 8192
debug3: mm_request_send entering: type 1
debug2: monitor_read: 0 used once, disabling now
debug3: mm_choose_dh: waiting for MONITOR_ANS_MODULI [preauth]
debug3: mm_request_receive_expect entering: type 1 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_choose_dh: remaining 0 [preauth]
debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent [preauth]
debug3: send packet: type 31 [preauth]
debug2: bits set: 3906/7680 [preauth]
debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT [preauth]
debug3: receive packet: type 32 [preauth]
debug2: bits set: 3905/7680 [preauth]
debug3: mm_key_sign entering [preauth]
debug3: mm_request_send entering: type 6 [preauth]
debug3: mm_key_sign: waiting for MONITOR_ANS_SIGN [preauth]
debug3: mm_request_receive_expect entering: type 7 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 6
debug3: mm_answer_sign
debug3: mm_answer_sign: hostkey proof signature 0x55689da150b0(271)
debug3: mm_request_send entering: type 7
debug2: monitor_read: 6 used once, disabling now
debug3: send packet: type 33 [preauth]
debug3: send packet: type 21 [preauth]
debug2: set_newkeys: mode 1 [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug3: receive packet: type 21 [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug2: set_newkeys: mode 0 [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: KEX done [preauth]
debug3: receive packet: type 5 [preauth]
debug3: send packet: type 6 [preauth]
debug3: receive packet: type 50 [preauth]
debug1: userauth-request for user ubuntu service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug3: mm_getpwnamallow entering [preauth]
debug3: mm_request_send entering: type 8 [preauth]
debug3: mm_getpwnamallow: waiting for MONITOR_ANS_PWNAM [preauth]
debug3: mm_request_receive_expect entering: type 9 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 8
debug3: mm_answer_pwnamallow
debug2: parse_server_config: config reprocess config len 371
debug3: auth_shadow_acctexpired: today 18180 sp_expire -1 days left -18181
debug3: account expiration disabled
debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1
debug3: mm_request_send entering: type 9
debug2: monitor_read: 8 used once, disabling now
debug2: input_userauth_request: setting up authctxt for ubuntu [preauth]
debug3: mm_inform_authserv entering [preauth]
debug3: mm_request_send entering: type 4 [preauth]
debug2: input_userauth_request: try method none [preauth]
debug3: userauth_finish: failure partial=0 next methods="publickey,password" [preauth]
debug3: send packet: type 51 [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 4
debug3: mm_answer_authserv: service=ssh-connection, style=, role=
debug2: monitor_read: 4 used once, disabling now
Connection closed by authenticating user ubuntu 10.20.30.3 port 42760 [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug3: mm_request_receive entering
debug1: do_cleanup
debug1: Killing privsep child 31423
debug1: audit_event: unhandled event 12

Client Logs

$ ssh -vvv [email protected]
OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /home/clientuser/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "10.20.30.2" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 10.20.30.2 [10.20.30.2] port 22.
debug1: Connection established.
debug1: identity file /home/clientuser/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/clientuser/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/clientuser/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/clientuser/.ssh/id_dsa-cert type -1
debug1: identity file /home/clientuser/.ssh/id_ecdsa type 3
debug1: key_load_public: No such file or directory
debug1: identity file /home/clientuser/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/clientuser/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/clientuser/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: Remote protocol version 2.0, remote software version SSH
debug1: no match: SSH
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 10.20.30.2:22 as 'ubuntu'
debug3: hostkeys_foreach: reading file "/home/clientuser/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/clientuser/.ssh/known_hosts:252
debug3: load_hostkeys: loaded 1 keys from 10.20.30.2
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: [email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa,[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group1-sha1,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,3des-cbc,arcfour
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,3des-cbc,arcfour
debug2: MACs ctos: hmac-md5,hmac-md5-96,hmac-sha1,hmac-sha1-96,hmac-sha2-256,hmac-sha2-512
debug2: MACs stoc: hmac-md5,hmac-md5-96,hmac-sha1,hmac-sha1-96,hmac-sha2-256,hmac-sha2-512
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug3: send packet: type 34
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<8192<8192) sent
debug3: receive packet: type 31
debug1: got SSH2_MSG_KEX_DH_GEX_GROUP
debug2: bits set: 4173/8192
debug3: send packet: type 32
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug3: receive packet: type 33
debug1: got SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: ssh-rsa SHA256:real_host_key_was_here
debug3: hostkeys_foreach: reading file "/home/clientuser/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/clientuser/.ssh/known_hosts:252
debug3: load_hostkeys: loaded 1 keys from 10.20.30.2
debug1: Host '10.20.30.2' is known and matches the RSA host key.
debug1: Found key in /home/clientuser/.ssh/known_hosts:252
debug2: bits set: 4062/8192
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug2: key: /home/clientuser/.ssh/id_rsa (0x55616bb74680), agent
debug2: key: /home/clientuser/.ssh/id_dsa ((nil))
debug2: key: /home/clientuser/.ssh/id_ecdsa (0x55616bb76e80)
debug2: key: /home/clientuser/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: password
debug3: start over, passed a different list password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup password
debug3: remaining preferred: ,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
[email protected]'s password: 

Server sshd_config

#   $OpenBSD: sshd_config,v 1.101 2017/03/14 07:19:07 djm Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
LogLevel DEBUG3

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
AuthorizedKeysFile  /home/%u/.ssh/authorized_keys

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication yes 
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM no

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem   sftp    /usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#   X11Forwarding no
#   AllowTcpForwarding no
#   PermitTTY no
#   ForceCommand cvs server

Permissions On server:

ubuntu@test-ubuntu:~$ ls -l /home/ | grep ubuntu
drwx------ 10 ubuntu ubuntu 4096 Oct 11 16:25 ubuntu
ubuntu@test-ubuntu:~$ ls -al ~ | grep .ssh
drwx------  2 ubuntu ubuntu 4096 Oct 11 16:45 .ssh
ubuntu@test-ubuntu:~$ ls -l ~/.ssh | grep authorized_keys
-rw------- 1 ubuntu ubuntu 1194 Oct 11 16:46 authorized_keys

On client:

clientuser@clienthost ~ $ ls -l /home | grep clientuser
drwx------ 88 clientuser  clientuser 4096 Oct 11 11:30 clientuser
clientuser@clienthost ~ $ ls -la ~ | grep .ssh
drwx------   3 clientuser clientuser      4096 Oct 11 11:46 .ssh
clientuser@clienthost ~ $ ls -l ~/.ssh | grep id_rsa
-rw------- 1 clientuser clientuser  1679 Feb  8  2018 id_rsa
-rw-r--r-- 1 clientuser clientuser   395 Feb  8  2018 id_rsa.pub

Answer 1

The server offers the public key:

debug3: userauth_finish: failure partial=0 next methods="publickey,password" [preauth]
debug3: send packet: type 51 [preauth]

but the client turns it down:

debug3: receive packet: type 51
debug1: Authentications that can continue: password

Check /etc/ssh/ssh_config (the client side global configuration) to see if

PubkeyAuthentication = yes

Also, since it seems to be a client-side issue, you might want to make the permissions of .ssh and all the files inside are right.

Answer 2

When you created the key-set, public and private, did you add a password to it? If so, this will request a password when you connect via ssh.

Please take a look at this link. https://askubuntu.com/questions/1019999/key-based-ssh-login-that-requires-both-key-and-password

Answer 3

Now, this is not an answer to your specific problem, just a recipe I use whenever I am faced with puzzling issues like you have. This is a copy from an answer I wrote on AskUbuntu.

To emphasize the benefits here:

1. faster "turnaround" ... you can pass configuration options (-o OptionName=OptionValue) on-the-fly without having to resort to editing the sshd_config file.
2. you may use an alternative copy of sshd_config (-f ...).
3. by using an alternative port you run less of a risk of locking yourself out if the remote server is only accessible via SSH.

---

Troubleshooting sshd

What I find generally very useful in any such cases is to start sshd without letting it daemonize. The problem in my case was that neither syslog nor auth.log showed anything meaningful.

When I started it from the terminal I got:

# $(which sshd) -Ddp 10222
/etc/ssh/sshd_config line 8: address family must be specified before ListenAddress.

Much better! This error message allowed me to see what's wrong and fix it. Neither of the log files contained this output.

NB: at least on Ubuntu the $(which sshd) is the best method to satisfy sshd requirement of an absolute path. Otherwise you'll get the following error: sshd re-exec requires execution with an absolute path. The -p 10222 makes sshd listen on that alternative port, overriding the configuration file - this is so that it doesn't clash with potentially running sshd instances. Make sure to choose a free port here.

This method has helped me many many times in finding issues, be it authentication issues or other types. To get really verbose output to stdout, use $(which sshd) -Ddddp 10222 (note the added dd to increase verbosity). For more debugging goodness check man sshd.