4

I've been through as many of the bugfixes as I can find for the classic ssh key issue : I've set up my public and private keys "correctly" but passwordless login still doesn't work. The details :

First, I know that nobody wants to read about yet another permission issue :

~$ ls -lhd $HOME
drwx------. 28 mrkelly mrkelly 4.0K May 13 16:23 /mnt/driveB/mrkelly
~$ ls -lhd $HOME/.ssh
drwx------. 2 mrkelly mrkelly 4.0K May 13 15:37 /mnt/driveB/mrkelly/.ssh
~$ ls -lh $HOME/.ssh
total 24K
-rwx------. 1 mrkelly mrkelly 1.7K May 13 15:37 authorized_keys
-rwx------. 1 mrkelly mrkelly  668 May 13 15:20 id_dsa
-rwx------. 1 mrkelly mrkelly  625 May 13 15:20 id_dsa.pub
-rwx------. 1 mrkelly mrkelly 1.7K May 13 15:11 id_rsa
-rwx------. 1 mrkelly mrkelly  417 May 13 15:11 id_rsa.pub
-rwx------. 1 mrkelly mrkelly  980 May 13 14:57 known_hosts
/$ ll -d /
dr-xr-xr-x. 17 root root 4.0K May 14 12:21 /
/$ ll -d /mnt
drwxr-xr-x. 4 root root 4.0K Nov 18 04:33 /mnt
/$ ll -d /mnt/driveB
drwxr-xr-x. 4 root root 4.0K May 13 17:31 /mnt/driveB
~$ uname -a 
Linux action-jackson.stanford.edu 3.19.7-200.fc21.x86_64 #1 SMP Thu May 7 22:00:21 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Having read that SELinux is a common villain in cases like these, I've disabled it :

~$ sudo systemctl status selinux 
● selinux.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

I'm admin on the host machine, so I use one terminal for the sshd server-side service and one as a client.

First, the working version:

  • Host terminal (sshd has been disabled) :

    ~$ sudo /usr/sbin/sshd -d   
    debug1: sshd version OpenSSH_6.6.1, OpenSSL 1.0.1k-fips 8 Jan 2015
    debug1: key_parse_private2: missing begin marker
    debug1: read PEM private key done: type RSA
    debug1: private host key: #0 type 1 RSA
    debug1: key_parse_private2: missing begin marker
    debug1: read PEM private key done: type ECDSA
    debug1: private host key: #1 type 3 ECDSA
    debug1: private host key: #2 type 4 ED25519
    debug1: rexec_argv[0]='/usr/sbin/sshd'
    debug1: rexec_argv[1]='-d'
    Set /proc/self/oom_score_adj from 0 to -1000
    debug1: Bind to port 22 on 0.0.0.0.
    Server listening on 0.0.0.0 port 22.
    debug1: Bind to port 22 on ::.
    Server listening on :: port 22.
    

    (We're here until logging in with the "client" terminal, and then:)

    debug1: Server will not fork when running in debugging mode.
    debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
    debug1: inetd sockets after dupping: 3, 3
    Connection from 127.0.0.1 port 49966 on 127.0.0.1 port 22
    debug1: Client protocol version 2.0; client software version OpenSSH_6.6.1
    debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
    debug1: Enabling compatibility mode for protocol 2.0
    debug1: Local version string SSH-2.0-OpenSSH_6.6.1
    debug1: SELinux support enabled [preauth]
    debug1: ssh_selinux_change_context: setting context from 'unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023' to 'unconfined_u:unconfined_r:sshd_net_t:s0-s0:c0.c1023' [preauth]
    debug1: permanently_set_uid: 74/74 [preauth]
    debug1: list_hostkey_types: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
    debug1: SSH2_MSG_KEXINIT sent [preauth]
    debug1: SSH2_MSG_KEXINIT received [preauth]
    debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none [preauth]
    debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none [preauth]
    debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16 [preauth]
    debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16 [preauth]
    debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
    debug1: SSH2_MSG_NEWKEYS sent [preauth]
    debug1: expecting SSH2_MSG_NEWKEYS [preauth]
    debug1: SSH2_MSG_NEWKEYS received [preauth]
    debug1: KEX done [preauth]
    debug1: userauth-request for user mrkelly service ssh-connection method none [preauth]
    debug1: attempt 0 failures 0 [preauth]
    debug1: PAM: initializing for "mrkelly"
    debug1: PAM: setting PAM_RHOST to "localhost.localdomain"
    debug1: PAM: setting PAM_TTY to "ssh"
    debug1: userauth-request for user mrkelly service ssh-connection method publickey [preauth]
    debug1: attempt 1 failures 0 [preauth]
    debug1: test whether pkalg/pkblob are acceptable [preauth]
    debug1: temporarily_use_uid: 1000/1000 (e=0/0)
    debug1: trying public key file /mnt/driveB/mrkelly/.ssh/authorized_keys
    debug1: fd 4 clearing O_NONBLOCK
    Found matching RSA key: d7:9e:aa:54:63:d7:2d:87:d3:b1:0e:83:3b:70:27:d4
    debug1: restore_uid: 0/0
    Postponed publickey for mrkelly from 127.0.0.1 port 49966 ssh2 [preauth]
    debug1: userauth-request for user mrkelly service ssh-connection method publickey [preauth]
    debug1: attempt 2 failures 0 [preauth]
    debug1: temporarily_use_uid: 1000/1000 (e=0/0)
    debug1: trying public key file /mnt/driveB/mrkelly/.ssh/authorized_keys
    debug1: fd 4 clearing O_NONBLOCK
    Found matching RSA key: d7:9e:aa:54:63:d7:2d:87:d3:b1:0e:83:3b:70:27:d4
    debug1: restore_uid: 0/0
    debug1: ssh_rsa_verify: signature correct
    debug1: do_pam_account: called
    Accepted publickey for mrkelly from 127.0.0.1 port 49966 ssh2: RSA d7:9e:aa:54:63:d7:2d:87:d3:b1:0e:83:3b:70:27:d4
    debug1: monitor_child_preauth: mrkelly has been authenticated by privileged process
    debug1: monitor_read_log: child log fd closed
    debug1: SELinux support enabled
    debug1: PAM: establishing credentials
    User child is on pid 8196
    debug1: PAM: establishing credentials
    debug1: permanently_set_uid: 1000/1000
    debug1: Entering interactive session for SSH2.
    debug1: server_init_dispatch_20
    debug1: server_input_channel_open: ctype session rchan 0 win 1048576 max 16384
    debug1: input_session_request
    debug1: channel 0: new [server-session]
    debug1: session_new: session 0
    debug1: session_open: channel 0
    debug1: session_open: session 0: link with channel 0
    debug1: server_input_channel_open: confirm session
    debug1: server_input_global_request: rtype no-more-sessions@openssh.com want_reply 0
    debug1: server_input_channel_req: channel 0 request x11-req reply 1
    debug1: session_by_channel: session 0 channel 0
    debug1: session_input_channel_req: session 0 req x11-req
    debug1: channel 1: new [X11 inet listener]
    debug1: channel 2: new [X11 inet listener]
    debug1: server_input_channel_req: channel 0 request pty-req reply 1
    debug1: session_by_channel: session 0 channel 0
    debug1: session_input_channel_req: session 0 req pty-req
    debug1: Allocating pty.
    debug1: session_new: session 0
    debug1: SELinux support enabled
    debug1: session_pty_req: session 0 alloc /dev/pts/4
    debug1: server_input_channel_req: channel 0 request env reply 0
    debug1: session_by_channel: session 0 channel 0
    debug1: session_input_channel_req: session 0 req env
    debug1: server_input_channel_req: channel 0 request env reply 0
    debug1: session_by_channel: session 0 channel 0
    debug1: session_input_channel_req: session 0 req env
    debug1: server_input_channel_req: channel 0 request shell reply 1
    debug1: session_by_channel: session 0 channel 0
    debug1: session_input_channel_req: session 0 req shell
    Starting session: shell on pts/4 for mrkelly from 127.0.0.1 port 49966
    debug1: Setting controlling tty using TIOCSCTTY.
    
  • Client terminal :

    ~$ ssh -v `whoami`@localhost
    OpenSSH_6.6.1, OpenSSL 1.0.1k-fips 8 Jan 2015
    debug1: Reading configuration data /etc/ssh/ssh_config
    debug1: /etc/ssh/ssh_config line 56: Applying options for *
    debug1: Connecting to localhost [127.0.0.1] port 22.
    debug1: Connection established.
    debug1: identity file /mnt/driveB/mrkelly/.ssh/id_rsa type 1
    debug1: identity file /mnt/driveB/mrkelly/.ssh/id_rsa-cert type -1
    debug1: identity file /mnt/driveB/mrkelly/.ssh/id_dsa type 2
    debug1: identity file /mnt/driveB/mrkelly/.ssh/id_dsa-cert type -1
    debug1: identity file /mnt/driveB/mrkelly/.ssh/id_ecdsa type -1
    debug1: identity file /mnt/driveB/mrkelly/.ssh/id_ecdsa-cert type -1
    debug1: identity file /mnt/driveB/mrkelly/.ssh/id_ed25519 type -1
    debug1: identity file /mnt/driveB/mrkelly/.ssh/id_ed25519-cert type -1
    debug1: Enabling compatibility mode for protocol 2.0
    debug1: Local version string SSH-2.0-OpenSSH_6.6.1
    debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
    debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
    debug1: SSH2_MSG_KEXINIT sent
    debug1: SSH2_MSG_KEXINIT received
    debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
    debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
    debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
    debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
    debug1: sending SSH2_MSG_KEX_ECDH_INIT
    debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
    debug1: Server host key: ECDSA 8d:cb:f2:94:da:97:7b:0d:ee:e6:bb:8e:3f:41:ae:d8
    debug1: Host 'localhost' is known and matches the ECDSA host key.
    debug1: Found key in /mnt/driveB/mrkelly/.ssh/known_hosts:3
    debug1: ssh_ecdsa_verify: signature correct
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug1: SSH2_MSG_NEWKEYS received
    debug1: Roaming not allowed by server
    debug1: SSH2_MSG_SERVICE_REQUEST sent
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug1: Authentications that can continue: publickey,password,keyboard-interactive
    debug1: Next authentication method: publickey
    debug1: Offering RSA public key: /mnt/driveB/mrkelly/.ssh/id_rsa
    debug1: Server accepts key: pkalg ssh-rsa blen 279
    debug1: key_parse_private2: missing begin marker
    debug1: read PEM private key done: type RSA
    debug1: Authentication succeeded (publickey).
    Authenticated to localhost ([127.0.0.1]:22).
    debug1: channel 0: new [client-session]
    debug1: Requesting no-more-sessions@openssh.com
    debug1: Entering interactive session.
    debug1: Requesting X11 forwarding with authentication spoofing.
    debug1: Sending environment.
    debug1: Sending env LANGUAGE = 
    debug1: Sending env LANG = en_US.UTF-8
    Last login: Wed May 13 15:58:17 2015 from localhost.localdomain
    Environment:
      LANGUAGE=
      LANG=en_US.UTF-8
      USER=mrkelly
      LOGNAME=mrkelly
      HOME=/mnt/driveB/mrkelly
      PATH=/usr/local/bin:/usr/bin
      MAIL=/var/mail/mrkelly
      SHELL=/bin/zsh
      SSH_CLIENT=127.0.0.1 49967 22
      SSH_CONNECTION=127.0.0.1 49967 127.0.0.1 22
      SSH_TTY=/dev/pts/4
      TERM=xterm-256color
      DISPLAY=localhost:11.0
      SELINUX_ROLE_REQUESTED=
      SELINUX_LEVEL_REQUESTED=
      SELINUX_USE_CURRENT_RANGE=
      XDG_SESSION_ID=21
      XDG_RUNTIME_DIR=/run/user/1000
      XDG_SEAT=seat0
      XDG_VTNR=1
    Running /usr/bin/xauth remove unix:11.0
    /usr/bin/xauth add unix:11.0 MIT-MAGIC-COOKIE-1 62951eb22f06d56df8189ee23126a19e
    

(Working, passwordless login)

Now, we restart sshd.service and show the non-working version.

  • Host terminal

    sudo systemctl start sshd
    
  • Client terminal

    ~$ ssh -v `whoami`@localhost
    OpenSSH_6.6.1, OpenSSL 1.0.1k-fips 8 Jan 2015
    debug1: Reading configuration data /etc/ssh/ssh_config
    debug1: /etc/ssh/ssh_config line 56: Applying options for *
    debug1: Connecting to localhost [127.0.0.1] port 22.
    debug1: Connection established.
    debug1: identity file /mnt/driveB/mrkelly/.ssh/id_rsa type 1
    debug1: identity file /mnt/driveB/mrkelly/.ssh/id_rsa-cert type -1
    debug1: identity file /mnt/driveB/mrkelly/.ssh/id_dsa type 2
    debug1: identity file /mnt/driveB/mrkelly/.ssh/id_dsa-cert type -1
    debug1: identity file /mnt/driveB/mrkelly/.ssh/id_ecdsa type -1
    debug1: identity file /mnt/driveB/mrkelly/.ssh/id_ecdsa-cert type -1
    debug1: identity file /mnt/driveB/mrkelly/.ssh/id_ed25519 type -1
    debug1: identity file /mnt/driveB/mrkelly/.ssh/id_ed25519-cert type -1
    debug1: Enabling compatibility mode for protocol 2.0
    debug1: Local version string SSH-2.0-OpenSSH_6.6.1
    debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
    debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
    debug1: SSH2_MSG_KEXINIT sent
    debug1: SSH2_MSG_KEXINIT received
    debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
    debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
    debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
    debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
    debug1: sending SSH2_MSG_KEX_ECDH_INIT
    debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
    debug1: Server host key: ECDSA 8d:cb:f2:94:da:97:7b:0d:ee:e6:bb:8e:3f:41:ae:d8
    debug1: Host 'localhost' is known and matches the ECDSA host key.
    debug1: Found key in /mnt/driveB/mrkelly/.ssh/known_hosts:3
    debug1: ssh_ecdsa_verify: signature correct
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug1: SSH2_MSG_NEWKEYS received
    debug1: Roaming not allowed by server
    debug1: SSH2_MSG_SERVICE_REQUEST sent
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug1: Authentications that can continue: publickey,password,keyboard-interactive
    debug1: Next authentication method: publickey
    debug1: Offering RSA public key: /mnt/driveB/mrkelly/.ssh/id_rsa
    debug1: Authentications that can continue: publickey,password,keyboard-interactive
    debug1: Offering DSA public key: /mnt/driveB/mrkelly/.ssh/id_dsa
    debug1: Authentications that can continue: publickey,password,keyboard-interactive
    debug1: Trying private key: /mnt/driveB/mrkelly/.ssh/id_ecdsa
    debug1: Trying private key: /mnt/driveB/mrkelly/.ssh/id_ed25519
    debug1: Next authentication method: keyboard-interactive
    Password: 
    

(and there it is).

A couple final remarks : I recently had to reformat the installation drive and I'm using my folder that was stored on another drive. When reinstalling Fedora 21, I was forced to create a home directory on that drive, but then I switched my user account to point to a backed-up version of my old home directory after I created the mount points for the other drives.

EDIT : Passwordless login also works using the following invocations :

/usr/sbin/sshd -D

(no daemon, but no debug messages)

/usr/shbin/sshd

( which I guess constitutes a temporary workaround,since this does invoke the daemon)

EDIT2: output from sudo journalctl -u sshd after sudo systemctl restart sshd (logging level debug3)

I'm at the character limit for submissions, so I'll link to a plain text file in dropbox with the output.

Thanks for any help you can provide!

  • So the only thing you're doing different between these test cases is that in the working one you're using sshd -d and in the other one you're not? And enabling debug logging makes it work? – Wumpus Q. Wumbley May 13 '15 at 23:37
  • Not quite-- in one case I've started sshd directly, as you say, but in the other, I've used Fedora's systemctl command, viz. systemctl start sshd.service. What I haven't done is try involking it explicitly WITHOUT the debug option, I'll report back in a second. – Mark Kelly May 13 '15 at 23:40
  • I'd try to get the -d into the other startup method, convince it to log the output somewhere. Or let it get started then strace -f -p it and connect. – Wumpus Q. Wumbley May 13 '15 at 23:56
  • What are you seeing in the logs in the failed case? (Possibly with LogLevel increased in sshd_config....) – mattdm May 14 '15 at 15:52
  • Updated with info requested by mattdm – Mark Kelly May 14 '15 at 19:02
1

This is an SELinux problem, caused by your home directory being in a weird location. The thing you did to disable it, didn't. SELinux isn't a service, and systemctl status selinux is just telling you that ("not-found (Reason: No such file or directory)").

You could run setenforce permissive or otherwise disable SELinux, but this kind of like taking the doors off of your house just because you misplaced your housekeys once. Running chcon -t ssh_home_t ~/.ssh/ should do it.

How do I know? Because that the type for the files in ~/.ssh on my system. But I could have installed sepolicy from the policycoreutils-devel package, and run sepolicy manpage -t ssh_d to generate the ssh_selinux.8 man page, which documents the actual policy in place.

Really, I'd most suggest forgetting all of that, just mounting your drive at /home rather than /mnt/driveB, and then running restorecon -R -v /home.

  • I'll accept this answer, but I don't think I'll quite be able to apply it. This device has several different shares of different formats going, and stopping SELinux has resolved several issues with each of them. That said, I'm sure there's a way to configure SELinux to handle them, but it will take quite a lot of doing. – Mark Kelly May 18 '15 at 1:30
0

It's SELinux. Even though I stopped and disabled it using systemctl, I still needed to use /usr/bin/setenforce 0. Passwordless login now works as expected.

  • Sorry I missed that earlier -- I skimmed over how you attempted to disable SELinux. See my answer for some alternate suggestions. – mattdm May 16 '15 at 2:51
0

I had (what I believe to be) a similar problem, wherein SSH attempts to users whose home directories were based on an NFS share were being repeatedly rejected, with only the following cryptic message written to the (debug/verbose) output of ssh:

debug1: key_parse_private2: missing begin marker

As indicated by the other answers, it turned out to be an SELinux problem. However, it appears that there is a solution more fine-grained than disabling SELinux enforcement entirely.

On the host which is serving the home directories, run the following command to tell SELinux to label the directory containing the user home directories the same way as the default home-directory-containing directory /home:

sudo semanage fcontext -a -e /home /path/to/custom/homedir/container/on/server

Then run this command to recursively reset SELinux labels on the home directories:

sudo restorecon -R /path/to/custom/homedir/container/on/server

If you are using NFS (as in my case), then take execute the following additional command on each client where the home directories are mounted, in order to set SELinux to allow the use of NFS-mounted home directories:

sudo setsebool -P use_nfs_home_dirs on

References:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.