3

I've been through as many of the bugfixes as I can find for the classic ssh key issue : I've set up my public and private keys "correctly" but passwordless login still doesn't work. The details :

First, I know that nobody wants to read about yet another permission issue :

~$ ls -lhd $HOME
drwx------. 28 mrkelly mrkelly 4.0K May 13 16:23 /mnt/driveB/mrkelly
~$ ls -lhd $HOME/.ssh
drwx------. 2 mrkelly mrkelly 4.0K May 13 15:37 /mnt/driveB/mrkelly/.ssh
~$ ls -lh $HOME/.ssh
total 24K
-rwx------. 1 mrkelly mrkelly 1.7K May 13 15:37 authorized_keys
-rwx------. 1 mrkelly mrkelly  668 May 13 15:20 id_dsa
-rwx------. 1 mrkelly mrkelly  625 May 13 15:20 id_dsa.pub
-rwx------. 1 mrkelly mrkelly 1.7K May 13 15:11 id_rsa
-rwx------. 1 mrkelly mrkelly  417 May 13 15:11 id_rsa.pub
-rwx------. 1 mrkelly mrkelly  980 May 13 14:57 known_hosts
/$ ll -d /
dr-xr-xr-x. 17 root root 4.0K May 14 12:21 /
/$ ll -d /mnt
drwxr-xr-x. 4 root root 4.0K Nov 18 04:33 /mnt
/$ ll -d /mnt/driveB
drwxr-xr-x. 4 root root 4.0K May 13 17:31 /mnt/driveB
~$ uname -a 
Linux action-jackson.stanford.edu 3.19.7-200.fc21.x86_64 #1 SMP Thu May 7 22:00:21 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Having read that SELinux is a common villain in cases like these, I've disabled it : 

~$ sudo systemctl status selinux 
● selinux.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

I'm admin on the host machine, so I use one terminal for the sshd server-side service and one as a client.

First, the working version:

  • Host terminal (sshd has been disabled) : 

    ~$ sudo /usr/sbin/sshd -d   
debug1: sshd version OpenSSH_6.6.1, OpenSSL 1.0.1k-fips 8 Jan 2015
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type ECDSA
debug1: private host key: #1 type 3 ECDSA
debug1: private host key: #2 type 4 ED25519
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Bind to port 22 on ::.
Server listening on :: port 22.

    (We're here until logging in with the "client" terminal, and then:) 

    debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 127.0.0.1 port 49966 on 127.0.0.1 port 22
debug1: Client protocol version 2.0; client software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: SELinux support enabled [preauth]
debug1: ssh_selinux_change_context: setting context from 'unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023' to 'unconfined_u:unconfined_r:sshd_net_t:s0-s0:c0.c1023' [preauth]
debug1: permanently_set_uid: 74/74 [preauth]
debug1: list_hostkey_types: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none [preauth]
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none [preauth]
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16 [preauth]
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16 [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user mrkelly service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "mrkelly"
debug1: PAM: setting PAM_RHOST to "localhost.localdomain"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user mrkelly service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug1: test whether pkalg/pkblob are acceptable [preauth]
debug1: temporarily_use_uid: 1000/1000 (e=0/0)
debug1: trying public key file /mnt/driveB/mrkelly/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
Found matching RSA key: d7:9e:aa:54:63:d7:2d:87:d3:b1:0e:83:3b:70:27:d4
debug1: restore_uid: 0/0
Postponed publickey for mrkelly from 127.0.0.1 port 49966 ssh2 [preauth]
debug1: userauth-request for user mrkelly service ssh-connection method publickey [preauth]
debug1: attempt 2 failures 0 [preauth]
debug1: temporarily_use_uid: 1000/1000 (e=0/0)
debug1: trying public key file /mnt/driveB/mrkelly/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
Found matching RSA key: d7:9e:aa:54:63:d7:2d:87:d3:b1:0e:83:3b:70:27:d4
debug1: restore_uid: 0/0
debug1: ssh_rsa_verify: signature correct
debug1: do_pam_account: called
Accepted publickey for mrkelly from 127.0.0.1 port 49966 ssh2: RSA d7:9e:aa:54:63:d7:2d:87:d3:b1:0e:83:3b:70:27:d4
debug1: monitor_child_preauth: mrkelly has been authenticated by privileged process
debug1: monitor_read_log: child log fd closed
debug1: SELinux support enabled
debug1: PAM: establishing credentials
User child is on pid 8196
debug1: PAM: establishing credentials
debug1: permanently_set_uid: 1000/1000
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 0 win 1048576 max 16384
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_global_request: rtype no-more-sessions@openssh.com want_reply 0
debug1: server_input_channel_req: channel 0 request x11-req reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req x11-req
debug1: channel 1: new [X11 inet listener]
debug1: channel 2: new [X11 inet listener]
debug1: server_input_channel_req: channel 0 request pty-req reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req pty-req
debug1: Allocating pty.
debug1: session_new: session 0
debug1: SELinux support enabled
debug1: session_pty_req: session 0 alloc /dev/pts/4
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request shell reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req shell
Starting session: shell on pts/4 for mrkelly from 127.0.0.1 port 49966
debug1: Setting controlling tty using TIOCSCTTY.

  • Client terminal :

    ~$ ssh -v `whoami`@localhost
OpenSSH_6.6.1, OpenSSL 1.0.1k-fips 8 Jan 2015
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug1: Connecting to localhost [127.0.0.1] port 22.
debug1: Connection established.
debug1: identity file /mnt/driveB/mrkelly/.ssh/id_rsa type 1
debug1: identity file /mnt/driveB/mrkelly/.ssh/id_rsa-cert type -1
debug1: identity file /mnt/driveB/mrkelly/.ssh/id_dsa type 2
debug1: identity file /mnt/driveB/mrkelly/.ssh/id_dsa-cert type -1
debug1: identity file /mnt/driveB/mrkelly/.ssh/id_ecdsa type -1
debug1: identity file /mnt/driveB/mrkelly/.ssh/id_ecdsa-cert type -1
debug1: identity file /mnt/driveB/mrkelly/.ssh/id_ed25519 type -1
debug1: identity file /mnt/driveB/mrkelly/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 8d:cb:f2:94:da:97:7b:0d:ee:e6:bb:8e:3f:41:ae:d8
debug1: Host 'localhost' is known and matches the ECDSA host key.
debug1: Found key in /mnt/driveB/mrkelly/.ssh/known_hosts:3
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /mnt/driveB/mrkelly/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to localhost ([127.0.0.1]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Requesting X11 forwarding with authentication spoofing.
debug1: Sending environment.
debug1: Sending env LANGUAGE = 
debug1: Sending env LANG = en_US.UTF-8
Last login: Wed May 13 15:58:17 2015 from localhost.localdomain
Environment:
  LANGUAGE=
  LANG=en_US.UTF-8
  USER=mrkelly
  LOGNAME=mrkelly
  HOME=/mnt/driveB/mrkelly
  PATH=/usr/local/bin:/usr/bin
  MAIL=/var/mail/mrkelly
  SHELL=/bin/zsh
  SSH_CLIENT=127.0.0.1 49967 22
  SSH_CONNECTION=127.0.0.1 49967 127.0.0.1 22
  SSH_TTY=/dev/pts/4
  TERM=xterm-256color
  DISPLAY=localhost:11.0
  SELINUX_ROLE_REQUESTED=
  SELINUX_LEVEL_REQUESTED=
  SELINUX_USE_CURRENT_RANGE=
  XDG_SESSION_ID=21
  XDG_RUNTIME_DIR=/run/user/1000
  XDG_SEAT=seat0
  XDG_VTNR=1
Running /usr/bin/xauth remove unix:11.0
/usr/bin/xauth add unix:11.0 MIT-MAGIC-COOKIE-1 62951eb22f06d56df8189ee23126a19e

(Working, passwordless login)

Now, we restart sshd.service and show the non-working version.

  • Host terminal 

    sudo systemctl start sshd

  • Client terminal 

    ~$ ssh -v `whoami`@localhost
OpenSSH_6.6.1, OpenSSL 1.0.1k-fips 8 Jan 2015
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug1: Connecting to localhost [127.0.0.1] port 22.
debug1: Connection established.
debug1: identity file /mnt/driveB/mrkelly/.ssh/id_rsa type 1
debug1: identity file /mnt/driveB/mrkelly/.ssh/id_rsa-cert type -1
debug1: identity file /mnt/driveB/mrkelly/.ssh/id_dsa type 2
debug1: identity file /mnt/driveB/mrkelly/.ssh/id_dsa-cert type -1
debug1: identity file /mnt/driveB/mrkelly/.ssh/id_ecdsa type -1
debug1: identity file /mnt/driveB/mrkelly/.ssh/id_ecdsa-cert type -1
debug1: identity file /mnt/driveB/mrkelly/.ssh/id_ed25519 type -1
debug1: identity file /mnt/driveB/mrkelly/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 8d:cb:f2:94:da:97:7b:0d:ee:e6:bb:8e:3f:41:ae:d8
debug1: Host 'localhost' is known and matches the ECDSA host key.
debug1: Found key in /mnt/driveB/mrkelly/.ssh/known_hosts:3
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /mnt/driveB/mrkelly/.ssh/id_rsa
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Offering DSA public key: /mnt/driveB/mrkelly/.ssh/id_dsa
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Trying private key: /mnt/driveB/mrkelly/.ssh/id_ecdsa
debug1: Trying private key: /mnt/driveB/mrkelly/.ssh/id_ed25519
debug1: Next authentication method: keyboard-interactive
Password:

(and there it is).

A couple final remarks : I recently had to reformat the installation drive and I'm using my folder that was stored on another drive. When reinstalling Fedora 21, I was forced to create a home directory on that drive, but then I switched my user account to point to a backed-up version of my old home directory after I created the mount points for the other drives.

EDIT : Passwordless login also works using the following invocations : 

/usr/sbin/sshd -D

(no daemon, but no debug messages)

/usr/shbin/sshd

( which I guess constitutes a temporary workaround,since this does invoke the daemon)

EDIT2: output from sudo journalctl -u sshd after sudo systemctl restart sshd (logging level debug3)

I'm at the character limit for submissions, so I'll link to a plain text file in dropbox with the output.

Thanks for any help you can provide!

Improve this question
8
  • So the only thing you're doing different between these test cases is that in the working one you're using sshd -d and in the other one you're not? And enabling debug logging makes it work?
    – user41515
    May 13, 2015 at 23:37
  • Not quite-- in one case I've started sshd directly, as you say, but in the other, I've used Fedora's systemctl command, viz. systemctl start sshd.service. What I haven't done is try involking it explicitly WITHOUT the debug option, I'll report back in a second.
    – Mark Kelly
    May 13, 2015 at 23:40
  • I'd try to get the -d into the other startup method, convince it to log the output somewhere. Or let it get started then strace -f -p it and connect.
    – user41515
    May 13, 2015 at 23:56
  • What are you seeing in the logs in the failed case? (Possibly with LogLevel increased in sshd_config....)
    – mattdm
    May 14, 2015 at 15:52
  • Updated with info requested by mattdm
    – Mark Kelly
    May 14, 2015 at 19:02

3 Answers 3

Reset to default
1

This is an SELinux problem, caused by your home directory being in a weird location. The thing you did to disable it, didn't. SELinux isn't a service, and systemctl status selinux is just telling you that ("not-found (Reason: No such file or directory)").

You could run setenforce permissive or otherwise disable SELinux, but this kind of like taking the doors off of your house just because you misplaced your housekeys once. Running chcon -t ssh_home_t ~/.ssh/ should do it.

How do I know? Because that the type for the files in ~/.ssh on my system. But I could have installed sepolicy from the policycoreutils-devel package, and run sepolicy manpage -t ssh_d to generate the ssh_selinux.8 man page, which documents the actual policy in place.

Really, I'd most suggest forgetting all of that, just mounting your drive at /home rather than /mnt/driveB, and then running restorecon -R -v /home.

Improve this answer
1
  • I'll accept this answer, but I don't think I'll quite be able to apply it. This device has several different shares of different formats going, and stopping SELinux has resolved several issues with each of them. That said, I'm sure there's a way to configure SELinux to handle them, but it will take quite a lot of doing.
    – Mark Kelly
    May 18, 2015 at 1:30
0

It's SELinux. Even though I stopped and disabled it using systemctl, I still needed to use /usr/bin/setenforce 0. Passwordless login now works as expected.

Improve this answer
1
  • Sorry I missed that earlier -- I skimmed over how you attempted to disable SELinux. See my answer for some alternate suggestions.
    – mattdm
    May 16, 2015 at 2:51
0

I had (what I believe to be) a similar problem, wherein SSH attempts to users whose home directories were based on an NFS share were being repeatedly rejected, with only the following cryptic message written to the (debug/verbose) output of ssh:

debug1: key_parse_private2: missing begin marker

As indicated by the other answers, it turned out to be an SELinux problem. However, it appears that there is a solution more fine-grained than disabling SELinux enforcement entirely.

On the host which is serving the home directories, run the following command to tell SELinux to label the directory containing the user home directories the same way as the default home-directory-containing directory /home:

sudo semanage fcontext -a -e /home /path/to/custom/homedir/container/on/server

Then run this command to recursively reset SELinux labels on the home directories:

sudo restorecon -R /path/to/custom/homedir/container/on/server

If you are using NFS (as in my case), then take execute the following additional command on each client where the home directories are mounted, in order to set SELinux to allow the use of NFS-mounted home directories:

sudo setsebool -P use_nfs_home_dirs on

References:

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.