SSH Pub Key Login without Password

Question

I am configuring an SSH Server on Ubuntu 14.04.1 Server Edition. The goal is to use public key authentication only and only permit certain usernames.

My public and private keys are created and a password is set on the private key. I copied the public key to the server (exact process described below) and was able to SSH into the server for the first time using this command

ssh -i /path/to/id_rsa -p 50000 [email protected]

I was prompted for the private key password and allowed to log in. Great.

However, every time I SSH back into the server I am no longer prompted for my private key password. I can even log in without specifying the path to my private key, like so:

ssh -p 50000 [email protected]

I can even delete ~/.ssh/known_hosts on the client (Mac OS X 10.8) and successfully SSH into the server via

ssh -p 50000 [email protected]

So, my questions are:

1. What is the server using to authenticate me if it's not using my private key, the key's password, nor the contents of the client's ~/.ssh/known_hosts ?
2. Is my SSH server insecure? Copy of sshd_config included below.

Thanks for your help.

Key creation process

- at your computer (not the server) do
    - generate the keys: ssh-keygen -t rsa -b 4096
        - public key is saved at ~/.ssh/id_rsa.pub
        - private key is saved at ~/.ssh/id_rsa
- copy id_rsa.pub to server and append to ~/.ssh/authorized_keys
    - ssh-copy-id username@remotehost
    - a more secure method is to copy via usb drive
        - make a backup: cp authorized_keys authorized_keys.original
        - add public key to file: cat id_rsa.pub >> authorized_keys
    - if your home directory is encrypted (mine is)
        - in sshd_config: AuthorizedKeysFile /etc/ssh/%u/authorized_keys
        - move the authorized_keys file to /etc/ssh/me/authorized_keys
            - mkdir /etc/ssh/me
                - chmod u=rwx,go= /etc/ssh/me
                - chown me /etc/ssh/me
            - mv ~/.ssh/authorized_keys /etc/ssh/me/authorized_keys
                - chmod u=rw,go= /etc/ssh/me/authorized_keys
                - chown me /etc/ssh/me/authorized_keys

sshd_config

# User modified sshd_config.
# See the sshd_config(5) manpage for details.


#### Networking options ####

# Listen on a non-standard port > 1024. Default is 22.
Port 50000

# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0

# Only use protocol version 2.
Protocol 2

X11Forwarding no
X11DisplayOffset 10

# Helps the server recognize problems and the connection will be killed.
TCPKeepAlive yes

#### Networking options ####


#### Key Configuration ####

# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

# Privilege Separation is turned on for security.
UsePrivilegeSeparation yes

# Use public key authentication
PubkeyAuthentication yes
#AuthorizedKeysFile %h/.ssh/authorized_keys
AuthorizedKeysFile /etc/ssh/%u/authorized_keys

#### Key Configuration ####


### Authentication ###

# 30 seconds to enter your key passphrase.
LoginGraceTime 30

# No root login.
PermitRootLogin no

# Force permissions checks on keyfiles and directories.
StrictModes yes

HostbasedAuthentication no

# Don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication.
IgnoreUserKnownHosts yes

# To enable empty passwords, change to yes (NOT RECOMMENDED).
PermitEmptyPasswords no

# Disable challenge and response auth. Unnecessary when using keys.
ChallengeResponseAuthentication no

# Disable the use of passwords completly, only use public/private keys.
PasswordAuthentication no

# Using keys, no need for PAM (Pluggable Authentication Modules).
# Also allows SSHD to be run as a non-root user.
UsePAM no

# Don't use login(1)
UseLogin no

AllowUsers me

### Authentication ###


### Misc ###

# Logging
SyslogFacility AUTH
LogLevel VERBOSE

# Print the last time the user logged in.
PrintLastLog yes

# Maximum number of concurrent unauthenticated connections to the SSH daemon (the number of users still logging in).
MaxStartups 10:30:60

# Display login banner.
Banner /etc/issue.net

# Allow client to pass locale environment variables.
# Accept language variables to help the shell session display properly for the client.
AcceptEnv LANG LC_*

# External file transfer daemon to use for sftp requests.
Subsystem sftp /usr/lib/openssh/sftp-server

# Should the SSH daemon itself read and display the message of the day file.
PrintMotd no

### Misc ###

Firewall Configuration

- allow incoming connections to port 50000
    - sudo ufw allow in 50000
- Rate-limit the connections
    For example, deny connections if an IP address has attempted to initiate
    6 or more connections in the last 30 seconds.
    - sudo ufw limit ssh

Answer 1

1. Your client computer likely cached the credentials so it probably is still using your private key. If you restart your computer you should have to enter in the passphrase again. (also, if your keys are at ~/.ssh/, then that is the default location ssh checks for them)

2. Your SSH configuration looks fine, as does your firewall settings (assuming it is set to default to deny). I cannot comment on the security of your system as a whole.

Let me know if this misses anything.