Weird behaviour in per-user SSH password authentication

Question

I'm using NixOS. Here's my SSHd config:

  services.openssh.enable = true;
  services.openssh.passwordAuthentication = false;
  services.openssh.challengeResponseAuthentication = false;
  services.openssh.permitRootLogin = "no";

  services.openssh.extraConfig = ''
    Match User dropbox
      PasswordAuthentication yes
  '';

As you can see, I'm disallowing SSH login with passwords, but as an exception, allowing it for the user dropbox. This Nix syntax results into following sshd_config:

UsePAM yes

AddressFamily any
Port 22

X11Forwarding no

Subsystem sftp /nix/store/6fkb47ri4xndlpszjrbw8ggd3vmb6in7-openssh-8.1p1/libexec/sftp-server

PermitRootLogin no
GatewayPorts no
PasswordAuthentication no
ChallengeResponseAuthentication no

PrintMotd no # handled by pam_motd

AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 /etc/ssh/authorized_keys.d/%u

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

KexAlgorithms [email protected],diffie-hellman-group-exchange-sha256
Ciphers [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
MACs [email protected],[email protected],[email protected],hmac-sha2-512,hmac-sha2-256,[email protected]

LogLevel VERBOSE

UseDNS no

Match User dropbox
  PasswordAuthentication yes

On the surface, this seems to work. It doesn't allow password login for other users, but allows for dropbox. This is with ssh -v dropbox@poi:

debug1: Next authentication method: password
dropbox@poi's password:
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.

However, it doesn't accept dropbox's password. It's the exact same (simple, three letter toy password) password that allows me to log in, so the password isn't wrong. I've even copy-pasted it to avoid the caps-lock trap. The same password that allows login doesn't allow SSH login.

HOWEVER, if I set passwordAuthentication allowed for ALL users, then dropbox magically is able to log in with it's password. I've verified that the Match part is always in the end of sshd_config, so this is not about ordering issues.

I've never heard this kind of behaviour. Is there any tricks that would allow me to debug this?

Answer 1

If you look at the nix source here you can see they are using passwordAuthentication to set a PAM rule. Effectively:

security.pam.services.sshd.unixAuth = <passwordAuthentication>;

There is no way I can come up with to disable PAM in the sshd config [note 2], the nix module hard-codes in "UsePAM yes" to the top of the file. What we can do, instead, is override that setting so that PAM will accept your password.

services.openssh.enable = true;
services.openssh.permitRootLogin = "no";
services.openssh.passwordAuthentication= false;
services.openssh.challengeResponseAuthentication = false;
services.openssh.extraConfig = "
    Match User bootstrap
    PasswordAuthentication yes
    Match All
";
security.pam.services.sshd.unixAuth = pkgs.lib.mkForce true;

Explanation:

PAM is a service available on most Linux systems that handles user authentication, among other things. It can be configured to authenticate in various ways. From the NixOS options reference:

security.pam.services.<name?>.unixAuth
   Description:   Whether users can log in with passwords defined in /etc/shadow.
   Default value: true

The challengeResponseAuthentication line is needed to really prevent password login, since challengeResponseAuthentication and passwordAuthentication refer to two different modes of "password" based login, and they are enabled/disabled independently.

Note: if you forget to use mkForce (or something similar), nix will yell at you:

error: The option `security.pam.services.sshd.unixAuth' has conflicting definitions, in `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/services/networking/ssh/sshd.nix' and `/etc/nixos/configuration.nix'.

Note 2: Some (maybe all, IDK) openssh options cannot be changed after they have been set in the config file. Subsequent definitions of the same value are ignored. Therefore, entering "UsePAM no" inside your extraConfig has no effect, since "UsePAM" is already set to "yes" at the top of the sshd_config.

Note 3: NixOS v20.03.1619 (Markhor), OpenSSH_8.2p1

If anyone has more specific info on specifically how PAM or sshd config, or NixOS work, mention it in the comments and I'll add it to the answer.