3

I'm setting up NFSv4.2 with MIT Kerberos (sec=krb5p) on two Hyper-V VMs running Debian 11 (Bullseye). When I use machine-based authentication (sec=sys), everything works fine. With Kerberos (sec=krb5p), I'm able to mount the share on the client, but I see Permission denied when I try to access the share. I've documented all of this in detail below, including all relevant configuration info that came to mind. I'm a bit out of my depth. What am I missing or doing wrong?

NFS works with sec=sys

/etc/exports on the server:

/exports/ned 192.168.1.0/24(sec=sys,rw,no_subtree_check)

/etc/fstab on the client:

test-debian-server:/exports/ned /imports/ned nfs sec=sys 0 0

Testing the share:

ned@test-debian-desktop:/imports$ su - -c "nfsstat -m"
Password:
/imports/ned from test-debian-server:/exports/ned
 Flags: rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.63,local_lock=none,addr=192.168.1.62

ned@test-debian-desktop:/imports$ id
uid=2001(ned) gid=2001(ned) groups=2001(ned)
ned@test-debian-desktop:/imports$ ls -l
total 4
drwxr-xr-x 2 ned ned 4096 Jan 26 17:57 ned
ned@test-debian-desktop:/imports$ cd ned
ned@test-debian-desktop:/imports/ned$ ls
ned@test-debian-desktop:/imports/ned$ touch test.txt
ned@test-debian-desktop:/imports/ned$ ls
test.txt

Verifying test.txt on the server:

ned@test-debian-server:/exports$ id
uid=2001(ned) gid=2001(ned) groups=2001(ned)
ned@test-debian-server:/exports$ ls -l
total 4
drwxr-xr-x 2 ned ned 4096 Jan 26 19:11 ned
ned@test-debian-server:/exports$ ls -l ned
total 0
-rw-r--r-- 1 ned ned 0 Jan 26 19:11 test.txt

Permission denied with sec=krb5p

Server configuration

/etc/hosts:

127.0.0.1       localhost
127.0.1.1       test-debian-server.test test-debian-server
192.168.1.63    test-debian-desktop
192.168.1.62    test-debian-server

/etc/default/nfs-kernel-server:

RPCNFSDCOUNT=8
RPCNFSDPRIORITY=0
RPCMOUNTDOPTS="--manage-gids"
NEED_SVCGSSD=""
RPCSVCGSSDOPTS=""

/etc/default/nfs-common:

NEED_STATD=
STATDOPTS=
NEED_IDMAPD=
NEED_GSSD=

/etc/exports:

/exports/ned 192.168.1.0/24(sec=krb5p,rw,no_subtree_check)

/etc/krb5.conf (heavily abridged):

[libdefaults]
        default_realm = TEST

        kdc_timesync = 1
        ccache_type = 4
        forwardable = true
        proxiable = true

[realms]
        TEST = {
                kdc = test-debian-server
                admin_server = test-debian-server
        }

Kerberos principals:

root@test-debian-server:~# kadmin.local listprincs
K/M@TEST
host/test-debian-desktop@TEST
host/test-debian-server@TEST
kadmin/admin@TEST
kadmin/changepw@TEST
kadmin/test-debian-server@TEST
kiprop/test-debian-server@TEST
krbtgt/TEST@TEST
nfs/test-debian-desktop@TEST
nfs/test-debian-server@TEST

Kerberos keytab:

root@test-debian-server:~# klist -ke
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   2 nfs/test-debian-server@TEST (aes256-cts-hmac-sha1-96)
   2 nfs/test-debian-server@TEST (aes128-cts-hmac-sha1-96)
   2 host/test-debian-server@TEST (aes256-cts-hmac-sha1-96)
   2 host/test-debian-server@TEST (aes128-cts-hmac-sha1-96)

NFS effective domain:

root@test-debian-server:~# nfsidmap -d
test

Permissions on exports:

root@test-debian-server:~# ls -ld /exports /exports/*
drwxr-xr-x 3 root root 4096 Jan 26 17:58 /exports
drwxr-xr-x 2 ned  ned  4096 Jan 26 19:11 /exports/ned

Client configuration

/etc/hosts:

127.0.0.1       localhost
127.0.1.1       test-debian-desktop.test        test-debian-desktop
192.168.1.63    test-debian-desktop
192.168.1.62    test-debian-server

/etc/default/nfs-common: same as server.

/etc/fstab:

test-debian-server:/exports/ned /imports/ned nfs sec=krb5p 0 0

/etc/krb5.conf: same as server.

Kerberos keytab:

root@test-debian-desktop:~# klist -ke
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   3 nfs/test-debian-desktop@TEST (aes256-cts-hmac-sha1-96)
   3 nfs/test-debian-desktop@TEST (aes128-cts-hmac-sha1-96)
   2 host/test-debian-desktop@TEST (aes256-cts-hmac-sha1-96)
   2 host/test-debian-desktop@TEST (aes128-cts-hmac-sha1-96)

NFS effective domain:

root@test-debian-desktop:~# nfsidmap -d
test

Permissions on imports (after unmounting share):

root@test-debian-desktop:~# umount /imports/ned
root@test-debian-desktop:~# ls -ld /imports /imports/*
drwxr-xr-x 3 root root 4096 Jan 26 19:08 /imports
drwxr-xr-x 2 root root 4096 Jan 26 17:02 /imports/ned

(Note: I get the same result when I chown ned:ned /imports/ned. I think root:root is correct, to prevent ned from accidentally writing to the local disk if the mount fails?)

Permission denied

After rebooting server and client:

ned@test-debian-desktop:/imports$ su -c "nfsstat -m"
Password:
/imports/ned from test-debian-server:/exports/ned
 Flags: rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5p,clientaddr=192.168.1.63,local_lock=none,addr=192.168.1.62

ned@test-debian-desktop:/imports$ ls -l
total 4
drwxr-xr-x 2 ned ned 4096 Jan 26 19:11 ned
ned@test-debian-desktop:/imports$ cd ned
bash: cd: ned: Permission denied
ned@test-debian-desktop:/imports$

Sometimes ned can list the share with ls -l initially but can't cd into it, as in the session above; a minute later, if ned tries to ls -l, they will see Permission denied and question marks for the ned folder's attributes. When this happens, I can do the following:

ned@test-debian-desktop:/imports$ ls -l
ls: cannot access 'ned': Permission denied
total 0
d????????? ? ? ? ?            ? ned
ned@test-debian-desktop:/imports$ su -c "ls -l /imports"
Password:
total 4
drwxr-xr-x 2 ned ned 4096 Jan 26 19:11 ned
ned@test-debian-desktop:/imports$ ls -l
total 4
drwxr-xr-x 2 ned ned 4096 Jan 26 19:11 ned
ned@test-debian-desktop:/imports$

This appears to be due to attribute cache expiration: if I disable the attribute cache with noac in /etc/fstab on the client and remount, ned reliably sees Permission denied and question marks when calling ls -l.

What I've tried

On both client and server, I have:

  • Set Verbosity = 4 in /etc/idmapd.conf
  • Used rpcdebug to set all flags for nfs, nfsd, nlm, and rpc
  • Scoured the systemd journal for relevant information (after rebooting and attempting to access the share).

I have checked systemctl status to verify that the following units are active, and I have read through the recent journal entries listed for each (and found no obvious indicators of what's wrong):

  • Server:
    • krb5-admin-server
    • krb5-kdc
    • nfs-kernel-server
    • nfs-idmapd
    • nfs-mountd
    • nfs-server
    • rpc-svcgssd
    • rpc-gssd
  • Client:
    • rpc-gssd
    • imports-ned.mount

As far as I can tell, Kerberos is authenticating correctly.

I've read through every relevant manpage and online manual I can find and every question on StackExchange that appeared to be related, and I haven't been able to fix the issue or get either client or server to generate an error message to tell me why permission is being denied.

Edited to add host/test-debian-desktop and host/test-debian-server Kerberos principals and keys in accordance with Unix Application Servers - MIT Kerberos Documentation

What I was missing

The chosen answer does a good job of explaining the reason my configuration wasn't working. In case anyone is in the same situation, here are the details of what I was missing - and please read the answer below for why.

Kerberos authenticates individual users (which for some reason I thought it didn't). So for ned to connect, Kerberos must have a ned@TEST principal, and the Linux user ned on the client PC must kinit to get a Ticket-Granting Ticket (TGT) for this principal. Then everything works beautifully. To more or less automate this, I did the following:

/etc/krb5/user/<uid>/client.keytab: holds keys for ned and is only readable by ned! I used 0600 permissions, with owner and group both as ned.

With this keytab, I can simply invoke kinit -ki to log in without needing to enter a password. The quick-and-dirty way I automated this was to add it to ~/.gnomerc, ~/.bashrc, and ~/.zshrc (since I use Gnome, Bash, and Zsh).

Additionally, since I deploy these configurations via Ansible and have some systems that share dotfiles but don't need NFS access, I got a bit fancier with Bash and Zsh, only running kinit if it and the client keytab are present:

~/.bashrc:

# Grab or renew a Kerberos ticket.
if type -t kinit >/dev/null && [[ -r /etc/krb/user/$(id -u)/client.keytab ]]
then
  kinit -ki
fi

~/.zshrc:

# Grab or renew a Kerberos ticket.
if type kinit >/dev/null && [[ -r /etc/krb/user/$(id -u)/client.keytab ]]
then
  kinit -ki
fi
2
  • Can you please copy here the value of klist command, without any flags?
    – aviro
    Jan 27, 2022 at 13:16
  • You definitely found the problem. I had no credentials cache, i.e. no tickets, on client. When I add ned@TEST to Kerberos and kinit on Ned, I can access the share correctly. I'm not sure if I should add this to my client's keytab or what exactly is the right way to set it up; if you let me know in an answer, I'll accept it. Thanks!
    – Dylan L
    Jan 27, 2022 at 19:45

1 Answer 1

2

The reason you got "permission denied" was because you didn't have a Kerberos TGT (ticket-granting-ticket) in your credentials cache, as you could see in the output of klist. You had to run kinit for the account and type your password. Once you authenticated and got a TGT ticket cached, you could use this ticket to authenticate to the NFS server.

You need to remember that the whole idea of Kerberos authentication is that you need to "prove" your identity to the NFS server (or any other service for that matter). You "prove" it by presenting your cached TGT that you generated by typing your password (that only you know) to the kinit command. So usually that's the way it's supposed to work. You can change the expiration and renew times in your krb5/sssd/whatever to make your tickets last longer before expiring, but eventually at some point you'll need to re-authenticate one way or the other.

Of course you could always create a keytab for your account. Don't add it to the client's keytab! The client keytab is for the client alone. You will need to create new keytab in a new file for your account and keep this file secure and private with minimum permissions, and use this keytab to generate the TGT in your credential cache without typing a password interactively. Theoretically you can even create a crontab that will do it for you periodically, but that's less secure, as someone could steal your keytab at some point (if he has root access, for instance) and pretend to be you for other services, but that's for your consideration.

Also, if you're going down the keytab road, don't forget that any time you change the account's password, you'll also need to update your keytab.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .