updates: 5 (20171209)
updates: 5 (20171210)
mount -t nfs4 [SERVER IP]:/archlinux /mntworks.ss -ntp | grep 2049the client establishes a connection to the server before systemd begins.- NSF4 id mapper can only be used with Kerberos?
the problem
I am attempting to set up a diskless node/workstation/system. The OS (4.13.12-1-ARCH) is installed on the SERVER /srv/archlinux. After a successful netboot from GRUB to NFSv4, systemd begins but fails at multiple stages, for example:
- Failed to mount Kernel Configuration File System.
- Failed to mount Kernel Debug File System.
- Failed to mount Huge Pages File System
- Failed to start Load/Save Random Seed.
- Failed to mount /tmp.
- Failed to start Rebuild Journal Catalog.
- Then ends with
Not tainted 4.13.12-1-ARCH #1...
Or,
- Failed to mount POSIX Message Queue File System.
- Failed to start Remount Root and Kernel File System.
- Failed to mount Huge Pages File System.
- Failed to mount Kernel Debug File System.
- Failed to mount Kernel Configuration File System.
- Then ends with
Not tainted 4.13.12-1-ARCH #1...
I suspect the failures are caused by an incorrect configuration of NFSv4 or the local network.
rpc.idmapd
/etc/idmapd.conf
[General]
Verbosity = 7
Pipefs-Directory = /var/lib/nfs/rpc_pipefs
Domain = localdomain
[Mapping]
Nobody-User = nobody
Nobody-Group = nobody
[Translation]
Method = nnswitch
/etc/exports
(printed using # exportfs -v)
/srv <world>(rw,sync,wdelay,hide,no_subtree_check,fsid=0,sec=sys,no_root_squash,no_all_squash)
/srv/archlinux <world>(rw,sync,wdelay,hide,no_subtree_check,sec=sys,no_root_squash,no_all_squash)
(Exposed to "world" for debugging purposes)
Running rpc.idmapd -fvvv on a separate tty during bootup logs the following:
rpc.idmapd: libnfsidmap: using domain: localdomain
rpc.idmapd: libnfsidmap: Realms list: 'LOCALDOMAIN'
rpc.idmapd: libnfsidmap: processing 'Method' list
rpc.idmapd: libnfsidmap: loaded plugin /usr/lib/libnfsidmap/nsswitch.so for method nsswitch
rpc.idmapd: Expiration time is 600 seconds.
rpc.idmapd: Opened /proc/net/rpc/nfs4.nametoid/channel
rpc.idmapd: Opened /proc/net/rpc/nfs4.idtoname/channel
rpc.idmapd: nfsdcb: authbuf=* authtype=user
rpc.idmapd: nfs4_uid_to_name: calling nsswitch->uid_to_name
rpc.idmapd: nfs4_uid_to_name: nsswitch->uid_to_name returned 0
rpc.idmapd: nfs4_uid_to_name: final return value is 0
rpc.idmapd: Server : (user) id "0" -> name "root@localdomain"
If exportfs sec=sys, it continues like:
rpc.idmapd: nfsdch: authbuf=* authtype=user
rpc.idmapd: nfs4_name_to_uid: calling nsswitch->name_to_uid
rpc.idmapd: nss_getpwnam: name '0' domain 'localdomain': resulting localname '(null)'
rpc.idmapd: nss_getpwnam: name '0' does not map into domain 'localdomain'
rpc.idmapd: nfs4_name_to_uid: nsswitch->name_to_uid returned -22
rpc.idmapd: nfs4_name_to_uid: final return value is -22
rpc.idmapd: Server : (user) name "0" -> id "99"
(stops here)
+(20171209) After making sure that the /etc/hostname for the CLIENT was set to client2 (duh), if exportfs sec=none or sec=sys, it continues like:
rpc.idmapd: nfsdch: authbuf=* authtype=group
rpc.idmapd: nfs4_gid_to_name: calling nsswitch->gid_to_name
rpc.idmapd: nfs4_gid_to_name: nsswitch->gid_to_name returned 0
rpc.idmapd: nfs4_gid_to_name: final return value is 0
rpc.idmapd: Server : (group) id "190" -> name "systemd-journal@localdomain"
rpc.idmapd: nfsdch: authbuf=* authtype=user
rpc.idmapd: nfs4_name_to_uid: calling nsswitch->name_to_uid
rpc.idmapd: nss_getpwnam: name '0' domain 'localdomain': resulting localname '(null)'
rpc.idmapd: nss_getpwnam: name '0' does not map into domain 'localdomain'
rpc.idmapd: nfs4_name_to_uid: nsswitch->name_to_uid returned -22
rpc.idmapd: nfs4_name_to_uid: final return value is -22
rpc.idmapd: Server : (user) name "0" -> id "99"
(stops here)
If I instead change method from nsswitch to static (UID mapping in NFS)
/etc/idmapd.conf
...
[Translation]
Method = static
[Static]
root@localdomain = root
The rpc.idmapd -fvvv on a separate tty during bootup logs the following:
rpc.idmapd: libnfsidmap: using domain: localdomain
rpc.idmapd: libnfsidmap: Realms list: 'LOCALDOMAIN'
rpc.idmapd: libnfsidmap: processing 'Method' list
rpc.idmapd: static_getpwnam: name 'root@localdomain' mapped to 'root'
rpc.idmapd: static_getpwnam: group 'root@localdomain' mapped to ' root'
rpc.idmapd: libnfsidmap: loaded plugin /usr/lib/libnfsidmap/static.so for method static
rpc.idmapd: Expiration time is 600 seconds.
rpc.idmapd: Opened /proc/net/rpc/nfs4.nametoid/channel
rpc.idmapd: Opened /proc/net/rpc/nfs4.idtoname/channel
rpc.idmapd: nfsdcb: authbuf=* authtype=user
rpc.idmapd: nfs4_uid_to_name: calling static->uid_to_name
rpc.idmapd: nfs4_uid_to_name: static->uid_to_name returned 0
rpc.idmapd: nfs4_uid_to_name: final return value is 0
rpc.idmapd: Server : (user) id "0" -> name "root@localdomain"
If exportfs sec=sys, it continues like:
rpc.idmapd: nfsdch: authbuf=* authtype=user
rpc.idmapd: nfs4_name_to_uid: calling static->name_to_uid
rpc.idmapd: nfs4_name_to_uid: static->name_to_uid returned -2
rpc.idmapd: nfs4_name_to_uid: final return value is -2
rpc.idmapd: Server : (user) name "0" -> id "99"
(stops here)
If exportfs sec=none, it continues like:
rpc.idmapd: nfsdch: authbuf=* authtype=group
rpc.idmapd: nfs4_gid_to_name: calling static->gid_to_name
rpc.idmapd: nfs4_gid_to_name: static->gid_to_name returned -2
rpc.idmapd: nfs4_gid_to_name: final return value is -2
rpc.idmapd: Server : (group) id "190" -> name "nobody"
rpc.idmapd: nfsdch: authbuf=* authtype=user
rpc.idmapd: nfs4_name_to_uid: calling static->name_to_uid
rpc.idmapd: nfs4_name_to_uid: static->name_to_uid returned -2
rpc.idmapd: nfs4_name_to_uid: final return value is -2
rpc.idmapd: Server : (user) name "0" -> id "99"
(stops here)
Similar problems with the user ID mapping:
- NFSv4 User Mapping
- NFS user mapping
- Mapping UID and GID of local user to the mounted NFS share
- And many many more... Often related to a switch from NFSv3 to NFSv4, and rarely about netboot.
troubleshooting
- No firewall
- No Kerberos, LDAP, etc.
- No SELinux
- The user
rootexists on both SERVER and CLIENT, with the same password.
SERVER
All other relevant configuration files for NFSv4 I could identify on the SERVER.
/etc/nsswitch.conf
passwd: compat mymachines systemd
group: compat mymachines systemd
shadow: compat
publickey: files
hosts: files mymachines resolve [!UNAVAIL=return] dns myhostname
networks: files
protocols: files
services: files
ethers: files
rpc: files
netgroup: files
/etc/nfs.conf
(all settings commented out)
/etc/conf.d/nfs-common.conf
(all settings commented out)
network configuration
- How to set the domain name on GNU/Linux?
- Archlinux Wiki Network configuration: Set the hostname
- Archlinux Wiki Network configuration: Local network hostname resolution
The SERVER hostname is server and has 3 network devices (nd[1-3]). The Gateway default via 192.168.0.1 nd1.
/etc/hosts
127.0.0.1 localhost.localdomain localhost
::1 ip6.localhost localhost
192.168.0.101 nd1.localdomain server servernd1
192.168.1.101 nd2.localdomain server servernd2
192.168.2.101 nd3.localdomain server servernd2
192.168.1.102 client1.localdomain client1
192.168.2.102 client2.localdomain client2
/etc/resolveconf.conf
name_servers=192.168.0.1
# hostname -f
# nd1.localdomain
# hostname -i
192.168.0.101 192.168.1.101 192.168.2.101
# getent hosts IP -> the corresponding line in /etc/hosts
# getent ahosts HOSTNAME -> the corresponding line in /etc/hosts
# ping -c 3 server.localdomain -> 0% packet loss
# id -u root -> 0
# id -un 0 -> root
Display the system's effective NFSv4 domain name on stdout.
# nfsidmap -d -> localdomain
Display on stdout all keys currently in the keyring used to cache ID mapping results. These keys are visible only to the superuser.
# nfsidmap -l -> nfsidmap: '.id_resolver' keyring was not found.
CLIENT
/etc/hostname +(20171209)
client2
/etc/hosts
(exactly the same as the hosts file on the server)
/etc/resolveconf.conf
name_servers=192.168.0.1
/etc/idmapd.conf
(exactly the same as the idmapd.conf file on the server)
/etc/fstab
# sys=sec or sys=none to correspond to server export settings.
/dev/nfs / nfs rw,hard,rsize=9151,sec=sys,clientaddr=192.168.2.102 0 0
devtmpfs /dev devtmpfs defaults
proc /proc proc defaults
none /run tmpfs defaults
sys /sys sysfs defaults
run /run tmpfs defaults
tmp /tmp tmpfs defaults
The fstab was defined by comparing the mounted directories on the server using findmnt -A.
net_nfs4
- +(20171210) NFS version on SERVER and CLIENT
cat /proc/fs/nfsd/versions -> -2 +3 +4 +4.1 +4.2 - On SERVER and CLIENT
cat /sys/module/nfsd/parameters/nfs4_disable_idmapping -> N. - On SERVER
echo "options nfsd nfs4_disable_idmapping=0" > /etc/modprobe.d/nfsd.conf. - On CLIENT the
/sys/module/nfs/parameters/nfs4_disable_idmappingdoes not exist, and not sure how to manually create it as the/sysis read only. - +(20171210) On CLIENT
echo "options nfs nfs4_disable_idmapping=0" > /etc/modprobe.d/nfs.conf.
The CLIENT IP is 192.168.2.102/24. The CLIENT network device is connected to SERVER nd2 192.168.2.101/24 (hostname: servernd2).
The network information during boot:
:: running early hook [udev]
starting version 235
:: running hook [udev]
:: Triggering uevents...
:: running hook [net_nfs4]
IP-Config: eth0 hardware address [CLIENT NETWORK DEVICE MAC] mtu 1500 DHCP
hostname client2 IP-Config: eth0 guessed broadcast address 192.168.2.255
IP-Config: eth0 complete (from 192.168.0.101):
address: 192.168.2.102 broadcast: 192.168.2.255 netmask: 255.255.255.0
gateway: 192.168.2.101 dns0 : 192.168.0.1 dns1 : 0.0.0.0
host : client2
domain : localdomain
rootserver: 192.168.0.101 rootpath: /srv/archlinux
filename : /netboot/grub/i386-pc/core.0
NFS-Mount: 192.168.2.101:/archlinux
Waiting 10 seconds for device /dev/nfs ...
(systemd takes over from here)
Why the NSFv4 errors occur?
Server : (group) id "190" -> name "nobody"
With NFSv4, things change: users are mapped by username, and the mapping between user names and user IDs is handled by a process called "ID map daemon" (idmapd). In particular, NFSv4 clients and server should use the same domain for the mapping to work properly, otherwise requests will be mapped to the anonymous user/group. -- Trying out NFSv4 (on Linux and Solaris) -- March 15th, 2012 - 13:03 / bronto
In an ideal world, the user and group of the requesting client would determine the permissions of the data returned. We don't live in an ideal world. Two real-world problems intervene:
- You might not trust the root user of a client with root access to the server's files.
- The same username on client and server might have different numerical ID's
Problem 1 is conceptually simple. John Q. Programmer is given a test machine for which he has root access. In no way does that mean that John Q. Programmer should be able to alter root owned files on the server. Therefore NFS offers root squashing, a feature that maps uid 0 (root) to the anonymous (nfsnobody) uid, which defaults to -2 (65534 on 16 bit numbers). -- NFS: Overview and Gotchas -- Copyright (C) 2003 by Steve Litt
+(20171209) rpc.idmapd: nss_getpwnam: name '0' domain 'localdomain': resulting localname '(null)'
According to Steve Dickson in a comment (2011-08-12 16:01:55 EDT) to a Red Hat Bugzilla – Bug 715430 report
The [error] statement explains the problem. DNS on the local machine was not set up (or returning NULL) and the Domain= variable in /etc/idmapd.conf was not set.
nss_getpwnam: name '0' does not map into domain
On the Debian Mailing Lists, in an e-mail correspondence between Jonas Meurer and Christian Seiler (20150722) concerning "Kerberos-secured NFSv4" the error is explained in detail. My summary of the discussion:
When the NFS client sends nss_getpwnam: name '8' domain 'freesources.org': resulting localname '(null)'
The NFS client sends just the uid converted to a string in some cases instead of the properly translated NFS username, which the server then rejects.
The client should send nss_getpwnam: name '[email protected]' domain 'freesources.org': resulting localname 'mail'
Here you can see that the owner name that was transmitted by the NFS client was '[email protected]' (and not simply '8'), so that does contain an @; nss_getpwname can see that the domain name matches and just strips it, resulting in a user name 'mail', which it looks up in /etc/passwd, returns the user id (in this case, 8, because it's the same on client and server) and the server is perfectly happy.
So why does the client send the wrong username? ... every once in a while, idmapping will fail, so the kernel will just send a number. But that number will cause the chown command to fail, since the server won't translate it back.
Short answer: I have no idea.
Longer answer: ...
If I understand the longer answer correctly, the problem could occur because the NFS client relies on the "kernel's key cache". For the NFS server this should never be a problem because the "kernel's key cache" is never used.
Nonetheless,
Since you are using just regular nsswitch via /etc/passwd, nss_getpwnam should never fail in your case, unless you do some weird stuff with /etc/passwd at the same time.
The answer also refers to an alternative method to idmapd; nfsidmap, although reading the man I cannot quite understand how it would replace idmapd.
+(20171209) nss_getpwnam: name '[email protected]' does not map into domain 'localdomain'
This error message does not seem to occur for me, I am however including the answer from SUSE's support knowledgebase -- 10-DEC-13 Modified Date: 12-OCT-17 -- because of the description of cause, and the proposed remedy which stands in contrast to the other found discussions.
NFSv4 handles user identities differently than NFSv3. In v3, an nfs client would simply pass a UID number in chown (and other requests) and the nfs server would accept that (even if the nfs server did not know of an account with that UID number). However, v4 was designed to pass identities in the form of @. To function correctly, that normally requires idmapd (id mapping daemon) to be active at client and server, and for each to consider themselves part of the same id mapping domain.
Chown failures or idmapd errors like the ones documented above are typically a result of either:
- The username is known to the client but not known to the server, or
- The idmapd domain name is set differently on the client than it is on the server.
Therefore, this issue can be fixed by insuring that the nfs server and client are configured with the same idmapd domain name (/etc/idmapd.conf) and both have knowledge of the usernames / accounts in question.
However, it is often not convenient to insure that both sides have the same user account knowledge, especially if the nfs server is a filer. The NFS community has recognized that this idmapd feature of NFSv4 is often more troublesome that it is worth, so there are steps and modifications coming into effect to allow the NFSv3 behavior to work even under NFSv4.
The proposed remedy is to disable idmapd.
nfs.nfs4_disable_idmapping=1
+(20171209) Wireshark
Analyzing the Wireshark log, it is quite extensive but begins with something like:
[IP CLIENT] -> [IP SERVER] NFS 226 V4 Call ACCESS FH: [HEX VALUE], [Check: RD LU MD XT DL]
[IP SERVER] -> [IP CLIENT] NFS 238 V4 Reply (Call In 34) ACCESS, [Allowed: RD LU MD XT DL]
[IP CLIENT] -> [IP SERVER] NFS 246 V4 Call LOOKUP DH: [HEX VALUE]/archlinux
where a similar pattern [A HEX VALUE]/[PATH] can be discerned for
/sbin, /usr, /bin, /init, /lib, /systemd, /dev, /proc, /sys, /run, /, /lib64.
When the CLIENT requests /Id-linux-x86-64.so.2 the first errors start to appear:
[IP CLIENT] -> [IP SERVER] NFS 342 V4 Call OPEN DH: [HEX VALUE]/Id-linux-x86-64.so.2
[SERVER IP] -> [CLIENT IP] NFS 166 V4 Reply (Call In 124) OPEN Status: NFS4ERR_SYMLINK
The pattern more or less repeats itself with more frequent errors, for example, LOOKUP Status; and OPEN Status: reporting NFS4ERR_NOENT.
Interestingly, it is at the very end of the log where to first and only reference to user permission is made,
[SERVER IP] -> [CLIENT IP] NFS 182 V4 Reply (Call In 9562) SETATTR Status: NFS4ERR_BADOWNER
RFC
According to
- RFC7530 (Network File System (NFS) Version 4 Protocol, 201503, PROPOSED STANDARD) -- Updated by RFC7931
- RFC5661 (Network File System (NFS) Version 4 Minor Version 1 Protocol, 201001, PROPOSED STANDARD) -- Updated by RFC8178
- RFC7862 (Network File System (NFS) Version 4 Minor Version 2 Protocol, 201001, PROPOSED STANDARD) -- Updated by RFC8178 -- which refers back to [RFC5661].
NFS4ERR_BADOWNER (Error Code 10039)
This error is returned when an owner or owner_group attribute value or the who field of an ACE within an ACL attribute value cannot be translated to a local representation.
The specifications discuss in Section 5.9. Interpreting owner and owner_group, I am not sure what to cite as relevant however.
NFS4ERR_SYMLINK (Error Code 10029)
The current filehandle designates a symbolic link when the current operation does not allow a symbolic link as the target.
NFS4ERR_NOENT (Error Code 2)
This indicates no such file or directory. The file system object referenced by the name specified does not exist.
The error could however be expected ...
The current filehandle is assumed to refer to a regular directory a named attribute directory. LOOKUPP assigns the filehandle for its parent directory to be the current filehandle. If there is no parent directory, an NFS4ERR_NOENT error must be returned. Therefore, NFS4ERR_NOENT will be returned by the server when the current filehandle is at the root or top of the server's file tree.
+(20171210) mount -t nfs4 [SERVER IP]:/archlinux /mnt
On the client computer, using the Archlinux "LiveUSB" I was able to mount the network drive, download the latest kernel (4.14-4-1-ARCH) via the SERVER internet connection, and install archlinux on the [SERVER IP]/archlinux.
During install rpc.idmapd -fvvv indicated a successful mapping of usernames, for example,
rpc.idmapd: Server : (user) id "0" -> name "root@localdomain"
rpc.idmapd: Server : (group) id "99" -> name "nobody@localdomain"
... -> name "tty@localdomain"
... -> name "systemd-journal-upload@localdomain"
... -> name rpc@localdomain
... -> name systemd-journal@localdomain
... -> name utmp@localdomain
The result of genfstab was also different:
[SERVER IP]:/archlinux / nfs4 rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,times=600,retrans=2,sec=sys,clientaddr=[CLIENT IP],local_lock=none,addr=[SERVER IP] 0 0
Nevertheless, after reboot systemd failed again with the same failures as described at the beginning of the post.
+(20171210) Is the remote directory on the server mounted to /new_root?
The mkinitcpio script uses the variable mount_handler to carry an assigned "mounting function", in this case nfs_mount_handler(), to which the "root path" is passed $1 at a later stage; /new_root.
I am trying to verify that the client has mounted the [SERVER IP]:/archlinux to the /new_root. On the server, I can only observe that the client has established a connection but not if the directory is mounted and to where?
showmount -a server -> All mount points on server: (empty)
ss -ntp | grep 2049 ->
ESTAB 0 0 192.168.2.101:2049 192.168.2.102:809 (random port)
+(20171210) NFS4, sec=sys and id mapper are incompatible?
Reading the doco, it looks like sec=sys and the id mapper can be used to correctly map uid/gid to name where the client and server have different mappings in /etc/passwd and /etc/group. This simply isn't true.
That's because with sec=sys the id mapper doesn't come into play in the authentication part of the nfs protocol, only the file attributes part. With sec=sys authentication, nfs just passes the client uid/gid which is used directly by the server. So permissions checks will be screwed if client and server uid and gid don't align. To confuse things further, when the client creates a new file it is the authentication credentials that are used, so the file gets created at the server with the client's uid/gid. After that nfs uses idmap to get the file attributes, so the uid/gid (which originally came from the client) gets mapped at the server, and you end up seeing the server's name for a client uid/gid. Borkage! On the other hand, if the file was originally created at the server, you will see the correct name at the client, even if the uid/gid differs. But permissions checking will still be broken. -- kimmie -- Posted: Wed Feb 20, 2013 3:14 am Post subject: -- Emphasis in original
