2
  • With RHEL 8.8 currently, and RHEL 9.x, the latest NFS version is 4.2.
  • When NFS 4 was introduced, it did away with a few things in NFS3 one of which was multiple port numbers:
    • NFS4 mandates all traffic now exclusively TCP uses the single well known port 2049.
    • https://www.snia.org/sites/default/files/SNIA_An_Overview_of_NFSv4-3_0.pdf
    • you can find more mostly reputable articles stating the same thing.
    • I have confirmed this by having only TCP 2049 open in firewalld for NFS 4.1 in RHEL 7.9; it does not use port 111 or any other unless you change the default configurations of /etc/nfs.conf or /etc/sysconfig/nfs. And in fact when I did get rdma working (over port 20049) that the rdma protocol specifically bypasses firewalld, an inherent aspect of why rdma saves cpy cycles and is faster i suppose.

The NFS insecure option in /etc/exports sets the server to listen to a request from any port on the client. Changing it to 'secure' (default) makes sure that the server will listen to only requests originating from ports 1-1024 of the client. Thus an unauthorized user on the client is kept from starting an NFS dialogue. For reference : https://security.stackexchange.com/questions/246527/what-is-insecure-about-the-insecure-option-of-nfs-exports

The default is secure vs insecure when doing an NFS4 export if neither is mentioned in /etc/exports.

With security rules it is oftentimes stated The NFS server must not have the insecure file locking option enabled..

First with the /etc/exports secure option in play, be default, the *will only operate on secure ports less than 1024` seems to be completely not true since NFS4 runs on port 2049. The number 2049 is greater than 1024... what am I missing?

With regards to RDMA which by convention happens on port 20049 there seems to be a little missed fact that one needs to explicitly state the insecure option in /etc/exports if a mount -o rdma is to be used otherwise the mount always happens as proto=tcp and not proto=rdma with no indication why.

I did validate that, using MLNX_OFED_LINUX-23.04-1.1.3.0-rhel8.8-x86_64.iso installed in place of the Redhat InfiniBand Support packages that a mount -o rdma,port=1023 does work with a mount on the client side showing proto=rdma.

However one must also do (with MLNX only?) an echo rdma 20049 > /proc/sys/nfsd/portlist. Or in the case with secure export an echo rdma 1023. Does anyone know how/why these values are not in /proc/sys/nfsd/portlist in the first place and why I must do them manually ? And then what is the correct way to put those numbers there, so that after boot my /etc/fstab nfs mounting of my data folder as rdma happens successfully? The MLNX instructional pdf falls short.

I have been banging my head against the wall getting RDMA to work, there seems to be a lot of shortcoming with NFS overall, and I have a paid for cluster mgr software that has RDMA placeholders for configuration, but all mounts are always proto=tcp. So if anyone can provide any information on anything described would be helpful, I will + any answer.

Also: I will end up doing /etc/exports with secure and choose some port number 1023 and below to satisfy security rules. How do I choose a proper number in that range? As ron nobody my understanding was I should never use port numbers below 1000 or 1024 for stuff I set up?

update: it appears that the /etc/exports parameter of secure or insecure is inconsequential. What matters is having rdma 20049 in /proc/fs/nfsd/portlist on the nfsserver. With that, or any number, it appears to work with the secure exportfs.

6
  • can someone post or link the official below 1024 port number list? internet assigned numbers authority (iana) ? I am thinking of using port 249 for RDMA
    – ron
    Jun 27, 2023 at 14:29
  • I think so, although it would be nicer if it was presented as a single list I can scroll down vs 145 individual pages
    – ron
    Jun 27, 2023 at 14:47
  • does anyone know if ports <= 1023 are all spoken for?
    – ron
    Jun 27, 2023 at 14:49
  • there seems to be an oversight regarding NFS_RDMA as 20049 and not allowing for NFS "secure" being below port 1024.
    – ron
    Jun 27, 2023 at 14:51

3 Answers 3

3

You have several questions and concerns regarding NFS (Network File System) version 4, its security options, port numbers, and the usage of RDMA (Remote Direct Memory Access).

NFS version 4, eliminated the need for multiple port numbers used in NFS version 3. Therefore, for NFSv4, all traffic is handled through TCP on port 2049.

Security and NFS

The insecure option in /etc/exports does not refer to the port number. It determines whether the NFS server will accept requests from any port on the client (insecure) or only from ports 1-1024 (secure). This option is unrelated to the well-known port 2049 used by NFSv4.

What is insecure about the "insecure" option of NFS exports?

The statement that the server will "only operate on secure ports less than 1024" is incorrect in the context of NFSv4, as it runs on port 2049, which is greater than 1024.

When using RDMA with NFS, you need to specify the insecure option in /etc/exports to allow RDMA mounts. Without this option, NFS will default to using TCP (proto=tcp) instead of RDMA (proto=rdma)

For choosing a port number below 1024 select a port within the range 1-1023 that is not already assigned to a well-known service.

You can refer to the IANA (Internet Assigned Numbers Authority) service name and port number registry to ensure you don't select a port already in use.

Service Name and Transport Protocol Port Number Registry

Check your /etc/services contains the mapping of service names to port numbers:

cat /etc/services

If you need a huge list of used numbers please install the nmap package.

3
  • I+'d your answer, however I still don't get your statement on how "insecure" option is incorrect in context of NFSv4 - you first say does not refer to the port number but then say It determines whether the NFS server will accept requests from any port.
    – ron
    Jun 27, 2023 at 16:46
  • 1
    @ron TCP connections use a port on the client and a separate port on the server. The "insecure"/"secure" option refers to the client-side port (i.e. whether or not to limit that to ports less than 1024, which require privileges on the client (at least on Linux)), not the server-side port which is 2049.
    – JoL
    Jun 28, 2023 at 1:02
  • 1
    As a bit of historical context: in early unix networks, individual users did not have administrative access to the network, but usually could bind to ports >=1024. That means that in such networks, connections from ports <1024 were guaranteed to come from non-user controlled system daemons. Nowadays, secure vs. insecure makes very little difference. Jun 28, 2023 at 4:20
2

regarding /proc/fs/nfsd/portlist which is largely the problem

https://patchwork.kernel.org/project/linux-nfs/patch/[email protected]/

the /etc/nfs.conf file in RHEL 8 interpreted as a template has #rdma=n in it. So one would logically think just uncomment and do rdma=y.... where y equals yes. This doesn't work completely.

You have to do rdma=nfsrdma instead of rdma=y. Then rdma 20049 will show up in /proc/fs/nfsd/portlist. Or I assume whatever rdma-port is specified as in /etc/nfs.conf. Then NFS RDMA works; this is with MLNX installed. It takes a minute after doing service nfs-server restart to observe the change in proc.

1
  • I have also observed rdma 20049 does not happen in /proc/fs/nfsd/portlist after boot. I have to manally do a service nfs-server restart and then it will be there. Seems like nfsv4beta.
    – ron
    Jun 27, 2023 at 20:30
0

On modern linux systems there are 2 issues that need to be addressed:

  • The rpcrdma kernel module must be loaded on both the server and the client
  • rdma associated with some port (doesn't matter which one) must appear in /proc/fs/nfsd/portlist

To accomplish the first of these, just add rpcrdma to /etc/modules-load.d/modules.conf

To accomplish the second, this argument needs to be on the /usr/sbin/rpc.nfsd command line: --rdma=2049. On Ubuntu, this is most easily accomplished by adding the following line to /etc/default/nfs-kernel-server:

RPCNFSDOPTS="--rdma=2049"

(I'm just using the normal NFSv4 port rather than 20049 to avoid potential firewall gotchas in the future.)

I should add for the sake of completeness that all remaining configuration is just the standard stuff that one does to set up an NFS server.

1
  • To mount on the client: mount -o rdma,port=2049 nfs-server:/share /mountpoint For some reason you have to specify the port when using rdma even if it's the standard one.
    – pgoetz
    Apr 22 at 19:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .