1

I'm currently helping setting up a lab that will use diskless nodes for some MPI and CUDA computing.

The distribution of choice is CentOS 7. To set up the diskless nodes I've followed the guide here.

I got to boot a diskless node successfully and even run some MPI test programs. So everything works fine in terms of connectivity,firewalls,nfs exports etc.

The problem is that after ~12 hours of having booted the diskless node, the main server which acts as a dhcp,tftp and nfs server seems to kick of the diskless node from the nfs service which results with the kernel: nfs: server <servername> not responding, still trying message appearing on the client. At that point I also stop getting ping replies from the diskless clients. Since the client has its root fs obtained by NFS I guess this leaves the client at a "corrupted" state only allowing me to reboot with Ctrl+Alt+Del or the machine's reset switch. No matter how much times passes the client won't connect back. Inspecting /var/log/messages on the main server I got this interesting in my opinion line: Oct 8 23:30:50 myhostname kernel: NFSD: purging unused client (clientid e87d62f6).

Here is a bigger part of the log: Oct 8 23:30:17 myhostname kernel: nfsv4 compound op ffff885c713d4080 opcnt 4 #3: 3: status 0 Oct 8 23:30:17 myhostname kernel: nfsv4 compound op #4/4: 9 (OP_GETATTR) Oct 8 23:30:17 myhostname kernel: nfsd: fh_verify(36: 01070001 00260308 00000000 996a1153 334c49c8 b8768c81) Oct 8 23:30:17 myhostname kernel: nfsv4 compound op ffff885c713d4080 opcnt 4 #4: 9: status 0 Oct 8 23:30:17 myhostname kernel: nfsv4 compound returned 0 Oct 8 23:30:17 myhostname kernel: --> nfsd4_store_cache_entry slot ffff885c72a66000 Oct 8 23:30:17 myhostname kernel: renewing client (clientid 5bbb153f/e87d62f7) Oct 8 23:30:50 myhostname kernel: NFSD: laundromat service - starting Oct 8 23:30:50 myhostname kernel: NFSD: purging unused client (clientid e87d62f6) Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: cmd: remove Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: arg: 4c696e7578204e465376342e31206e6f64653033 Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: env0: (null) Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: env1: (null) Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: /sbin/nfsdcltrack return value: 0 Oct 8 23:30:50 myhostname kernel: NFSD: laundromat_main - sleeping for 57 seconds Oct 8 23:31:48 myhostname kernel: NFSD: laundromat service - starting Oct 8 23:31:48 myhostname kernel: NFSD: purging unused client (clientid e87d62f7) Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: cmd: remove Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: arg: 4c696e7578204e465376342e31206e76696469613031 Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: env0: (null) Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: env1: (null) Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: cmd: remove Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: arg: 4c696e7578204e465376342e31206e6f64653033 Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: env0: (null) Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: env1: (null) Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: /sbin/nfsdcltrack return value: 0 Oct 8 23:30:50 myhostname kernel: NFSD: laundromat_main - sleeping for 57 seconds Oct 8 23:31:48 myhostname kernel: NFSD: laundromat service - starting Oct 8 23:31:48 myhostname kernel: NFSD: purging unused client (clientid e87d62f7) Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: cmd: remove Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: arg: 4c696e7578204e465376342e31206e76696469613031 Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: env0: (null) Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: env1: (null) Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: /sbin/nfsdcltrack return value: 0 Oct 8 23:31:48 myhostname kernel: NFSD: laundromat_main - sleeping for 90 seconds Oct 8 23:33:18 myhostname kernel: NFSD: laundromat service - starting Oct 8 23:33:18 myhostname kernel: NFSD: laundromat_main - sleeping for 90 seconds Oct 8 23:34:48 myhostname kernel: NFSD: laundromat service - starting Oct 8 23:34:48 myhostname kernel: NFSD: laundromat_main - sleeping for 90 seconds

Afterwards it just continues looping the laundromat service starting/sleeping message forever.

nfsstat does not reveal anything weird on the server like badcalls etc. I've also tried to force use NFSv3 version. I got the same problem, however the purging unused client and laundromat message does not appear in the logs now (guessing it was added in v4?).

Now onto some details regarding how the connectivity is. The main server features 2 network interfaces. One realtek one (which works by default with the kernel drivers) and one which is nvidia nforce and needs kmod-forcedeth from elrepo. All server services are on the nvidia-nforce card. The diskless node and the server connect via a gigabit switch (can't remember brand name/model sorry).

1 Answer 1

0

Did you solve this problem ? I get same problem with Centos7 diskless clients -- all fine for 12+ hours then all clients dropped more or less at the same time.

I found that by adding the following to the relevant part of my dhcpd.conf file the problem goes away :

default-lease-time infinite;
max-lease-time infinite;

I ran into this problem after migrating from centos6 to centos7, I suppose because the dhcpd default ip leasetime had changed. You may want to choose different values. dhcpd man pages covers it too.

2
  • This is not an answer to the question. And it is about 4 years late anyways :-)
    – stoney
    Jan 15, 2023 at 9:16
  • This does not really answer the question. If you have a different question, you can ask it by clicking Ask Question. To get notified when this question gets new answers, you can follow this question. Once you have enough reputation, you can also add a bounty to draw more attention to this question. - From Review
    – doneal24
    Jan 15, 2023 at 15:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .