1

I'm currently helping setting up a lab that will use diskless nodes for some MPI and CUDA computing.

The distribution of choice is CentOS 7. To set up the diskless nodes I've followed the guide here.

I got to boot a diskless node successfully and even run some MPI test programs. So everything works fine in terms of connectivity,firewalls,nfs exports etc.

The problem is that after ~12 hours of having booted the diskless node, the main server which acts as a dhcp,tftp and nfs server seems to kick of the diskless node from the nfs service which results with the kernel: nfs: server <servername> not responding, still trying message appearing on the client. At that point I also stop getting ping replies from the diskless clients. Since the client has its root fs obtained by NFS I guess this leaves the client at a "corrupted" state only allowing me to reboot with Ctrl+Alt+Del or the machine's reset switch. No matter how much times passes the client won't connect back. Inspecting /var/log/messages on the main server I got this interesting in my opinion line: Oct 8 23:30:50 myhostname kernel: NFSD: purging unused client (clientid e87d62f6).

Here is a bigger part of the log: Oct 8 23:30:17 myhostname kernel: nfsv4 compound op ffff885c713d4080 opcnt 4 #3: 3: status 0 Oct 8 23:30:17 myhostname kernel: nfsv4 compound op #4/4: 9 (OP_GETATTR) Oct 8 23:30:17 myhostname kernel: nfsd: fh_verify(36: 01070001 00260308 00000000 996a1153 334c49c8 b8768c81) Oct 8 23:30:17 myhostname kernel: nfsv4 compound op ffff885c713d4080 opcnt 4 #4: 9: status 0 Oct 8 23:30:17 myhostname kernel: nfsv4 compound returned 0 Oct 8 23:30:17 myhostname kernel: --> nfsd4_store_cache_entry slot ffff885c72a66000 Oct 8 23:30:17 myhostname kernel: renewing client (clientid 5bbb153f/e87d62f7) Oct 8 23:30:50 myhostname kernel: NFSD: laundromat service - starting Oct 8 23:30:50 myhostname kernel: NFSD: purging unused client (clientid e87d62f6) Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: cmd: remove Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: arg: 4c696e7578204e465376342e31206e6f64653033 Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: env0: (null) Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: env1: (null) Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: /sbin/nfsdcltrack return value: 0 Oct 8 23:30:50 myhostname kernel: NFSD: laundromat_main - sleeping for 57 seconds Oct 8 23:31:48 myhostname kernel: NFSD: laundromat service - starting Oct 8 23:31:48 myhostname kernel: NFSD: purging unused client (clientid e87d62f7) Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: cmd: remove Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: arg: 4c696e7578204e465376342e31206e76696469613031 Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: env0: (null) Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: env1: (null) Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: cmd: remove Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: arg: 4c696e7578204e465376342e31206e6f64653033 Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: env0: (null) Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: env1: (null) Oct 8 23:30:50 myhostname kernel: nfsd4_umh_cltrack_upcall: /sbin/nfsdcltrack return value: 0 Oct 8 23:30:50 myhostname kernel: NFSD: laundromat_main - sleeping for 57 seconds Oct 8 23:31:48 myhostname kernel: NFSD: laundromat service - starting Oct 8 23:31:48 myhostname kernel: NFSD: purging unused client (clientid e87d62f7) Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: cmd: remove Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: arg: 4c696e7578204e465376342e31206e76696469613031 Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: env0: (null) Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: env1: (null) Oct 8 23:31:48 myhostname kernel: nfsd4_umh_cltrack_upcall: /sbin/nfsdcltrack return value: 0 Oct 8 23:31:48 myhostname kernel: NFSD: laundromat_main - sleeping for 90 seconds Oct 8 23:33:18 myhostname kernel: NFSD: laundromat service - starting Oct 8 23:33:18 myhostname kernel: NFSD: laundromat_main - sleeping for 90 seconds Oct 8 23:34:48 myhostname kernel: NFSD: laundromat service - starting Oct 8 23:34:48 myhostname kernel: NFSD: laundromat_main - sleeping for 90 seconds

Afterwards it just continues looping the laundromat service starting/sleeping message forever.

nfsstat does not reveal anything weird on the server like badcalls etc. I've also tried to force use NFSv3 version. I got the same problem, however the purging unused client and laundromat message does not appear in the logs now (guessing it was added in v4?).

Now onto some details regarding how the connectivity is. The main server features 2 network interfaces. One realtek one (which works by default with the kernel drivers) and one which is nvidia nforce and needs kmod-forcedeth from elrepo. All server services are on the nvidia-nforce card. The diskless node and the server connect via a gigabit switch (can't remember brand name/model sorry).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.