0

I use an external nfs server to mount a home directory into a kubernetes container. Sporadically when deleting the container, the nfs connection gets stuck in "waiting to close".

The fallout of this, is that the node, where the container lived, develops a huge iowait time value. Then all containers suffer nfs disk performance until this node is rebooted.

I was able to manually umount the offending mount however the iowait didn't come down.

I installed iotop to help diagnose the issue, however it errors out with a python error.

Two questions:

  1. Is there a way to fix the condition without a reboot, it affects so many customers that it is no idea?

  2. Thoughts as why it is happening in the first place and how to solve it. The nfs server is configured for sync. Would moving to async help this issue?

I don't believe that this is an kubernetes issue but the one downside to kubernetes/nfs is that you can't configure the client connection in anyway.

1 Answer 1

-1

To address NFS mount issues in Kubernetes, leading to high I/O wait times without rebooting, you can attempt a forceful (umount -f) or lazy (umount -l) unmount of the NFS mount points. Switching the NFS server's configuration from sync to async may improve performance, though it risks data integrity. Enhancing Persistent Volume (PV) definitions in Kubernetes with specific mount options, such as nfsvers=4.1, rsize=1048576, and wsize=1048576, can potentially alleviate performance issues. For example, in your PV configuration, include:

nfs:
  path: /path/to/dir
  server: nfs-server.example.com
  mountOptions:
    - hard
    - nfsvers=4.1
    - rsize=1048576
    - wsize=1048576
    - timeo=14

This approach, coupled with monitoring and possibly exploring alternative storage solutions like Ceph or GlusterFS, aims to mitigate NFS-related performance degradation in Kubernetes environments.

1
  • In my use case I am mounting it as a volume and not a pvc. You can't use mountOptions in this case. However you can use a global configuration file on the node "nfsmount.conf" to specify mount options for all containers. I will test out these values and report back.
    – Daxcor
    Mar 12 at 15:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .