pNFS seems a great method for multiple concurrent access to centralised shared storage, but it has a problem: the single NFS server providing NFS metadata (meta data server, MDS) is a single point of failure. If the MDS becomes inaccessible the shared storage will become inaccessible, before long.
To avoid this problem I am looking to set up a MDS cluster. How can this be achieved without denying clients direct I/O with the centralised storage?
There are solutions to HA-NFS, but they all break the direct I/O pNFS feature.
The long story
I am exploring (multiple) options to implement a shared "datastore" for an OpenStack environment.
I am currently looking at scenarios where storage is served as block storage from a centralised storage system. In these scenarios problems arise, because most filesystems cannot be mounted multiple times concurrently safely, at least not when there is concurrent read and write I/O from multiple hosts.
One possible approach to the multi-mount problem can be NFS 4.1 in combination with XFS. Starting with this version, there is pNFS, which allows NFS clients direct block-based I/O with the storage system. According to the kernel documentation, currently only XFS supports the nfs block layout.
With this approach the NFS server exports the share to clients as usual, but when a client actually does I/O, it directly interacts with the storage system, not the NFS server. Especially with the use case of virtual machines (which can be assumed to run only on one host at a time) this seems like a great fit.
The NFS server which exports the share to the clients is a single point of failure. If it goes down, no additional clients can mount the share and no unopened files can be accessed. The solution is a cluster of NFS servers where another server can take over if a server fails. How can pNFS servers be clustered?
There are quite a few tutorials on clustering NFS with the help of gluster or drbd. Solutions based on these helpers imply that data I/O is performed between NFS client and NFS server. Yet the intriguing feature of pNFS is that data I/O performed between NFS client and central storage system, which is not the NFS server. Therefore, using either of these approaches breaks the important feature of pNFS, thus cannot be considered a solution.
I would greatly appreciate existing work on this. But I also value comments on ...
my own theories
Even though pNFS has been around and stable for a couple of years, there is not a lot of noise around it. The little information I have tells me that:
most/all states lie with the client. This means that if a NFS server goes down, the states are not lost immediately.
as the client performs I/O directly with the storage system, no ongoing data flow is interrupted by the server going down.
With this information I'd guess that the following can be a viable approach:
- Create the shared LUN and use one server to put an XFS on it.
- Set up the kernel nfsd on every NFS server.
- Have the bunch of NFS servers all mount the same LUN read-only.
- Create a ramdisk on each server
- Use DRBD to set up a synchronous mirror accross all the ramdisks
- Use the synchronous mirrored ramdisk to store the NFS servers state by mounting it on /var/lib/nfs (and restart the nfs service)
keepalivedto organise the bunch of servers into an active/passive cluster with a VIP where only one server is active at a time.
keepalived's notify mechanism to have the active NFS server mount the LUN read-write and have the other servers mount the LUN read-only.
The load on the "primary" NFS server would be fairly low and directly correlate with the changes in virtual machines. Therefore having only one NFS server at a time doesn't seem to be a problem in small to medium sized virtualization environments.
With the way NFS 4.*, works the synchronously mirrored NFS server state should allow another server to take over after the primary server failed. When using gluster for the shared server state directory, I believe this should even allow for an active/active cluster, but I am unable to find any work on this.
Even if the mirrored server state does not work, the bump when another server takes over in an active/passive cluster shouldn't be that big, as I/O is performed directly with the central storage system. Correct?