I'm setting up a small Linux cluster for my laboratory. Current we have an login node for job submit and 3 computational node, and they are connected by Gigabit LAN (sadly no infiniband).
At first I simply used NFS to export shared storage (home folder) for cluster users. But soon we found out that NFS doesn't perform well under real-world payload.
It turned out that it's very common for my cluster users to read and write GBs of small picture files (one is <100KB) in shared storage. NFS performs very badly for small file IO.
I have a simple test of NFS/CIFS/SSHFS performance for small files writing. The times to untar 20000 small files (about 600MB total) on each filesystem are:
- CIFS: 25~30s
- SSHFS: 45~55s
- NFS(v3/v4): so slow that I didn't wait (estimation is 10mins)
CIFS/SSHFS performs very well comparing to NFS. But it's not a general practice to use CIFS/SSHFS for Linux home folders in a computer cluster scenario (and CIFS isn't a really posix-compliant filesystem AFAIK).
What's your opinion on a network filesystem choice for small files payload? Should I replace NFS with other cluster/distributed filesystems? Or is there a way to optimize NFS for small files?