2

I'm setting up a small Linux cluster for my laboratory. Current we have an login node for job submit and 3 computational node, and they are connected by Gigabit LAN (sadly no infiniband).

At first I simply used NFS to export shared storage (home folder) for cluster users. But soon we found out that NFS doesn't perform well under real-world payload.

It turned out that it's very common for my cluster users to read and write GBs of small picture files (one is <100KB) in shared storage. NFS performs very badly for small file IO.

I have a simple test of NFS/CIFS/SSHFS performance for small files writing. The times to untar 20000 small files (about 600MB total) on each filesystem are:

  • CIFS: 25~30s
  • SSHFS: 45~55s
  • NFS(v3/v4): so slow that I didn't wait (estimation is 10mins)

CIFS/SSHFS performs very well comparing to NFS. But it's not a general practice to use CIFS/SSHFS for Linux home folders in a computer cluster scenario (and CIFS isn't a really posix-compliant filesystem AFAIK).

What's your opinion on a network filesystem choice for small files payload? Should I replace NFS with other cluster/distributed filesystems? Or is there a way to optimize NFS for small files?

Improve this question
  • 1
    Don't use SSHFS if you wish performance in production. It is more like for corner cases. For performance, there are different filesystems such as ZFS, Gluster and so on (though little bit more complicated to set up). – Jakuje Apr 8 '17 at 13:24
  • I agree with @Jakuje, SSHFS must be a corner case which does not scale. I would setup a file server with FreeBSD+ZFS. – Rui F Ribeiro Apr 8 '17 at 13:32
  • @Ribeiro But how do you export the filesystem on file server to every cluster node? Local IO/filesystem isn't a major bottleneck as far as my oberservation. – cuihao Apr 8 '17 at 13:58
  • In fact I've tried glusterfs before. It's easy to setup and powerful. But we don't have a storage cluster (maybe I should ask for some new servers), so distributed filesystem is not suitable. Maybe I could use existing nodes to assemble a storage cluster, but I've heard that it's a bad idea to make a node both server and client of a distributed filesystem. – cuihao Apr 8 '17 at 14:05
  • No problem making a node both consumer and server of Gluster; we do it all the time for local testing of Gluster bricks. – Michael Mol Apr 8 '17 at 19:23
0

If you had a large-enough setup, I would have recommended IBM Spectrum Scale (formerly GPFS), but given the scale of your operations, I suggest you go with ZFS with a twist: Set atime to off. If you don't really care for recording access time on files, ZFS will perform quite well in your scenario.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.