2

I'm setting up a small Linux cluster for my laboratory. Current we have an login node for job submit and 3 computational node, and they are connected by Gigabit LAN (sadly no infiniband).

At first I simply used NFS to export shared storage (home folder) for cluster users. But soon we found out that NFS doesn't perform well under real-world payload.

It turned out that it's very common for my cluster users to read and write GBs of small picture files (one is <100KB) in shared storage. NFS performs very badly for small file IO.

I have a simple test of NFS/CIFS/SSHFS performance for small files writing. The times to untar 20000 small files (about 600MB total) on each filesystem are:

  • CIFS: 25~30s
  • SSHFS: 45~55s
  • NFS(v3/v4): so slow that I didn't wait (estimation is 10mins)

CIFS/SSHFS performs very well comparing to NFS. But it's not a general practice to use CIFS/SSHFS for Linux home folders in a computer cluster scenario (and CIFS isn't a really posix-compliant filesystem AFAIK).

What's your opinion on a network filesystem choice for small files payload? Should I replace NFS with other cluster/distributed filesystems? Or is there a way to optimize NFS for small files?

5
  • 1
    Don't use SSHFS if you wish performance in production. It is more like for corner cases. For performance, there are different filesystems such as ZFS, Gluster and so on (though little bit more complicated to set up).
    – Jakuje
    Apr 8, 2017 at 13:24
  • I agree with @Jakuje, SSHFS must be a corner case which does not scale. I would setup a file server with FreeBSD+ZFS. Apr 8, 2017 at 13:32
  • @Ribeiro But how do you export the filesystem on file server to every cluster node? Local IO/filesystem isn't a major bottleneck as far as my oberservation.
    – cuihao
    Apr 8, 2017 at 13:58
  • In fact I've tried glusterfs before. It's easy to setup and powerful. But we don't have a storage cluster (maybe I should ask for some new servers), so distributed filesystem is not suitable. Maybe I could use existing nodes to assemble a storage cluster, but I've heard that it's a bad idea to make a node both server and client of a distributed filesystem.
    – cuihao
    Apr 8, 2017 at 14:05
  • No problem making a node both consumer and server of Gluster; we do it all the time for local testing of Gluster bricks. Apr 8, 2017 at 19:23

1 Answer 1

0

If you had a large-enough setup, I would have recommended IBM Spectrum Scale (formerly GPFS), but given the scale of your operations, I suggest you go with ZFS with a twist: Set atime to off. If you don't really care for recording access time on files, ZFS will perform quite well in your scenario.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .