I am trying to understand the differences between these three file system at a very basic level.
- Distributed FS: HDFS
- Parallel FS : Lustre
- Traditional FS : ext4/ext3/ NTFS/FAT etc.
I want to know what are the basic conceptual differences between these three file system. Most of my knowledge is of the traditional file systems , i.e. ext3/4 superblock , inode etc.
- If a MPI based process(np=8) tries to read a file or write a file A from file system, then how does the file access mechanism differ in these contexts
- also how is a file stored in this environment? i.e. File A will be split across multiple disks or file A will have redundant copies on storage. or a more simple scenario will be say multiple users opens a word document then saves it, then how does the write-back/synchronization differ in these 3 scenarios
So far I have formed a few concepts that:-
- In local file system , the storage is physically mounted on server/nodes.
- In parallel file system , a disk is shared (mount) on multiple nodes, and,
- In distributed FS, the multiple nodes have multiple local storage but all of them are synchronized by some mechanism.
If I have A,B are a workstation and C,D is the disk:
- If C is physically mounted on A & formatted as ext4 then it is traditional file system.
- If C is physically mounted on storage server Z + C is network mounted (NFS) on both A & B then this is cluster FS.
- If C is physically mounted on A and network mounted on B, D is physically on B and network mounted on A. Then this gives rise to Distributed FS.
Though some answers state that metadata and data are on separate servers in parallel file systems , but here too I wish to understand how metadata is managed in Distributed File Systems?