1

Hi my use case is quite specific.

I have 20 Windows 7 machines constantly creating files in my storage; around 98% of these files are 2.1 MB. On average we create 24 million files every 3 days, and this number may increase in the near future as we may need to add new clients to our system.

I do not modify files (just create, read, copy and delete). I have seen Reiser4, which looks promising, but I also would like to have the capability to replicate the files across multiple storage nodes across the network, so I can have a fault tolerance system in place.

Any suggestion?

closed as off-topic by Anthon, dr01, Archemar, cas, Jenny D Mar 29 '16 at 10:11

  • This question does not appear to be about Unix or Linux within the scope defined in the help center.
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    So the small files are the remaining 2%. Why would you want Reiser4 for that percentage of your files? Isn't the performance on the remaining 98% more important? – Anthon Mar 29 '16 at 4:33
  • 3
    I'm voting to close this question as off-topic because it is about Windows. – dr01 Mar 29 '16 at 7:24
  • @dr01 this question is not about Windows, the server we are running is Linux CentOS 6.5 hence the filesystem is going to be installed on linux, that is why I put this question here – Manuel Sopena Ballesteros Mar 29 '16 at 11:21
  • @Anthon 2.1MB files are small and they don't perform same as 2GB files. I can prove this because we are using Panasas (PanFS) as file system and we get bandwidth of 400 MBs on big files (2Gigs) and 10 MBs on 2.1MB files. – Manuel Sopena Ballesteros Mar 29 '16 at 11:23
  • @dr01 The reason I mentioned that our clients are Windows 7 is to give you more details so you can think that I am using CIFS as protocol to share files from the clients. So this is still a Linux question as my problem is the solution I need to install on my CentOS server. – Manuel Sopena Ballesteros Mar 29 '16 at 11:28
2

Check out GlusterFS. I had similar needs a few months ago and tested it on VM and physical nodes. Replication is quick and setup isn't that difficult.

One caveat. If you do go with this, you need a dedicated machine to act as the file server. That machine will then handle replication between the slave nodes. I tried editing a file on one of the end nodes - i.e. to see if multiple nodes could act as the file server - to see if replication would sync the changes. It didn't; but when I edited the file via the master/controller the changes were synced.

Not the answer you're looking for? Browse other questions tagged or ask your own question.