Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 3 down vote favorite
1

I have a bunch of different binary files (total size of all files is circa ~500GB) stored in a hard drive that is mounted to machine001.

I have 200 other machines that need to access this data for read-only purposes.

What is the best way to share this data with all other computers, so that reads are fastest? (I use Ubuntu Linux)

Is there a different scheme that would be even better for fastest reads?

share|improve this question
What sort of data is it? You say it gets updated often, what updates it? – pjc50 Jun 13 '11 at 12:54

3 Answers

up vote 0 down vote accepted

Well, NFS uses UDP, and maybe you'ill need some extra coding to ensure that no files get lost. I don't like it and it's not fully supported on all OSes.

Maybe FTP is your answer for FAST reading, so FAST transferences since no software is faster than FTP to do so.

But you say that you have a total size of ~500GB. You can implement some VCS using checksums to know what files need to be updated, but it's more coding and there are lots of VCS's. However, I think FTP is the best way if you want portability, and maybe read access does not imply a copy of everything. Even this way, FTP is the fastest one.

So, if you want a VCS, to transfer just needed files, just fast. If you have Unix-like systems on all computers, you can install GIT, and use git pull to update the repos. Nobody will be able to write to your repo, and just needed files will be updated, fast.

share|improve this answer
thank you D4RIO. what do I need to do in order to setup FTP? do I need a server? do I need lots of RAM? – user3077 Apr 1 '11 at 17:41
You need a server, see proftpd.org – D4RIO Apr 4 '11 at 18:57
FTP fastest? On what benchmarks? For random-access data? I've downvoted this answer because I think this is not correct and imposes an extra level of inconvenience. – pjc50 Jun 13 '11 at 12:51
up vote 1 down vote

NFS is the obvious answer.

If the clients have space for caches, and the data updates infrequently, you could do various things such as storing the data in a version control system or mirroring it with rsync.

share|improve this answer
Isn't AFS supposed to be better than NFS? – Faheem Mitha Apr 1 '11 at 16:27
the data does update frequently. is NFS the fastest way to go? is there a better way? Will storing the data in a central DB lead to faster reads? or would it be better to try and 'cache' as much as possible on every machine, and each machine will use its 'own' version of the data? – user3077 Apr 1 '11 at 16:30
Possibly AFS is better, but NFS is simpler, older and more widely used. I don't have any benchmarks.<br> Database: this depends what your data is and how it's indexed, and what your access patterns are. – pjc50 Jun 13 '11 at 12:52
up vote 0 down vote

The ultimate way to make reads fasts would be to copy all the data to all the clients once and for all, but I guess you don't want to spend the space.

Are people likely to access the same file more than once from a given client? If so, I'd go for a simple approach with aggressive caching. Something like a WebDAV or FTP server, Fusedav or CurlFTPfs on the clients, and a caching HTTP proxy on each client (Squid or wwwoffle).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.