I have a disk with a lot of data on it and I want to write some code such that thousands of threads can operate on the data. Each thread would work on its own individual file. This will speed up processing. They will read some input files and write a few output files. Each thread will work independently, so I can spin up as much as I want.

However I am concerned about thousands upon thousands of these threads accessing the file system and I feel that there will be contention in the file access.

What methods are there to enable fast file access for many threads or processes accessing a file system?

  • RAM disks, and weeks (or months) of debugging your code. – Satō Katsura Jul 1 '16 at 5:34
  • 1
    How do you plan to make so many processes all work on the same file? Do you want them to do the same thing repeatedly or what? Also, what is the point in making thousands of threads? The biggest machine I've seen has 200 cores and if your project is well organized, creating more threads than cores will unlikely show any improvement (on the contrary). – Julie Pelletier Jul 1 '16 at 5:42
  • Clarification - Each thread is working on a SEPARATE file - There are thousands of them. I can have one script work on all the files but I would rather have thousands of threads work on their own separate file. I would just spin them up (any number of ways - this part I know). – Xofo Jul 1 '16 at 5:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.