I have a disk with a lot of data on it and I want to write some code such that thousands of threads can operate on the data. Each thread would work on its own individual file. This will speed up processing. They will read some input files and write a few output files. Each thread will work independently, so I can spin up as much as I want.
However I am concerned about thousands upon thousands of these threads accessing the file system and I feel that there will be contention in the file access.
What methods are there to enable fast file access for many threads or processes accessing a file system?