We would like to store millions of text files in a Linux filesystem, with the purpose of being able to zip up and serve an arbitrary collection as a service. We've tried other solutions, like a key/value database, but our requirements for concurrency and parallelism make using the native filesystem the best choice.
The most straightforward way is to store all files in a folder:
$ ls text_files/ 1.txt 2.txt 3.txt
which should be possible on an EXT4 file system, which has no limit to number of files in a folder.
The two FS processes will be:
- Write text file from web scrape (shouldn't be affected by number of files in folder).
- Zip selected files, given by list of filenames.
My question is, will storing up to ten million files in a folder affect the performance of the above operations, or general system performance, any differently than making a tree of subfolders for the files to live in?