My task is to store a list of JSONs on disk (without using any database) and I have these options:
Store them in a single, large file.
Store them in separate files, keyed by their IDs.
Personally I prefer the second option since it allows direct addressing any JSON by their ID without ever having to touch any other JSONs. However, there are almost 0.1 to 1 million JSON entries and I'm afraid of the possible negative consequences on the underlying filesystem (ext4 in my case):
Will this go over filesystem limits about the number of files (either in a directory or in a whole filesystem)?
Will this cause a slowdown while retrieving a specific ID?
To be more specific, I believe the list of files under a directory are maintained by the directory's inode structure, but I'm not sure what data structures (list or map) it uses to keep the file list. Is there any performance gain in the lookup if I use a hierarchy of directories? For example, put