2

My task is to store a list of JSONs on disk (without using any database) and I have these options:

  • Store them in a single, large file.

  • Store them in separate files, keyed by their IDs.

Personally I prefer the second option since it allows direct addressing any JSON by their ID without ever having to touch any other JSONs. However, there are almost 0.1 to 1 million JSON entries and I'm afraid of the possible negative consequences on the underlying filesystem (ext4 in my case):

  • Will this go over filesystem limits about the number of files (either in a directory or in a whole filesystem)?

  • Will this cause a slowdown while retrieving a specific ID?

    To be more specific, I believe the list of files under a directory are maintained by the directory's inode structure, but I'm not sure what data structures (list or map) it uses to keep the file list. Is there any performance gain in the lookup if I use a hierarchy of directories? For example, put 0123456789.json into root/01/0123456789.json instead of root/0123456789.json?

4

Having 1 million files in a single directory would slow things down, but so would parsing an aggregate JSON with 1 million entries. Your best bet is indeed to use hashed directories, but you probably want to go two levels deep rather than just one. Namely, put 0123456789.json in root/0/01/0123456789.json, and 987654321.json in root/9/98/987654321.json.

4
  • Could you please be more elaborate on the first statement: Having 1 million files in a single directory would slow things down. So why would this happen? Does it mean directory inodes are storing filelist in a plain array? – Cyker Nov 22 '16 at 23:07
  • If you want to understand the details consider reading about directory access on ext4. A comment on SO isn't going to enlighten you. – Satō Katsura Nov 22 '16 at 23:09
  • 1
    Maybe One billion files on Linux can shed some light on this. While you are only storing 1 million files, the more files in a directory (or per filesystem), the more work is to be done for the kernel and for userspace utilities too. You can test for yourself and try to store 1 million files in a directory and see how it goes. – ckujau Nov 22 '16 at 23:20
  • @ckujau One million files in a filesystem != one million files in the same directory. – Satō Katsura Nov 22 '16 at 23:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.