2

My task is to store a list of JSONs on disk (without using any database) and I have these options:

  • Store them in a single, large file.

  • Store them in separate files, keyed by their IDs.

Personally I prefer the second option since it allows direct addressing any JSON by their ID without ever having to touch any other JSONs. However, there are almost 0.1 to 1 million JSON entries and I'm afraid of the possible negative consequences on the underlying filesystem (ext4 in my case):

  • Will this go over filesystem limits about the number of files (either in a directory or in a whole filesystem)?

  • Will this cause a slowdown while retrieving a specific ID?

    To be more specific, I believe the list of files under a directory are maintained by the directory's inode structure, but I'm not sure what data structures (list or map) it uses to keep the file list. Is there any performance gain in the lookup if I use a hierarchy of directories? For example, put 0123456789.json into root/01/0123456789.json instead of root/0123456789.json?

1 Answer 1

Reset to default
4

Having 1 million files in a single directory would slow things down, but so would parsing an aggregate JSON with 1 million entries. Your best bet is indeed to use hashed directories, but you probably want to go two levels deep rather than just one. Namely, put 0123456789.json in root/0/01/0123456789.json, and 987654321.json in root/9/98/987654321.json.

4
  • Could you please be more elaborate on the first statement: Having 1 million files in a single directory would slow things down. So why would this happen? Does it mean directory inodes are storing filelist in a plain array?
    – Cyker
    Nov 22, 2016 at 23:07
  • If you want to understand the details consider reading about directory access on ext4. A comment on SO isn't going to enlighten you.
    – Satō Katsura
    Nov 22, 2016 at 23:09
  • 1
    Maybe One billion files on Linux can shed some light on this. While you are only storing 1 million files, the more files in a directory (or per filesystem), the more work is to be done for the kernel and for userspace utilities too. You can test for yourself and try to store 1 million files in a directory and see how it goes.
    – ckujau
    Nov 22, 2016 at 23:20
  • @ckujau One million files in a filesystem != one million files in the same directory.
    – Satō Katsura
    Nov 22, 2016 at 23:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .