2

I have a directory (/srv/data) with a large number (20,000,000) with a number of small (4-40KB) files. These files are all located in subdirectories of /srv/data, where the concatenation of the subdirectory names and file names form UUIDs. For example, the UUID 833ac041-28c4-4ea4-9bac-81fe781732b4 would maps to the path /srv/data/83/3a/c0/41-28c4-4ea4-9bac-81fe781732b4. Using 2 hex digits for each subdirectory level gives a fan-out of 255. E.g., there are 255 directories in /srv/data, each with up to 255 subdirectories, and each of these with up to 255 subdirectories.

Is it possible to tell the kernel to give caching preferences to these files, or to a subset of these files?

E.g., "Hey Linux, cache as many files as you can from /srv/data. If you can't fit it all in memory, cache as much as you can and don't worry about any files outside of this directory. If you can fit it all in memory, then feel free to use whatever is left to cache files outside of this directory."

  • 2
    I think you're solving the wrong problem. Any directory with such a vast number of files is going to be slow. Has nothing to do with RAM speed vs. hard disk speed. – Wildcard Nov 14 '16 at 22:18
  • Some possible alternate avenues of attack, if the goal is performance: (1) Use a hierarchical directory structure rather than a flat structure; (2) Use a tmpfs to only store the files in RAM, if they are volatile, not-to-be-saved-permanently information; (3) Use a proper database such as Postgres. – Wildcard Nov 14 '16 at 22:30
  • Thank you. I have added additional information about the subdirectory hierarchy in place, which I forgot to mention. The original question was just trying to highlight that the "caching preferences" should be directed to the /srv/data if possible. – magnus Nov 14 '16 at 23:53
  • Thanks for the edit; it's a very interesting question actually. (I was even going to suggest ~256 subdirectories per level, so it looks like it's well designed.) Upvoted and hope you get a good answer. – Wildcard Nov 15 '16 at 0:02
  • Many thanks for the suggestions thus far. If worst comes to worst, I can design a component which manually caches as many files as possible, much like databases have their own page caches. However, there are cons associated with such an approach (duplicate caching, lack of optimisation according to disk block sizes, etc). I'd prefer if the OS can handle the file optimisation. – magnus Nov 15 '16 at 0:06
3

sysctl vm.vfs_cache_pressure=10 might help. To speed up working with thousands of small files, leave more dentries and inodes in the cache, not the file's contents. The biggest overhead is locating the small files on hdd, which takes much more read operation (more time) than reading some kilobytes from an exact location. When the kernel already knows the physical positions -stored in cache-, the file management will be noticeably faster.

  • +1 but 10 is too low IMO as this affects all processes, which might not be desirable. Perhaps a sane compromise would be 50 (as 100 is the default). – heemayl Nov 15 '16 at 3:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.