2

I have a directory (/srv/data) with a large number (20,000,000) with a number of small (4-40KB) files. These files are all located in subdirectories of /srv/data, where the concatenation of the subdirectory names and file names form UUIDs. For example, the UUID 833ac041-28c4-4ea4-9bac-81fe781732b4 would maps to the path /srv/data/83/3a/c0/41-28c4-4ea4-9bac-81fe781732b4. Using 2 hex digits for each subdirectory level gives a fan-out of 255. E.g., there are 255 directories in /srv/data, each with up to 255 subdirectories, and each of these with up to 255 subdirectories.

Is it possible to tell the kernel to give caching preferences to these files, or to a subset of these files?

E.g., "Hey Linux, cache as many files as you can from /srv/data. If you can't fit it all in memory, cache as much as you can and don't worry about any files outside of this directory. If you can fit it all in memory, then feel free to use whatever is left to cache files outside of this directory."

Improve this question
5
  • 2
    I think you're solving the wrong problem. Any directory with such a vast number of files is going to be slow. Has nothing to do with RAM speed vs. hard disk speed.
    – Wildcard
    Commented Nov 14, 2016 at 22:18
  • Some possible alternate avenues of attack, if the goal is performance: (1) Use a hierarchical directory structure rather than a flat structure; (2) Use a tmpfs to only store the files in RAM, if they are volatile, not-to-be-saved-permanently information; (3) Use a proper database such as Postgres.
    – Wildcard
    Commented Nov 14, 2016 at 22:30
  • Thank you. I have added additional information about the subdirectory hierarchy in place, which I forgot to mention. The original question was just trying to highlight that the "caching preferences" should be directed to the /srv/data if possible.
    – magnus
    Commented Nov 14, 2016 at 23:53
  • Thanks for the edit; it's a very interesting question actually. (I was even going to suggest ~256 subdirectories per level, so it looks like it's well designed.) Upvoted and hope you get a good answer.
    – Wildcard
    Commented Nov 15, 2016 at 0:02
  • Many thanks for the suggestions thus far. If worst comes to worst, I can design a component which manually caches as many files as possible, much like databases have their own page caches. However, there are cons associated with such an approach (duplicate caching, lack of optimisation according to disk block sizes, etc). I'd prefer if the OS can handle the file optimisation.
    – magnus
    Commented Nov 15, 2016 at 0:06

1 Answer 1

Reset to default
2

sysctl vm.vfs_cache_pressure=10 might help. To speed up working with thousands of small files, leave more dentries and inodes in the cache, not the file's contents. The biggest overhead is locating the small files on hdd, which takes much more read operation (more time) than reading some kilobytes from an exact location. When the kernel already knows the physical positions -stored in cache-, the file management will be noticeably faster.

Improve this answer
1
  • +1 but 10 is too low IMO as this affects all processes, which might not be desirable. Perhaps a sane compromise would be 50 (as 100 is the default).
    – heemayl
    Commented Nov 15, 2016 at 3:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .