Force directory to always be in cache

Question

I've been testing out different methods to improve the time it takes to compile my entire c++ project. Currently it takes ~5 minutes. I experimented with distcc, ccache, and others. Recently, I discovered that if I copy my entire project onto a RAM-drive, and then compile from there, it cuts the compile time down to 30% of its original-- just 1.5 minutes.

Obviously, working from the RAM drive isn't practical. So, does anyone know of a way I can force the OS to always keep a certain directory cached? I still want the directory to get synced back to disk like normal, but I always want a copy of the data in memory as well. Is this possible?

EDIT: As a possible solution, we just thought of launching a daemon that runs rsync every 10 seconds or so to sync the disk drive with a RAM drive. Then we run the compilation from the RAM drive. The rsync is blazing fast, but would this really work? Surely the OS could do better....

Answer 1

The obvious way to keep a bunch of files in the cache is to access them often. Linux is pretty good at arbitrating between swapping and caching, so I suspect that the speed difference you observe is actually not due to the OS not keeping things in the cache, but to some other difference between your usage of tmpfs and your other attempts.

Try observing what is doing IO in each case. The basic tool for that is iotop. Other tools may be useful; see Linux disk IO load breakdown, by filesystem path and/or process?, What program in Linux can measure I/O over time?, and other threads at Server Fault.

Here are a few hypotheses as to what could be happening. If you take measurements, please show them so that we can confirm or disprove these hypotheses.

• If you have file access times turned on, the OS may waste quite a bit of time writing these access times. Access times are useless for a compilation tree, so make sure they're turned off with the noatime mount option. Your tmpfs+rsync solution never reads from the hard disk, so it never has to spend extra time writing atimes.
• If the writes are synchronizing, either because the compiler calls sync() or because the kernel frequently flushes its output buffers, the writes will take longer to a hard disk than to tmpfs.

Answer 2

Linux by default use the RAM as disk cache. As a demonstration, try to run time find /some/dir/containing/a/lot/of/files > /dev/null two times, the second time is a lot faster as every disk inodes are cached. The point here is how to make use of this kernel feature and stop your attempt to replace it.

The point is to change the swappiness. Let's consider three main types of memory use: active programs, inactive programs and disk cache. Obviously memory used by active programs should not be swapped out and the choice between to two others is quite arbitrary. Would you like fast program switching or fast file access? A low swappiness prefers to keep programs in memory (even if not used for long time) and a high swappiness prefers to keep more disk cache (by swapping unused programs). (swappiness scale is from 0 to 100 and the default value is 60)

My solution to your problem is to change the swappiness to very high (90-95 not to say 100) and to load the cache:

echo 95 | sudo tee /proc/sys/vm/swappiness > /dev/null # once after reboot
find /your/source/directory -type f -exec cat {} \; > /dev/null

As you guess it, you must have enough free memory to hold in cache all your source files and object files as well as the compiler, included headers files, linked libraries, your IDE and other used programs.

Answer 3

This thing seems to work for me if I want to keep certain files or all files in a certain directory in cache.

vmtouch seems to do just the thing. Example 5 there might be what you need.

vmtouch -dl /whatever/directory/

I needed to run it as root with sudo

Answer 4

Forcing cache isn't the right way to do this. Better to keep sources on hard drive and compile them on tmpfs. Many build systems, such as qmake and CMake, supports out-of-source builds.

Answer 5

The inosync daemon sounds like it does exactly what you want if you're going to rsync to a ramdisk. Instead of rsyncing every 10 seconds or so, it uses Linux's inotify facility to rsync when a file changes. I found it in the Debian repository as the inosync package, or its source is available at http://bb.xnull.de/projects/inosync/.

Answer 6

Given sufficient memory your build out of the ramdisk does no I/O. This can speed up anything that reads or writes files. I/O is one of the slowest operations. Even if you get everything cached before the build you still have the I/Os for write, although they should have minimal impact.

You may get some speedup by pre-loading all the files into cache, but the time taken to to that should be included in the total build times. This may not give you much advantage.

Building the object and intermediate files into RAM rather than disk. Doing incremental builds may get you significant gains on frequent builds. On most projects I do a daily clean build and incremental builds in between. Integration builds are always clean builds, but I try to limit them to less than one per day.

You may gain some performance by using an ext2 partition with atime turned off. Your source should be in version control on a journaled file system like ext3/4.

Answer 7

As stated previously, the obvious way is to read all the directory structure and file contents of what you wish to be cached.

You can automate this by writing a script to monitor the output of vmstat 1 (use whatever equivalent tool for your OS) and keep a sum of the number of blocks written and read. Once the sum passes a threshold of your choosing, read all the files you intend to cache, reset the sum, then continue monitoring vmstat output. For quickly reading files: if your tree contains many files, avoid find ... -exec cat, instead try find ... -print0 | xargs -0 cat or a custom program that won't execute cat for each file.

Monitoring disk IO is preferable to using a fixed interval because it signals to reread your data more or less frequently depending on disk IO load.

I've used this automated method successfully on systems where I needed some index file reads to always be quick, avoiding hard drive I/O. I've also used strace to make a list of every file that gets accessed when I log in so I can keep everything hot in cache for fast logins.

This may not be the best possible solution but it suited me well.

Answer 8

This script will keep directories cached.

The script loops through each directory, reads part of the directory, then goes to sleep for 5 seconds. The script will run in the background to keep each directory cached.

echo $smallnumber > /proc/sys/vm/vfs_cache_pressure is how Linux automatically tunes the directory cache. It works with this script to tell Linux which directories have been used recently.