Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I've been testing out different methods to improve the time it takes to compile my entire c++ project. Currently it takes ~5 minutes. I experimented with distcc, ccache, and others. Recently, I discovered that if I copy my entire project onto a RAM-drive, and then compile from there, it cuts the compile time down to 30% of its original-- just 1.5 minutes.

Obviously, working from the RAM drive isn't practical. So, does anyone know of a way I can force the OS to always keep a certain directory cached? I still want the directory to get synced back to disk like normal, but I always want a copy of the data in memory as well. Is this possible?

EDIT: As a possible solution, we just thought of launching a daemon that runs rsync every 10 seconds or so to sync the disk drive with a RAM drive. Then we run the compilation from the RAM drive. The rsync is blazing fast, but would this really work? Surely the OS could do better....

share|improve this question
    
Cache isn't the only difference between tmpfs and ext3/4; they have journaling, for example, which will be written regardless of caching. –  André Paramés Jan 28 '11 at 16:02
    
Could you time your compilation and share the result with us? It would dispel some raising controversy. make clean && /usr/bin/time -v make (do not use the bash built in time command) –  shellholic Feb 15 '11 at 10:11
1  
@she Why not bash's built-in command? –  Tshepang Feb 15 '11 at 13:25
2  
@Tshepang the time built in bash (help time) has a lot fewer details (no verbose option) than the GNU time (man time) concerning the I/O, context switches,... –  shellholic Feb 15 '11 at 15:37

6 Answers 6

The obvious way to keep a bunch of files in the cache is to access them often. Linux is pretty good at arbitrating between swapping and caching, so I suspect that the speed difference you observe is actually not due to the OS not keeping things in the cache, but to some other difference between your usage of tmpfs and your other attempts.

Try observing what is doing IO in each case. The basic tool for that is iotop. Other tools may be useful; see Linux disk IO load breakdown, by filesystem path and/or process?, What program in Linux can measure I/O over time?, and other threads at Server Fault.

Here are a few hypotheses as to what could be happening. If you take measurements, please show them so that we can confirm or disprove these hypotheses.

  • If you have file access times turned on, the OS may waste quite a bit of time writing these access times. Access times are useless for a compilation tree, so make sure they're turned off with the noatime mount option. Your tmpfs+rsync solution never reads from the hard disk, so it never has to spend extra time writing atimes.
  • If the writes are synchronizing, either because the compiler calls sync() or because the kernel frequently flushes its output buffers, the writes will take longer to a hard disk than to tmpfs.
share|improve this answer
    
I have this feeling too. Compiling is CPU intensive, rather than IO. –  phunehehe Feb 15 '11 at 4:38
    
Hmmm, I'd like to see a comment from @JaredC here confirming or denying Gilles hypothesis. 1.5 vs. 5 minues is quite a big difference... –  Daniel Alder Mar 15 at 16:46

Forcing cache isn't the right way to do this. Better to keep sources on hard drive and compile them on tmpfs. Many build systems, such as qmake and CMake, supports out-of-source builds.

share|improve this answer

Linux by default use the RAM as disk cache. As a demonstration, try to run time find /some/dir/containing/a/lot/of/files > /dev/null two times, the second time is a lot faster as every disk inodes are cached. The point here is how to make use of this kernel feature and stop your attempt to replace it.

The point is to change the swappiness. Let's consider three main types of memory use: active programs, inactive programs and disk cache. Obviously memory used by active programs should not be swapped out and the choice between to two others is quite arbitrary. Would you like fast program switching or fast file access? A low swappiness prefers to keep programs in memory (even if not used for long time) and a high swappiness prefers to keep more disk cache (by swapping unused programs). (swappiness scale is from 0 to 100 and the default value is 60)

My solution to your problem is to change the swappiness to very high (90-95 not to say 100) and to load the cache:

echo 95 | sudo tee /proc/sys/vm/swappiness > /dev/null # once after reboot
find /your/source/directory -type f -exec cat {} \; > /dev/null

As you guess it, you must have enough free memory to hold in cache all your source files and object files as well as the compiler, included headers files, linked libraries, your IDE and other used programs.

share|improve this answer
    
This is useful in general, but what I really want is for my source code to have low swappiness, but everything else to have normal swappiness. Essentially, I have a lot of stuff going on in the background, but I want to limit them to 6 of 8 GBs, while always keeping the other 2 GB for the source code. I don't want to take the chance that it gets swapped...ever...because that's annoying. –  JaredC Feb 15 '11 at 5:20
    
Swappiness is system wide. In fact if you are doing something else and your files get unloaded from memory, you just have to reload it with the second line. If memory has to be freed for something else, you really don't "want to take the chance" it to be done from swap. BTW, tmpfs in the same case would also be swapped away. –  shellholic Feb 18 '11 at 11:14
    
Personally I fell a high swappiness is aboslutely horrible on workstations. Although some functions might be accelerated by the bigger cache (i.e. more cached files) this comes at a price: you pay for this in terms of responsiveness when switching between programs, which is what users notice first when working on a system. When switching from browser to office to an other browser to email I just can not abide having to wait the 1-2 seconds for each program to swap back in. On all my linux machines I generally set swappiness to a low value of 10. –  fgysin Dec 3 '13 at 13:22

The inosync daemon sounds like it does exactly what you want if you're going to rsync to a ramdisk. Instead of rsyncing every 10 seconds or so, it uses Linux's inotify facility to rsync when a file changes. I found it in the Debian repository as the inosync package, or its source is available at http://bb.xnull.de/projects/inosync/.

share|improve this answer
    
That sounds quite useful. I'll look into it and report back. Thanks! –  JaredC Feb 15 '11 at 5:14

Given sufficient memory your build out of the ramdisk does no I/O. This can speed up anything that reads or writes files. I/O is one of the slowest operations. Even if you get everything cached before the build you still have the I/Os for write, although they should have minimal impact.

You may get some speedup by pre-loading all the files into cache, but the time taken to to that should be included in the total build times. This may not give you much advantage.

Building the object and intermediate files into RAM rather than disk. Doing incremental builds may get you significant gains on frequent builds. On most projects I do a daily clean build and incremental builds in between. Integration builds are always clean builds, but I try to limit them to less than one per day.

You may gain some performance by using an ext2 partition with atime turned off. Your source should be in version control on a journaled file system like ext3/4.

share|improve this answer

As stated previously, the obvious way is to read all the directory structure and file contents of what you wish to be cached.

You can automate this by writing a script to monitor the output of vmstat 1 (use whatever equivalent tool for your OS) and keep a sum of the number of blocks written and read. Once the sum passes a threshold of your choosing, read all the files you intend to cache, reset the sum, then continue monitoring vmstat output. For quickly reading files: if your tree contains many files, avoid find ... -exec cat, instead try find ... -print0 | xargs -0 cat or a custom program that won't execute cat for each file.

Monitoring disk IO is preferable to using a fixed interval because it signals to reread your data more or less frequently depending on disk IO load.

I've used this automated method successfully on systems where I needed some index file reads to always be quick, avoiding hard drive I/O. I've also used strace to make a list of every file that gets accessed when I log in so I can keep everything hot in cache for fast logins.

This may not be the best possible solution but it suited me well.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.