1

I read from here that I could load file into RAM for faster accessing using the below command.

cat filename > /dev/null

However, I wanted to test if the above statement is really true. So, I did the below testing.

  1. Create a 2.5 GB test file as below.

    dd if=/dev/zero of=demo.txt bs=100M count=10
    
  2. Now, calculated the file access time as below.

    mytime="$(time ( cat demo.txt ) 2>&1 1>/dev/null )"
    echo $mytime
    real 0m19.191s user 0m0.007s sys 0m1.295s
    
  3. As per the command suggests, now I needed to add the file to cache memory. So, I did,

    cat demo.txt > /dev/null
    
  4. Now, I assume the file is loaded into the cache. So I calculate the time to load the file again. This is the value I get.

    mytime="$(time ( cat demo.txt ) 2>&1 1>/dev/null )"
    echo $mytime
    real 0m18.701s user 0m0.010s sys 0m1.275s
    
  5. I repeated step 4 for 5 more iterations to calculate the time and these are the values I got.

    real 0m18.574s user 0m0.007s sys 0m1.279s
    real 0m18.584s user 0m0.012s sys 0m1.267s
    real 0m19.017s user 0m0.009s sys 0m1.268s
    real 0m18.533s user 0m0.012s sys 0m1.263s
    real 0m18.757s user 0m0.005s sys 0m1.274s
    

So my question is, why the time varies even when the file is loaded into the cache? I was expecting since the file is loaded into the cache, the time should come down in each iteration but that doesn't seem to be the case.

3

Nope nope nope!

This is not how it is done. Linux (the kernel) can choose to put some files in the cache and to remove them whenever it wants. You really can't be sure that anything is in the cache or not. And this command won't change that (a lot).

The advice in the link you provided is so wrong in so many ways!

  1. The cache is an OS thing. You don't need to cat the file to /dev/null to take advantage of this. This is actually a very stupid thing to do because you are forcing Linux to read the file one extra time. For instance, if you plan to read one file 4 times. If you don't care about it, the first reading will be quite slow, the 3 subsequent ones should be faster (because of caching). If you are using this "trick", the first reading will be quite slow, all the 4 subsequent ones should be faster (but not null). Just let Linux handle it.
  2. This command is only useful if you want to make sure that Linux keep it in RAM. So you have to perform it often when your system is idle. However, as I said, this is also stupid because you can never be sure that Linux actually cached the file in RAM and even if it did, you would spend time to read it in RAM or on disk (if it was not cached or already removed from the cache).
  3. By doing this repetitively on a big file, you basically trick Linux into thinking that this file should be in RAM at the expense of other files that you actually use more often.

So the conclusion here: don't do this kind of tricks, this is usually counterproductive.

However, if you know that some small files (compared to your RAM size) would really benefit from being accessed from RAM, you can use a tmpfs mount and store your file there. On modern distribs, the /tmp folder is usually a tmpfs one.

Another alternative that I personally found worthy is to compress your file at the FS level with BTRFS for instance or manually (but this one requires that the program that access the file has the ability to decompress it). Of course, your files should benefit from compression, otherwise this is useless. This way, you could be much more confident that Linux keeps your compressed file in RAM (since it's smaller) and if your application is IO bound, loading 100MB from disk instead of loading 10GB should be much faster.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.