0

I've got some code that processes a load of data stored on an external HDD. There is about 6.5 GiB of data on the disk, but only a few files are processed rather than the entire contents.

I created a ramdisk using a tmpfs filesystem to store a copy of the disk contents in memory to speed up the processing time required. Every time I reboot my system, I have to copy the contents of the disk back into this ramdisk.

This works ok, because the entire contents can fit into the amount of available memory that I have.

However it may not work later on when I change to a different, larger dataset.

This is quite inefficient, because the amount of data that I actually read is only about 1 GB. (A bit over 10 % in my estimates.)

Is there a way to create a ramdisk in memory which caches data read from this external disk?

I'm sure such a thing must be possible with linux - but I don't know what to search for. Ramdisks are already quite a niche topic.

Btw I'm using Debian Testing, which at the time of writing is the testing branch ahead of Debian 10.

Edit: I do not write back to this disk, if it makes any difference.

1
0

You might want to look into using an OverlayFS. An overlay filesystem would allow you to merge your disk mountpoint and your ramdisk mountpoint into one stacked filesystem (the overlay).

OverlayFS operates with copy-on-write, so if your upper layer is an empty ramdisk at boot time, data will only be written to the ramdisk when you open the files with write permissions. In theory, you could touch any of the files you wanted copied into RAM up front.

Though, I should add, Linux already has robust disk caching. You may want to make sure you're not reinventing the wheel here.

0

This does not answer your question, but might help with the problem described. You want to

store a copy of the disk contents in memory to speed up the processing time required.

You don't need to do anything. This happens by default.

Try for yourself:

dd if=/dev/urandom bs=1M count=1M of=test.bin # create a file of 10 GB random data
echo 3 | sudo tee /proc/sys/vm/drop_caches # drop caches
dd if=test.bin bs=1G count=1 of=/dev/null # read 1 GB of data
dd if=test.bin bs=1G count=1 of=/dev/null # read again

The first read of takes a while:

1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10,43 s, 103 MB/s

After the first read, the data remains in memory. Subsequent reads are much faster:

1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.320478 s, 3.4 GB/s

Linux will happily support your endeavors by keeping recently read file-content in memory as long there is enough free memory available.

5
  • I didn't know about this. How does Linux decide how large to make this buffer? Is it dynamic? May 5 '20 at 21:51
  • Interesting. I fail to find any concrete information on this. Looking at my systems, I can only see that up to 50 % of total ram is typically used for the cache. I do not know if 50 % is a hard upper limit or not.
    – Hermann
    May 6 '20 at 10:34
  • Does that amount show up in things like system monitors / htop ? My system has 24 GiB of ram in it - I've never noticed 12 GiB + typical Chrome usage of ~ few GB being used on my system, so I wonder if this is something that is done in a transparent way so that the user is totally unaware of it. May 6 '20 at 13:59
  • Yes, it does. Buffers (data to write) and cache (on-disk data available from RAM) are shown blue in htop. You can also look at the column "buff/cache" in the output of env LC_ALL=C free -h. Please be aware that buffers and cache do not count towards "used" RAM as it can be flushed to disk (buffers) or discarded (cache). "used" is for application data only – this is why the amount of cached data typically is not visible. Also, you cannot really control which data is cached and which is not.
    – Hermann
    May 6 '20 at 15:00
  • Strange, I guess I never read that much data from my disk on a day-to-day basis then, because I can't say I've ever noticed. May 6 '20 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.