Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I’d like to set the system up to use most RAM for file system metadata caching, but only a reasonably small amount for read/write caching and prefetching files. Ideally I would like to be able to browse the filesystem (as much as it fits in the RAM) without spinning up the disks until I actually open a file.

Here are the details:

I have a home-made file server. It’s got five disks in a LVM volume around 9TB, but only 4GB of RAM. Since the server doesn’t do much else then serve files, most of the RAM is used for caching. (“free” reports 3.4G out of 3.9G used for cache.)

The server lives in my bedroom, and if all disks are spinning it makes enough noise to be annoying when it’s quiet. (I don’t mean seek noise, just spinning noise. The disks are of various makes and models, and I think slight differences in rotation speeds cause interference. No disk is noisy on its own, but if some of them are spinning together there’s a slight noise with a sub-Hertz period.) So I configured the server to spin down the disks most of the time.

Of course, if the disks are spun down when I open a folder in my file manager, there’s a delay while whichever of the disks has that folder spins up. Just that is no big deal. But depending on where I look, it can happen several times in a row, if LVM happened to spread the metadata for each subfolder on the different disks.

I suspect that Linux mostly fills its cache with file contents, and possibly prefetched data. The caching isn’t very useful beyond a few MB to ensure smooth playback; if I just watched a movie I probably won’t look at it again anytime soon. Prefetching, if it happens, is also completely useless in my case, after more than a few MB.

But one would think that 4GB should be plenty to be able to cache most file-system metadata, at least those parts that were already visited, so that I could browse the files without needing to spin up the disks if it turns out they’re sleeping.

There would still be a delay when opening the file, but that’s OK. Compare “click; wait; click; wait; click; wait; play; watch” with “click; click; click; play; wait; watch”. The former is incredibly frustrating; the latter is almost expected.

Notes:

  1. Should it matter, the kernel is 3.2, the OS is Debian, the volume is lvm2, and the FS is ext4.

  2. The only reason for the spin down is noise during the night; the server is otherwise running continuously. (I made it as low-power as reasonable.) The spin-down delay is varied depending on the time of day.

  3. The hard disks are only for media. The OS is on a separate (small) flash drive. (Which means any spin-up delays comes from the data, not just because it needed something in /usr or whatever. I could spare a few GB on it if it would help with my problem somehow.

  4. A reasonable impact on performance is not a big deal. The disks are faster than my network anyway.

share|improve this question
    
Not related to your question but if you're on very little memory, you should look into using KSM. It's a kernel thread that goes through your memory, looks for identical pages and merges them into one, updating the page table for the process that had the page it's removing. Updates to the page are done copy-on-write style. I have a 8 GB hypervisor running about 10 VMs mostly out of physical memory because of this. On your thing, you might look into vfs_cache_pressure. That affects the page cache (contents) to metadata eviction ratio. –  Joel Davis May 22 '13 at 23:02
    
Also, swappiness hits application data exclusively, so you may try to amp up how much gets swapped out. –  Joel Davis May 22 '13 at 23:09
add comment

1 Answer

To control how Linux caches things refer to this https://www.kernel.org/doc/Documentation/sysctl/vm.txt

In particular look at vfs_cache_pressure, you probably want a really low value or maybe even zero (1 sounds a bit safer to me though):

vfs_cache_pressure
------------------

Controls the tendency of the kernel to reclaim the memory which is used for
caching of directory and inode objects.

At the default value of vfs_cache_pressure=100 the kernel will attempt to
reclaim dentries and inodes at a "fair" rate with respect to pagecache and
swapcache reclaim.  Decreasing vfs_cache_pressure causes the kernel to prefer
to retain dentry and inode caches. When vfs_cache_pressure=0, the kernel will
never reclaim dentries and inodes due to memory pressure and this can easily
lead to out-of-memory conditions. Increasing vfs_cache_pressure beyond 100
causes the kernel to prefer to reclaim dentries and inodes.

Also you may want to modify swappiness so that you never swap data or make it so that it only happens in extreme cases.

The drop_caches option might be handy for explicitly dropping the data you don't want cached anymore.

I'm sure there are probably other options that may help, so review the kernel documentation.

To apply them I'd put the settings you want to change in /etc/sysctl.conf or whatever your OS has to restore them at boot.

share|improve this answer
2  
Good post, but you would want as much swapping as feasible given the OP's goal. Swapping only hits user memory so increasing it's tendency to swap out to disk then leaves more physical memory for the caches to take up. Increasing swappiness frees memory but can slow down applications if it's increased too far (determining the sweet spot is basically iterative guestimation) –  Joel Davis May 23 '13 at 0:45
    
Hi Kyle, thanks for the idea. vfs_cache_pressure sort of works, but it’s not quite enough. Here’s what I did: –  bogdanb May 24 '13 at 8:39
    
When I set vcp to 0, if I do a find / -ls > /dev/null, then spin down the disks, then find all the files again, the disks don't spin up. free shows buffers rising to about 202MB when doing this. But, if I do the find, then cat /file/bigger/than/ram > /dev/null, then free shows cached rising to fill the empty space, and for some reason buffers goes down to about 195MB. Then if I spin down the discs and do the find again the disks still spin up :-( –  bogdanb May 24 '13 at 8:47
    
About swappiness: It’s set to the default of 60, but the machine doesn’t have any swap partition, so I’m not sure if it does much. I guess I could put a swap file on the flash drive, but I’ve no idea how it would help, or how to size it. –  bogdanb May 24 '13 at 8:56
    
By the way, guys, is there a way to tell the kernel something like “whatever you do, don’t use more than 2GB for caches”? I keep looking through the docs but I can’t seem to find anything that simple... –  bogdanb May 24 '13 at 9:00
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.