Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am neither concerned about RAM usage (as I've got enough) nor about losing data in case of an accidental shut-down (as my power is backed, the system is reliable and the data are not critical). But I do a lot of file processing and could use some performance boost.

That's why I'd like to set the system up to use more RAM for file system read and write caching, to prefetch files aggressively (e.g. read-ahead the whole file accessed by an application in case the file is of sane size or at least read-ahead a big chunk of it otherwise) and to flush writing buffers less frequently. How to achieve this (may it be possible)?

I use ext3 and ntfs (I use ntfs a lot!) file systems with XUbuntu 11.10 x86.

share|improve this question
    
Do you have a raid-controller or a "normal" disc controller capable of doing write-ahead? –  Nils Mar 18 '12 at 20:37
3  
If you have lots of RAM, care a lot about performance and don't care about data loss, just copy all your data to a RAM disk and serve it from there, discarding all updates on crash/shutdown. If that won't work for you, you may need to qualify "enough" for RAM or how critical the data isn't. –  James Youngman Mar 19 '12 at 0:16
    
@Nils, the computer is a laptop, so, I believe, the controller is pretty ordinary. –  Ivan Mar 19 '12 at 0:17
    
So the controller won`t help you here. Can you comment on the answers how the according settings improved your throughput? –  Nils Mar 19 '12 at 21:57
    
One way to improve performance a lot is to skip durability of data. Simply disable syncing to disk even if some apps requests for sync. This will cause data loss if your storage device ever suffers loss of electricity. If you want to do it anyway, simply execute sudo mount -o ro,nobarrier /path/to/mountpoint or adjust /etc/fstab to include nobarrier for any filesystem that you're willing to sacrifice for improved performance. However, if your storage device has internal battery such as Intel 320 SSD series, using nobarrier causes no data loss. –  Mikko Rantalainen Apr 11 at 8:39

5 Answers 5

Improving disk cache performance in general is more than just increasing the file system cache size unless your whole system fits in RAM in which case you should use RAM drive (tmpfs is good because it allows falling back to disk if you need the RAM in some case) for runtime storage (and perhaps an initrd script to copy system from storage to RAM drive at startup).

You didn't tell if your storage device is SSD or HDD. Here's what I've found to work for me (in my case sda is a HDD mounted at /home and sdb is SSD mounted at /).

First optimize the load-stuff-from-storage-to-cache part:

Here's my setup for HDD (make sure AHCI+NCQ is enabled in BIOS if you have toggles):

echo cfq > /sys/block/sda/queue/scheduler
echo 10000 > /sys/block/sda/queue/iosched/fifo_expire_async
echo 250 > /sys/block/sda/queue/iosched/fifo_expire_sync
echo 80 > /sys/block/sda/queue/iosched/slice_async
echo 1 > /sys/block/sda/queue/iosched/low_latency
echo 6 > /sys/block/sda/queue/iosched/quantum
echo 5 > /sys/block/sda/queue/iosched/slice_async_rq
echo 3 > /sys/block/sda/queue/iosched/slice_idle
echo 100 > /sys/block/sda/queue/iosched/slice_sync
hdparm -q -M 254 /dev/sda

Worth noting for the HDD case is high fifo_expire_async (usually write) and long slice_sync to allow a single process to get high throughput (set slice_sync to lower number if you hit situations where multiple processes are waiting for some data from the disk in parallel). The slice_idle is always a compromise for HDDs but setting it somewhere in range 3-20 should be okay depending on disk usage and disk firmware. I prefer to target for low values but setting it too low will destroy your throughput. The quantum setting seems to affect throughput a lot but try to keep this as low as possible to keep latency on sensible level. Setting quantum too low will destroy throughput. Values in range 3-8 seem to work well with HDDs. The worst case latency for a read is (quantum * slice_sync) + (slice_async_rq * slice_async) ms if I've understood the kernel behavior correctly. The async is mostly used by writes and since you're willing to delay writing to disk, set both slice_async_rq and slice_async to very low numbers. However, setting slice_async_rq too low value may stall reads because writes cannot be delayed after reads any more. My config will try to write data to disk at most after 10 seconds after data has been passed to kernel but since you can tolerate loss of data on power loss also set fifo_expire_async to 3600000 to tell that 1 hour is okay for the delay to disk. Just keep the slice_async low, though, because otherwise you can get high read latency.

The hdparm command is required to prevent AAM from killing much of the performance that AHCI+NCQ allows. If your disk makes too much noise, then skip this.

Here's my setup for SSD (Intel 320 series):

echo cfq > /sys/block/sdb/queue/scheduler
echo 1 > /sys/block/sdb/queue/iosched/back_seek_penalty
echo 10000 > /sys/block/sdb/queue/iosched/fifo_expire_async
echo 20 > /sys/block/sdb/queue/iosched/fifo_expire_sync
echo 1 > /sys/block/sdb/queue/iosched/low_latency
echo 6 > /sys/block/sdb/queue/iosched/quantum
echo 2 > /sys/block/sdb/queue/iosched/slice_async
echo 10 > /sys/block/sdb/queue/iosched/slice_async_rq
echo 1 > /sys/block/sdb/queue/iosched/slice_idle
echo 20 > /sys/block/sdb/queue/iosched/slice_sync

Here it's worth noting the low values for different slice settings. The most important setting for an SSD is slice_idle which must be set to 0-1. Setting it to zero moves all ordering decisions to native NCQ while setting it to 1 allows kernel to order requests (but if the NCQ is active, the hardware may override kernel ordering partially). Test both values to see if you can see the difference. For Intel 320 series, it seems that setting slide_idle to 0 gives the best throughput but setting it to 1 gives best (lowest) overall latency.

For more information about these tunables, see http://www.linux-mag.com/id/7572/.

Now that we have configured kernel to load stuff from disk to cache with sensible performance, it's time to adjust the cache behavior:

According to benchmarks I've done, I wouldn't bother setting read ahead via blockdev at all.

Set system to prefer swapping file data over application code (this does not matter if you have enough RAM to keep whole filesystem and all the application code and all virtual memory allocated by applications in RAM). This reduces latency for swapping between different applications over latency for accessing big files from a single application:

echo 15 > /proc/sys/vm/swappiness

Do not ever set this to 0 but if you prefer to keep applications pretty much always in RAM you could set this to 1. If you were memory limited and working with big files (e.g. HD video editing), then it might make sense to set this close to 100.

Next, tell kernel to prefer keeping directory hierarchy in memory over file contents in case some RAM needs to be freed (again, if everything fits in RAM, this setting does nothing):

echo 10 > /proc/sys/vm/vfs_cache_pressure

Setting vfs_cache_pressure to low value makes sense because in most cases, the kernel needs to know the directory structure before it can use file contents from the cache and flushing the directory cache too soon will make the file cache next to worthless. Consider going all the way down to 1 with this setting if you have lots of small files (my system has around 150K 10 megapixel photos and counts as "lots of small files" system). Never set it to zero or directory structure is always kept in memory even if the system is running out of the memory. Setting this to big value is sensible only if you have only a few big files that are constantly being re-read (again, HD video editing without enough RAM would be an example case).

Finally tell the kernel to use up to 99% of the RAM as cache for writes and instruct kernel to use up to 50% of RAM before slowing down the process that's writing (default for dirty_background_ratio is 10). Warning: I personally would not do this but you claimed to have enough RAM and are willing to lose the data.

echo 99 > /proc/sys/vm/dirty_ratio
echo 50 > /proc/sys/vm/dirty_background_ratio

And tell that 1h write delay is ok to even start writing stuff on the disk (again, I would not do this):

echo 360000 > /proc/sys/vm/dirty_expire_centisecs
echo 360000 > /proc/sys/vm/dirty_writeback_centisecs

If you put all of those to /etc/rc.local and include following at the end, everything will be in cache as soon as possible after boot (only do this if your filesystem really fits in the RAM):

(nice find / -type f -and -not -path '/sys/*' -and -not -path '/proc/*' -print0 2>/dev/null | nice ionice -c 3 wc -l --files0-from - > /dev/null)&

Or a bit simpler alternative which might work better (cache only /home and /usr, only do this if your /home and /usr really fit in RAM):

(nice find /home /usr -type f -print0 | nice ionice -c 3 wc -l --files0-from - > /dev/null)&
share|improve this answer
2  
A well-informed and overall much better answer than the accepted one! This one is underrated... I guess most people just want simple instructions without bothering to understand what they really do... –  CyberShadow Jan 26 '13 at 18:17
1  
@Phpdevpad: In addition, the question said "I am neither concerned about RAM usage [...]"--I don't think any Maemo device qualifies. –  Mikko Rantalainen Jan 28 '13 at 7:11
    
Isn't noop or deadline a better scheduler for SSDs? –  rep_movsd Aug 14 '13 at 7:58
    
@rep_movsd I've been using only intel SSD drives but at least these drives are still slow enough to have better overall performance with more intelligent schedulers such as CFQ. I'd guess that if your SSD drive can deal with more than 100K random IOPS, using noop or deadline would make sense even with fast CPU. With "fast CPU" I mean something that has at least multiple 3GHz cores available for IO only. –  Mikko Rantalainen Aug 15 '13 at 8:58

Firstly, I DO NOT recommend you continue using NTFS, as ntfs implemention in Linux would be performance and security trouble at any time.

There are several things you can do:

  • use some newer fs such as ext4 or btrfs
  • try to change your io scheduler, for example bfq
  • turn off swap
  • use some automatic preloader like preload
  • use something like systemd to preload while booting
  • ... and something more

Maybe you want to give it a try :-)

share|improve this answer
    
I've already moved entirely away from NTFS to ext4 once, leaving the only NTFS partition to be the Windows system partition. But it turned in many inconveniences for me and I have turned back to NTFS as the main data partition (where I store all my documents, downloads, projects, source code etc.) file system. I don't give up rethinking my partitions structure and my workflow (to use less Windows) but right now giving up NTFS doesn't seem a realistic option. –  Ivan Feb 3 '12 at 12:39
    
If you have to use your data inside Windows too, NTFS may be the only option. (many other options available if you can use your Windows just as a VM inside linux) –  Felix Yan Feb 3 '12 at 12:41

Read ahead:

On 32 bit systems:

blockdev --setra 8388607 /dev/sda

On 64 bit systems:

blockdev --setra 4294967295 /dev/sda

Write behind cache:

echo 100 > /proc/sys/vm/dirty_ratio

This will use up to 100% of your free memory as write cache.

Or you can go all out and use tmpfs. This is only relevant if you have RAM enough. Put this in /etc/fstab. Replace 100G with the amount of physical RAM.

tmpfs /mnt/tmpfs tmpfs size=100G,rw,nosuid,nodev 0 0

Then:

mkdir /mnt/tmpfs; mount -a

Then use /mnt/tmpfs.

share|improve this answer
3  
3GB or 2TB readahead? really? Do you even know what these options do? –  Cobra_Fast Dec 25 '13 at 4:11

You can set the read-ahead size with blockdev --setra sectors /dev/sda1, where sectors is the size you want in 512 byte sectors.

share|improve this answer

My killer setting is very simple and very effective:

echo "2000" > /proc/sys/vm/vfs_cache_pressure

The explanation from kernel documentation:

vfs_cache_pressure

Controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects.

At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. When vfs_cache_pressure=0, the kernel will never reclaim dentries and inodes due to memory pressure and this can easily lead to out-of-memory conditions. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.

vfs_cache_pressure at 2000 causes that most of computing happens in the RAM and very late disk writes.

share|improve this answer
    
Setting vfs_cache_pressure too high (I would consider 2000 too high) will cause unnecessary disk access even for simple stuff such as directory listings which should easily fit in cache. How much RAM do you have and what are you doing with the system? As I wrote in my answer, using high value for this setting makes sense for e.g. HD video editing with limited RAM. –  Mikko Rantalainen Sep 30 at 10:51

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.