Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

My scripts uses tons of read and writes. Most of them are small and random. I want Linux to wait 10 seconds before writing any data to the disk.

How would this be achieved?

I think Linux can already do this automatically. I want to do it in a more extreme way. It's like supercache in Windows.

Note: This is related to question Does linux have something like supercache? . I already know that by default linux use "supercache" like system. However the supercache-like setting for linux must be low. I want more share of memory is for supercache. I also want lazy writes every 10 seconds.

share|improve this question
1  
Any userspace application can tell the kernel to write it's dirty buffers to disk via the sync() system call. Patching the kernel to have it ignore sync() would break things in userspace. Lots of small random seeks is what spinning disks are the worst at. –  jordanm Jan 25 '13 at 5:43
    
possible duplicate of Does linux have something like supercache? –  jordanm Jan 25 '13 at 5:47
2  
@jordanm, that's not a duplicate. Here, the OP's asking for tuning of the I/O scheduling in a specific way. This is a good question that calls for a several page answer as it is a complicated matter and there are few exhaustive resources on that subject on the net. –  Stephane Chazelas Jan 25 '13 at 7:19
    
I already know that by default linux use "supercache". However the supercache setting must be low. I want more share of memory is for supercache. I also want lazy writes every 10 seconds. –  Jim Thio Jan 25 '13 at 7:38
    
This can be tweaked, but the real question is if it is worth it in this specific case. What makes you think that Linux' I/O performance is really the bottleneck in your application? Can you save the data in a tmpfs temporarily? Is using a SSD an option? –  vonbrand Jan 25 '13 at 8:13
show 3 more comments

2 Answers

There are 5 tunables in the /proc file system to change linux' writeback behavior:

dirty_async_ratio
dirty_background_ratio
dirty_sync_ratio
dirty_expire_centisecs
dirty_writeback_centisecs

The configuration is quite complicated and documentation can be found at kernel.org. However, as jordanm already said, "Any userspace application can tell the kernel to write its dirty buffers to disk via the sync() system call." which means that any other process might render your configuration useless.

Also keep your Filesystem settings in mind: Mount options like noatime, data=writeback and nobarrier can dramatically improve your throughput but will also put your data at risk, if your disk controllers are not battery backed.

share|improve this answer
    
Don't worry. I control the scripts. –  Jim Thio Jan 25 '13 at 9:13
    
Hmm... dirty_expire_centisecs is 3000. Yet somehow the IO is all the way to 100%. Changing to SSD works but still strange. –  Jim Thio Jan 25 '13 at 9:20
    
And you can use eatmydata for those programs that you don't control to disable the syncs. –  Stephane Chazelas Jan 25 '13 at 9:28
    
@JimThio dirty_expire_centisecs is not the only value you have to tweak. dirty_async_ratio controls how much of your ram will be used for writecache, until disk io starts. Also keep your Filesystem settings like in mind. What mount options are you using? –  taffer Jan 25 '13 at 9:36
1  
There are a few settings you can tweak in the I/O scheduler of each block device as well. this page gives a few ideas. It would be nice if there was a document that covers the full picture and details how each setting affect the performance in a few usage patterns, but I've not found any so far. –  Stephane Chazelas Jan 25 '13 at 9:39
add comment

You could write your changes to a file/files in memory (tmpfs) and then write these changes out to disk every 10 seconds.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.