5

After reading some articles on the Linux VFS page cache and the tunable parameters like dirty_ratio i was under the impression that page cache would operate as both read and write caching layer.

But using the simple test below it works well to improve read speed for files that are located in the page cache but doesn't seem to work on writes.

e.g.

Clear the cache and write to file.

# swapoff -a
# echo 3 > /proc/sys/vm/drop_caches

# dd if=/dev/zero of=/home/flo/test bs=1M count=30
30+0 records in
30+0 records out
31457280 bytes (31 MB) copied, 0.182474 s, 172 MB/s

Check that file is actually in page cache

# vmtouch /home/flo/test 
           Files: 1
     Directories: 0
  Resident Pages: 7680/7680  30M/30M  100%
         Elapsed: 0.000673 seconds

Read from file to confirm is actually coming from cache.

# dd if=/home/flo/test of=/dev/null bs=1M count=30
30+0 records in
30+0 records out
31457280 bytes (31 MB) copied, 0.00824169 s, 3.8 GB/s

Drop cache and read again to prove speed difference.

# echo 3 > /proc/sys/vm/drop_caches
# dd if=/home/flo/test of=/dev/null bs=1M count=30
30+0 records in
30+0 records out
31457280 bytes (31 MB) copied, 0.132531 s, 237 MB/s

Since i'm not using DIRECT_IO with dd I was expecting the page cache to be used as a writeback type of cache. And based on dirty_ratio or dirty_expire_centiseconds... eventually the data would be committed to disk.

Can someone please explain how VFS handles the read and write process differently, especially during writes and why there is no speed gain.

Is there any way to make the vfs more aggressive in write caching so it behaves more like the writeback cache you might find on a raid controller for example.

Thank you

fLo

1

To see the fast behaviour, I have to do rm test first. E.g. I see dd report 1GB/s instead of 150MB/s.

References:

Although the references only explain why I thought to try this, it doesn't actually explain why it causes the IO to block.

On my computer the blocking only seemed to happen inside the new WBT ("writeback throttling") code... which was added in 2016, after you asked your question. I haven't analyzed why it would cause this. And it went away when WBT is disabled.

My kernel version is 4.18.16-200.fc28.x86_64.

strace -T shows that all the time was spent in close(), which makes the most sense to me. I tried to use perf as well. It didn't work how it was supposed to, but it showed stack traces like

dd 17068 [003] 475165.381526:       sched:sched_switch: dd:17068 [120] T ==> kworker/3:1H:19326 [100]
    ffffffffa390c172 __sched_text_start+0x352 ([kernel.kallsyms])
    ffffffffa390c172 __sched_text_start+0x352 ([kernel.kallsyms])
    ffffffffa390c6a8 schedule+0x28 ([kernel.kallsyms])
    ffffffffa30def32 io_schedule+0x12 ([kernel.kallsyms])
    ffffffffa3461ed7 wbt_wait+0x337 ([kernel.kallsyms])
    ffffffffa342ee33 blk_queue_bio+0x123 ([kernel.kallsyms])
    ffffffffa342d114 generic_make_request+0x1a4 ([kernel.kallsyms])
    ffffffffa342d3c5 submit_bio+0x45 ([kernel.kallsyms])
    ffffffffa3377d78 ext4_io_submit+0x48 ([kernel.kallsyms])
    ffffffffa335da2c ext4_writepages+0x70c ([kernel.kallsyms])
    ffffffffa3209311 do_writepages+0x41 ([kernel.kallsyms])
    ffffffffa31f808e __filemap_fdatawrite_range+0xbe ([kernel.kallsyms])
    ffffffffa334b9ec ext4_release_file+0x6c ([kernel.kallsyms])
    ffffffffa32a9d4e __fput+0xae ([kernel.kallsyms])
    ffffffffa30cf474 task_work_run+0x84 ([kernel.kallsyms])
    ffffffffa3003e6e exit_to_usermode_loop+0xce ([kernel.kallsyms])
    ffffffffa300425d do_syscall_64+0x14d ([kernel.kallsyms])
    ffffffffa3a00088 entry_SYSCALL_64_after_hwframe+0x44 ([kernel.kallsyms])
        7fcca3a60654 __close+0x14 (/usr/lib64/libc-2.27.so)

which reminded me I was currently testing the deadline I/O scheduler, with WBT ("writeback throttling") enabled. Disabling WBT (including by switching to CFQ, which is incompatible) gave me the fast behaviour again!

The perf commands I used to see this were:

sudo perf record -e sched:sched_stat_sleep -e sched:sched_switch -e sched:sched_process_exit -gP -o ~/perf.data dd if=/dev/zero of=test bs=1M count=30
sudo perf script -i ~/perf.data | cat
| improve this answer | |
1

man ext4 has this introdction about (no)auto_da_alloc option:

Many broken applications don't use fsync()...

There seems to be a long story (some tragedy about data loss) behind this. It has to do with delayed allocation of filesystem blocks. Ext2/3 did not have that, but it is quite an important feature of not only ext4.

If the application doesn't sync, and neither the user manually, and the kernel only after 30 secs, then the filesystem better do it right away when some file re-write is involved. Otherwise, with DA, bad things can happen easily in a power failure. Worse things than just losing the last changes.

Without conv=notruncate the dd command acts like an "application" when overwriting. It has to get rid of the existing file to create the new, otherwise you will get a mix if the existing file is longer.

With mount -o remount,noauto_da_alloc ... you can switch this behaviour off on ext4. Now the block writing can be done long time after the truncating.

The next level of aggression would be to raise the 30s expiry time and 5s check interval (the dirty_..._centisecs values in /proc/sys/vm/) for the periodic writebacks. With the defauilt 30/5 some new files will just be written half a minute later, unless you are very fast to delete faster.

The more aggressive the VFS is to unused pages, the less aggressive the filesystem has to be with the block device.


Mount options and Writeback parameters

]# findmnt --real
TARGET       SOURCE     FSTYPE OPTIONS
/            /dev/sda3  ext4   rw,relatime,noauto_da_alloc
|-/root/sda1 /dev/sda1  ext2   rw,relatime
`-/root/16   /dev/sda16 ext4   rw,relatime

In a setup like this an overwrite is synced immediately on sda16, but not on the two others.

For the moment I (think I) turned off the periodic writeback completely.

]# grep '' /proc/sys/vm/*centisecs
/proc/sys/vm/dirty_expire_centisecs:720000
/proc/sys/vm/dirty_writeback_centisecs:0

And now I finally am gathering dirty pages:

]# grep nr_dirty /proc/vmstat 
nr_dirty 10077
nr_dirty_threshold 437320
nr_dirty_background_threshold 174671

I try to gather them and get somehow close to the default 10% backgroud ratio - I got synced yesterday when I went to suspend-to-ram sleep. Makes sense: who wants to sleep with MBs of dirty pages?

mm/writeback.c is very complex in detail, the comments tell themselves. One problem is not to miss the throttling point when "1000 dd start dirtying at once". "Writeback" seems to aim at around these 10% bg-ratio, in the long run. As my example above shows these 10% (of total/available RAM) take long to fill up under normal (minimal) use. A minute of browsing dirties 1000 pages, roughly.


After the theory, the specific proof

I tested 10 blocks on two of the above listed filesystems:

]# dd if=/dev/zero of=test10  bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0076396 s, 1.4 GB/s

]# dd if=/dev/zero of=test10  bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.00514406 s, 2.0 GB/s

-> with noauto_da_alloc on the root partition (sda3, above) the overwrite is even faster.

On the default mounted ext4 (sda16 above) it slows down:

]# rm test10 

]# dd if=/dev/zero of=test10  bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.00800839 s, 1.3 GB/s

]# dd if=/dev/zero of=test10  bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0740824 s, 142 MB/s

...because the whole overwrite is synced, as vmstat 1 |cut... shows:

    0     0
    0     0
    0     0
-----io----
   bi    bo
    0 10240
    0     0
    0     0

Manually sync with delayed allocation

The good thing about it is: you do it when you want to, and you can do it to single files, but also to the whole drive(s).

Also: unmounting, shutting down (and suspending) has it included.

Bad thing is: zero-length "corruption" risk when a crash/powerfailure happens between a (over)write and a sync. Meaning you only really have for safe only what you put on an external storage, or two.


I can not find a bottom line. There is no easy solution, only long (but at least logical) explanations.

| improve this answer | |
0

Just don't use dd. For example, use cp and you will get pagecache for writes alright.

| improve this answer | |
  • nope. dd does the same by default. E.g. for me time head -c 30M /dev/zero >test shows 0.23s, same as the dd command. – sourcejedi Nov 23 '18 at 21:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.