6

I have 3 HDD and 1 SSD, I have successfully mounted all drives to bcache.

pavs@VAS:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       132G   35G   90G  28% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            3.9G  8.0K  3.9G   1% /dev
tmpfs           786M  2.3M  784M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.9G  152K  3.9G   1% /run/shm
none            100M   52K  100M   1% /run/user
/dev/bcache1    2.7T  2.1T  508G  81% /var/www/html/directlink/FTP1
/dev/bcache2    1.8T  614G  1.2T  36% /var/www/html/directlink/FTP2
/dev/bcache0    1.8T  188G  1.6T  11% /var/www/html/directlink/FTP3
/dev/sdf1       367G  284G   65G  82% /media/pavs/e93284df-e52e-4a5d-a9e1-323a388b332f

The drives that are being cached are not OS drive. Three HDD with lots of BIG files, on average the files sizes goes from 600mb to 2GB, the smallest file size being 500mb and largest being 10GB.

The files are being downloaded constantly through apache webserver. But I am only seeing marginally or no speed up in IO even on frequently accessed files. I don't know what type of cache formula bcache uses or if it can be tweaked for maximum cache performance. Ideally I would like to see frequently accessed files to be cached for at leased a day until there is no request for that file. I don't know if that level of granular cache tweaking is possible. I care about read performance only and would like to see maximum utilization of the SSD drive.


EDIT: According to this. bcache "discourages" sequential cache, which if I understand correctly, is a problem for me as most of my files are large sequential files. The default sequential cutoff was 4.0M, it might have prevented the files from being cached (I don't know), so I disabled the cutoff by doing this for each backup drives:

echo 0 > /sys/block/bcache0/bcache/sequential_cutoff

Now wait and see if it actually improves performance.


According to bcache stats all three of the drives are being cached

bcache0

pavs@VAS:~$ tail /sys/block/bcache0/bcache/stats_total/*
==> /sys/block/bcache0/bcache/stats_total/bypassed <== 
461G

==> /sys/block/bcache0/bcache/stats_total/cache_bypass_hits <== 
9565207

==> /sys/block/bcache0/bcache/stats_total/cache_bypass_misses <== 
0

==> /sys/block/bcache0/bcache/stats_total/cache_hit_ratio <== 
63

==> /sys/block/bcache0/bcache/stats_total/cache_hits <== 
3003399

==> /sys/block/bcache0/bcache/stats_total/cache_miss_collisions <== 
659

==> /sys/block/bcache0/bcache/stats_total/cache_misses <== 
1698297

==> /sys/block/bcache0/bcache/stats_total/cache_readaheads <== 
0

bcache1

pavs@VAS:~$ tail /sys/block/bcache1/bcache/stats_total/*
==> /sys/block/bcache1/bcache/stats_total/bypassed <==
396G

==> /sys/block/bcache1/bcache/stats_total/cache_bypass_hits <==
9466833

==> /sys/block/bcache1/bcache/stats_total/cache_bypass_misses <==
0

==> /sys/block/bcache1/bcache/stats_total/cache_hit_ratio <==
24

==> /sys/block/bcache1/bcache/stats_total/cache_hits <==
749032

==> /sys/block/bcache1/bcache/stats_total/cache_miss_collisions <==
624

==> /sys/block/bcache1/bcache/stats_total/cache_misses <==
2358913

==> /sys/block/bcache1/bcache/stats_total/cache_readaheads <==
0

bcache2

pavs@VAS:~$ tail /sys/block/bcache2/bcache/stats_total/*
==> /sys/block/bcache2/bcache/stats_total/bypassed <==
480G

==> /sys/block/bcache2/bcache/stats_total/cache_bypass_hits <==
9202709

==> /sys/block/bcache2/bcache/stats_total/cache_bypass_misses <==
0

==> /sys/block/bcache2/bcache/stats_total/cache_hit_ratio <==
58

==> /sys/block/bcache2/bcache/stats_total/cache_hits <==
4821439

==> /sys/block/bcache2/bcache/stats_total/cache_miss_collisions <==
1098

==> /sys/block/bcache2/bcache/stats_total/cache_misses <==
3392411

==> /sys/block/bcache2/bcache/stats_total/cache_readaheads <==
0

1 Answer 1

8

I solved the same issue by setting congested_read_threshold_us and congested_write_threshold_us following bcache documentation:

  • Traffic's still going to the spindle/still getting cache misses

    In the real world, SSDs don't always keep up with disks - particularly with slower SSDs, many disks being cached by one SSD, or mostly sequential IO. So you want to avoid being bottlenecked by the SSD and having it slow everything down.

    To avoid that bcache tracks latency to the cache device, and gradually throttles traffic if the latency exceeds a threshold (it does this by cranking down the sequential bypass).

    You can disable this if you need to by setting the thresholds to 0:

    # echo 0 > /sys/fs/bcache/<cache set>/congested_read_threshold_us

    # echo 0 > /sys/fs/bcache/<cache set>/congested_write_threshold_us

    The default is 2000 us (2 milliseconds) for reads, and 20000 for writes.

All disk IO are sent to my SSD(sde) now:

Device:    wrqm/s     r/s     w/s    rkB/s    wkB/s await svctm  %util
sdb          0.00    0.00    0.30     0.00     0.00  0.00  0.00   0.00
sdd          0.00    0.10    0.30     0.80     0.00  3.00  3.00   0.12
sdc          0.00    2.20    0.30    26.00     0.00  1.76  1.76   0.44
sda          0.00    0.20    0.20     0.80     0.00  8.00 13.00   0.52
sde        293.20   81.70  232.70  1129.20 58220.00 21.05  3.18 100.00
md1          0.00    2.50    0.30    27.60     0.00  0.00  0.00   0.00
md0          0.00    0.00    0.00     0.00     0.00  0.00  0.00   0.00
bcache0      0.00   83.00  402.40  1156.80 28994.80 31.70  2.06  99.92

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .