My system has 2 physical 4 TB hds partially md mirrored, and a very fast 512GB ssd M.2 device that stores the root filesystems and caches key larger filesystems on disks. One particular fs stores VMWare Workstation virtual machine disk files. These files can be very large (10-70GB). The most common VM I boot is a Windows 10 image with a 78GB base image and another 6GB snapshot file.
I'm looking for LVM cache tunable parameters that would allow this filesystem and these files in particular to perform better.
For comparison, the same M.2 SSD also has a real Win 10 image on it, and booting that image straight will take about 8 seconds from Grub selection to Windows login screen. By comparison, from VMWare boot selection to login is about 28 seconds; not a lot better than if caching was turned off (though I haven't done that test recently so I don't have a quotable number).
The Win 10 VM total directory is 82GB, and here is some specifics of my lvm (focus on the vmCache at the end)
lvs -a -o+devices LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices games cache Cwi-aoC--- 200.00g [gamesDataCache] 11.37 16.05 0.00 games_corig(0) [gamesDataCache] cache Cwi---C--- 10.00g 11.37 16.05 0.00 gamesDataCache_cdata(0) [gamesDataCache_cdata] cache Cwi-ao---- 10.00g /dev/nvme0n1p6(23015) [gamesDataCache_cmeta] cache ewi-ao---- 12.00m /dev/nvme0n1p6(23012) [games_corig] cache owi-aoC--- 200.00g /dev/md126(0) home cache Cwi-aoC--- 300.00g [homeDataCache] 100.00 16.05 0.01 home_corig(0) [homeDataCache] cache Cwi---C--- 10.00g 100.00 16.05 0.01 homeDataCache_cdata(0) [homeDataCache_cdata] cache Cwi-ao---- 10.00g /dev/nvme0n1p6(3) [homeDataCache_cmeta] cache ewi-ao---- 12.00m /dev/nvme0n1p6(0) [home_corig] cache owi-aoC--- 300.00g /dev/md127(128000) [lvol0_pmspare] cache ewi------- 79.90g /dev/md127(204800) vm cache Cwi-aoC--- 500.00g [vmCache] 100.00 19.01 0.00 vm_corig(0) [vmCache] cache Cwi---C--- 79.80g 100.00 19.01 0.00 vmCache_cdata(0) [vmCache_cdata] cache Cwi-ao---- 79.80g /dev/nvme0n1p6(2563) [vmCache_cmeta] cache ewi-ao---- 80.00m /dev/nvme0n1p6(22992) [vm_corig] cache owi-aoC--- 500.00g /dev/md127(0) root0 fedora -wi-ao---- 39.00g /dev/nvme0n1p5(1)
The cache size is almost 80GB, and this Win 10 is the only VM I boot, so I'd hope it would be able to cache pretty much the entire image. The data usage is 100%, but yet performance is far below what I hoped for.
I can provide any more detailed LVM configs on request, but assume most values are default right now.