3

My system has 2 physical 4 TB hds partially md mirrored, and a very fast 512GB ssd M.2 device that stores the root filesystems and caches key larger filesystems on disks. One particular fs stores VMWare Workstation virtual machine disk files. These files can be very large (10-70GB). The most common VM I boot is a Windows 10 image with a 78GB base image and another 6GB snapshot file.

I'm looking for LVM cache tunable parameters that would allow this filesystem and these files in particular to perform better.

For comparison, the same M.2 SSD also has a real Win 10 image on it, and booting that image straight will take about 8 seconds from Grub selection to Windows login screen. By comparison, from VMWare boot selection to login is about 28 seconds; not a lot better than if caching was turned off (though I haven't done that test recently so I don't have a quotable number).

The Win 10 VM total directory is 82GB, and here is some specifics of my lvm (focus on the vmCache at the end)

lvs -a -o+devices
LV                     VG     Attr       LSize   Pool         Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices                            
games                  cache  Cwi-aoC--- 200.00g [gamesDataCache]        11.37  16.05           0.00             games_corig(0)
[gamesDataCache]       cache  Cwi---C---  10.00g                         11.37  16.05           0.00             gamesDataCache_cdata(0)
[gamesDataCache_cdata] cache  Cwi-ao----  10.00g                                                                 /dev/nvme0n1p6(23015)  
[gamesDataCache_cmeta] cache  ewi-ao----  12.00m                                                                 /dev/nvme0n1p6(23012)  
[games_corig]          cache  owi-aoC--- 200.00g                                                                 /dev/md126(0)          
home                   cache  Cwi-aoC--- 300.00g [homeDataCache]         100.00 16.05           0.01             home_corig(0)          
[homeDataCache]        cache  Cwi---C---  10.00g                         100.00 16.05           0.01             homeDataCache_cdata(0) 
[homeDataCache_cdata]  cache  Cwi-ao----  10.00g                                                                 /dev/nvme0n1p6(3)      
[homeDataCache_cmeta]  cache  ewi-ao----  12.00m                                                                 /dev/nvme0n1p6(0)      
[home_corig]           cache  owi-aoC--- 300.00g                                                                 /dev/md127(128000)     
[lvol0_pmspare]        cache  ewi-------  79.90g                                                                 /dev/md127(204800)     
vm                     cache  Cwi-aoC--- 500.00g [vmCache]               100.00 19.01           0.00             vm_corig(0)            
[vmCache]              cache  Cwi---C---  79.80g                         100.00 19.01           0.00             vmCache_cdata(0)       
[vmCache_cdata]        cache  Cwi-ao----  79.80g                                                                 /dev/nvme0n1p6(2563)
[vmCache_cmeta]        cache  ewi-ao----  80.00m                                                                 /dev/nvme0n1p6(22992)
[vm_corig]             cache  owi-aoC--- 500.00g                                                                 /dev/md127(0)      
root0                  fedora -wi-ao----  39.00g                                                                 /dev/nvme0n1p5(1)

The cache size is almost 80GB, and this Win 10 is the only VM I boot, so I'd hope it would be able to cache pretty much the entire image. The data usage is 100%, but yet performance is far below what I hoped for.

I can provide any more detailed LVM configs on request, but assume most values are default right now.

Any suggestions?

Thanks,

Brian

1

What kind of system do you use (OS + version + what virtualization suite / package)?

I found out the LVM cache on CentOS 6.8 has problems with a large cache. If I use 100GB SSD and combine it with 2 TB HDD (software raid SSD and HDD) I notice no improvement in speed! The SATA controller though

iostat -m -x 2

shows that the SSD sata ports are 100% busy but only writing ca. 120MB / sec (same speed as HDD). When i shrink the size of the cache to 10GB thinks get a lot better! Both with same chunksize btw:

lvconvert --type cache-pool --chunksize 960 --cachemode writeback --poolmetadata ${VGBASE}/cachemeta ${VGBASE}/cachedata

dmesg also reports (when having the 100G cache)

device-mapper: cache: You have created a cache device with a lot of individual cache blocks (1670400)

Tuning the chunksize could help you probably, but there seems a problem in setting this.

The block size must be between 64 (32KB) and 2097152 (1GB) and a multiple of 64 (32KB). https://www.kernel.org/doc/Documentation/device-mapper/cache.txt

Following these rules you can actually change the chunksize but with a lot of chunksizes the flushing back to disk does not work!! I recommend you to try to uncache the cached volume and see if you don't get in an endless loop like this:

X blocks must still be flushed (this keeps on coming for ever!)

and use iostat to investigate if SSD takes the writes fast and the HDD actually gets them afterwards (see Cpy%Sync column when issuing lvs -a).

So make sure you can remove it (flush) and make sure it behaves the way it should via iostat.

Of course, if the cache is full... the speed will drop to that of the HDD because it needs to flush before it can write new stuff there.

But my conclusion so far is that it is difficult to tune and probably some bugs that have been fixed in CentOS 7 are making it difficult on CentOS 6.8.

P.S. for your windows VM to boot up fast, it should be on the SSD. So in this example case it would probably not work; you have dozens of OS images running at the same time that do not fit the cache. When rebooting you HW node, the complete OS image will not be on the cache anymore (only partly) because the sections needed for booting have not been touched for long. After reboot the image must be reinitialized through the HDD. But this probably is not your problem I guess.

  • On Ubuntu 18.04, I'm seeing the same issue with larger cache having slow performance. I've done a lot of incremental testing and found the tipping point on my system at 60.0 GB size cache. This is using the default chunk size, 64k, on my system. I haven't tried it, but I also wonder if I doubled the chunk size to 128k if I would see improved cache performance up to 120.0 GB instead of 60.0 GB. – James Oct 14 '18 at 3:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.