3

I've got two physical devices, a fast SSD and a large HDD. The SSD is in VG ssd and ends up serving as mountpoint /, the HDD is in VG dump and serves as mountpoint /home. Both go through LUKS for encryption, both are formatted as ext4. I could easily spare 40gb on the SSD to serve as a cache for the HDD and I'm wondering if I can change my LV-setup without reformatting.

Is it possible to reduce the size of the root filesystem, then reduce the size of the LV, add two new LVs for cache and metadata and attach the newly created cache-pool to serve the HDD? Remind me on how to setup in such a way, that unencrypted data does not leak to the cache...

2
  • Were you able to address your issue? Did you utilize anything in the answer below or did you take another approach?
    – JonathanS
    Apr 5, 2015 at 15:51
  • I now used a mixture of the information below to reduce the size of the SSD's filesystem, it's LUKS-container, it's LVM-device and it's partition, create a new encrypted LVM-cache device on it and add it to the HDD's volume group, effectively caching the HDD to the SSD without losing encryption. I will add a answer with all the necessary steps to my own question once I extracted them from my .bash_history Apr 7, 2015 at 16:52

2 Answers 2

3

I've now distilled enough information to answer my own question. Here are my steps for other people to follow.

Usual disclaimer: Modifying existing filesystems and their underlying devices may cause corruption and loss of a data. Plan ahead, backup your data :-)

If you follow these steps, parts of the filesystem layout I started with may not be applicable to your setup, adapt accordingly:

  • I'm using Fedora 21.
  • The SSD is on /dev/sda, provides /boot (sda1), swap (sda2) and a LUKS-container that hosts LVM-volume root in volume-group ssd for ext4-formatted / (/dev/sda3 -> /dev/mapper/luks-... -> /dev/mapper/ssd-root -> /).
  • The HDD is on /dev/sdb and provides a LUKS-container that hosts a LVM-volume dump in volume-group dump for /home (/dev/sdb1 -> /dev/mapper/luks-... -> /dev/mapper/dump-dump -> /home).
  • I will resize the /-filesystem to make room for about 7.5gb of encrypted(!) cache to serve /home.
  • Note that LVM, LUKS, the partition table and the filesystems use different block sizes (e.g. my LVM-volumes contain extents of 4mb each, my ext4-filesystems, LUKS, and the partitions are blocks/sectors of 512 bytes each). Chose your sizes so that the alignments match.
  • Double-check all numbers you enter manually for consistency. Triple-check that you don't end up with a filesystem that is larger than the logical volume, a logical volume that is larger than the physical volume, a physical volume that is larger than the LUKS-container or a LUKS-container that is larger than the partition.

  1. Use lsblk to remind yourself about your own setup again. Reboot to a LiveCD and open a new terminal session. Switch to root if you want to avoid all the calls to sudo...

  2. We will need the Fedora-usual names of the LUKS-containers several times. Put them in variables first

    SDA3UUID=luks-`sudo cryptsetup luksUUID /dev/sda3`
    SDB1UUID=luks-`sudo cryptsetup luksUUID /dev/sdb1`
    

    $SDA3UUID now holds the UUID for the /-device, $SDB1UUID is /home.

    Open the /-device using LUKS. LVM will discover the volume inside the container and make it available (as /dev/mapper/ssd-root in my case).

    sudo cryptsetup open /dev/sda3 $SDA3UUID
    
  3. Reduce the filesystems's size.

    resize2fs enforces a filesystem-check first. I will resize the filesystem for / to 20 gigabytes.

    sudo fsck -f /dev/mapper/ssd-root
    sudo resize2fs -p /dev/mapper/ssd-root 20g
    

    This process is going to take a while. Note the output "the filesystem is now XYZ blocks long" and the size of each block (5242880 blocks of 4k each in my case).

  4. Deactivate and reduce the size of the logical volume that backs the filesystem.

    The -L-parameter is given in megabytes by default.

    sudo lvchange -an /dev/mapper/ssd-root
    sudo lvreduce -L 20480 /dev/mapper/ssd-root
    
  5. Reduce the size of the physical volume that backs the logical volume.

    Note that there is some overhead for LUKS and LVM, so the physical volume should remain a little larger than the logical volume. I added two extents.

    sudo pvresize --setphysicalvolumesize 20488 /dev/mapper/$SDA3UUID
    
  6. Reduce the size of the LUKS-container.

    The -b-parameter is given in blocks. Use pvdisplay and calculate as total numbers of physical extents * size of extent * 1024**2 / 512 (e.g. python -c "print 5121*4*1024**2/512")

    sudo cryptsetup resize -b 41951232 $SDA3UUID
    

    Use sudo cryptsetup status $SDA3UUID to find out about the container's new size. In my case, the container is 2mb larger than the physical volume and filesystem it contains (there goes your alignment).

  7. Reduce the size of the partition and create a new partition for the cache.

    Deactivate the logical volume and the container first.

    sudo cryptsetup luksClose $SDA3UUID
    sudo fdisk /dev/sda
    

    Delete the partition that used to be be sda3. Create a new partition that starts on the same sector as the old one (!) and which is "+XYZ" in size ("+41951232" in my case). Also create a new partition that fills the device (7.5gb in my case). Write the partition table to disk and exit fdisk.

    Check that the filesystem is still alive:

    sudo cryptsetup open /dev/sda3 $SDA3UUID
    

    lvs should show the logical volume.

    sudo fsck -f /dev/mapper/ssd-root
    
  8. Create a new LUKS-container on the new partition to host the encrypted cache.

    sudo cryptsetup luksFormat /dev/sda4
    SDA4UUID=luks-`sudo cryptsetup luksUUID /dev/sda4`
    
  9. Open the HDD's LUKS-container and extend the volume group that ends up hosting /home with the new device.

    sudo cryptsetup open /dev/sdb1 $SDB1UUID
    

    lvs should show the volume group dump being active now.

    sudo cryptsetup open /dev/sda4 $SDA4UUID
    sudo vgextend dump /dev/mapper/$SDA4UUID
    
  10. Create a logical volume for the the cache's metadata and the cache itself.

    It is said that the metadata-volume should be 0.1% the size of the cache itself but 8mb at minimum. Notice again that there is some overhead, so I chose the size of the cache to be 99% of the physical volume's remaining size.

    sudo lvcreate -L 20M -n lv_cache_meta dump /dev/mapper/$SDA4UUID
    sudo lvcreate -l 99%FREE -n lv_cache dump /dev/mapper/$SDA4UUID
    sudo lvconvert --type cache-pool --poolmetadata dump/lv_cache_meta dump/lv_cache
    
  11. Add the cache to the logical volume that hosts /home.

    sudo lvconvert --type cache --cachepool dump/lv_cache dump/dump
    

    The output should be something to the sound of "Logical volume dump/dump is now cached."

  12. Update crypttab so the new LUKS-container is opened during boot.

    mkdir root
    sudo mount /dev/mapper/ssd-root root
    sudo su
    UUID=`cryptsetup luksUUID /dev/sda4`
    echo "luks-$UUID UUID=$UUID none" >> root/etc/crypttab
    exit
    sudo umount root
    
  13. Cross your fingers and reboot into your original system.

    After reboot you can get some statistics about the cache like so:

    lvs --rows -o +devices,cache_total_blocks,cache_used_blocks,cache_dirty_blocks,cache_read_hits,cache_read_misses,cache_write_hits,cache_write_misses dump
    

And there you have it.

1

I would use lvmcache(7) which utilizes dm-cache. Nice writeup here on doing so:

https://rwmj.wordpress.com/2014/05/22/using-lvms-new-cache-feature/

This, according to the author, requires the use of a single volume group. Thus you'll need to merge your volume groups. See following link which uses vgmerge(8) to do so:

https://sites.google.com/site/rhelworldexperience/home/rhel-disk-management---tips-and-tricks/mergingvolumegroupsinrhel-vgmerge

If you have LVM on top of LUKS, I believe the caching will be unencrypted based on the explanation of the device manager found in https://help.ubuntu.com/community/EncryptedFilesystemHowto. LUKS on top of LVM I believe will cache encrypted data.

Here is a good reference for resizing when LVM is on top of LUKS: https://help.ubuntu.com/community/ResizeEncryptedPartitions. One should be able to adapt this for the other way around.

Note that it is probably best to setup a test system to verify the process on and/or do a full back with a tested/verified restore process in place.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .