How to clone your hard drive (SSD or HDD) and expand your LUKS-encrypted partition to fill the new full disk space:
All I was missing were these 2 commands:
sudo e2fsck -f /dev/mapper/system-root
sudo resize2fs -p /dev/mapper/system-root
I found this problem quite challenging, but here are the steps I just took tonight (25 Oct. 2021) to get it done.
The steps below were determined by looking here: https://help.ubuntu.com/community/ResizeEncryptedPartitions#Detailed_resizing_.2BAH4_Enlarging_an_encrypted_partition, as well as by looking at most of the other answers here, as well as by doing a lot of trial and error and just figuring things out.
Tested using an Ubuntu 20.04.3 LTS live USB image, created using mkusb, as described here: https://www.howtogeek.com/howto/14912/create-a-persistent-bootable-ubuntu-usb-flash-drive/.
Hardware I used:
- PNY CS2130 2TB M.2 PCIe NVMe Gen3 x4 Internal Solid State Drive (SSD), Read up to 3,500 - M280CS2130-2TB-RB, $200
- Sabrent USB 3.2 Type-C Tool-Free Enclosure for M.2 PCIe NVMe and SATA SSDs (EC-SNVE), $27
TLDR summary:
While running Linux from an external live USB, do the following:
- Carefully clone your disk using
ddrescue (see below for full details).
- Use gparted to unlock your LUKS-encrypted partition and grow (resize) it. Leave ~10% of the space at the end of the SSD unpartitioned for over-provisioning.
- With the now-unlocked/decrypted file system NOT mounted, run
e2fsck to check and fix the disk. In the command below, system-root is what I see as my most-logical option when I type sudo e2fsck -f /dev/mapper/ and then press Tab Tab to have it list out all valid auto-complete options.:
sudo e2fsck -f /dev/mapper/system-root
- Now run the
resize2fs command to expand the unlocked file system to fill the full size of your newly-expanded LUKS-encrypted partition it is in:
sudo resize2fs -p /dev/mapper/system-root
- Mount your unlocked and newly-resized LUKS partition file system to ensure that it actually grew
cd ~
mkdir mnt
sudo mount /dev/mapper/system-root mnt
df -h # look for the new size of the `/dev/mapper/system-root` file system
# optionally unmount
sudo umount ~/mnt
- Reboot and use your new disk as normal.
df -h will now show your expanded size! Everything will work exactly as it did before you cloned your drive!
Details and full steps:
How to clone your hard drive (SSD or HDD) and expand your LUKS-encrypted partition to fill the new full disk space:
1. clone your disk
I'm assuming you've just cloned your disk to a larger drive using dd or ddrescue, as I explain in my steps on my personal website article here: https://www.electricrcaircraftguy.com/2018/01/how-to-clone-your-hard-drive.html.
In short, while booted onto an Ubuntu live USB flash drive (make one with mkusb), do:
# Install GNU ddrescue (gddrescue)
sudo add-apt-repository universe
sudo apt update
sudo apt install gddrescue
# Do the clone.
# WARNING WARNING WARNING: do NOT get these drives backwards or you will
# WIPE ALL YOUR DATA ON YOUR ORIGINAL DISK!!
# A 500 GB clone will take:
# 1. ~3~6 hrs using an external spinning USB 2.0 HDD
# 2. ~2.5~3 hrs using an external spinning USB 3.0 HDD
# 3. ~20 minutes using a brand new m.2 SSD in a USB C external enclosure
time sudo ddrescue -f -n /dev/FROM_DRIVE /dev/TO_DRIVE dd_rescue.log
# Try to copy bad sectors 3 more times each; takes a few seconds usually
time sudo ddrescue -d -f -r3 /dev/FROM_DRIVE /dev/TO_DRIVE dd_rescue.log
Example output of the first clone cmd above:
ubuntu@ubuntu:~$ time sudo ddrescue -f -n /dev/sda /dev/sdb dd_rescue.log
GNU ddrescue 1.23
Press Ctrl-C to interrupt
ipos: 512110 MB, non-trimmed: 0 B, current rate: 558 MB/s
opos: 512110 MB, non-scraped: 0 B, average rate: 410 MB/s
non-tried: 0 B, bad-sector: 0 B, error rate: 0 B/s
rescued: 512110 MB, bad areas: 0, run time: 20m 46s
pct rescued: 100.00%, read errors: 0, remaining time: n/a
time since last successful read: n/a
Finished
real 21m2.331s
user 0m6.264s
sys 8m24.021s
2. Use gparted to grow your partition
While still running Ubuntu from its installation media (a live USB probably), open gparted, which is the GNU Partition Editor GUI. Choose your external /dev/TO_DRIVE in the drop-down menu on the top right. Ex: mine is /dev/sdb. You'll see something like this in the destination disk after doing the clone:
Notice that ddrescue automatically cloned the partitions too, since it does a direct byte-for-byte copy, which is a lower-level copy than the file system even.
As you can see above, /dev/sdb3 shows that it is [Encrypted]. That is what a LUKS-encrypted partition looks like in gparted.
Right-click the LUKS-locked partition and go to Open Encryption:
Type in your LUKS password to unlock it:
That partition's name will now change from [Encrypted] to [Encrypted] lvm2 pv, indicating it is unlocked. The mount point will now show system, as shown here:
You can optionally right-click the partition and go to "Information" for more information.
Next, Right-click the partition and go to Resize/Move.
As shown below, drag the right-side of the graphical partition box to the right to grow the partition. If this is an ssd, leave about 10% of the total disk space UNUSED on the far right, for dynamic over-provisioning. See here for what that means:
- https://en.wikipedia.org/wiki/Write_amplification#Over-provisioning
- https://www.seagate.com/tech-insights/ssd-over-provisioning-benefits-master-ti/
Essentially, it improves SSD read and write speed and reduces wear, as the SSD will automatically use this unpartitioned space as a sort of SSD-firmware-level swap space.
Click "Resize/Move" when done, then click the green check mark at the top of the gparted window to apply these changes (hovering over the green check mark shows "Apply All Operations").
If you look at the gparted output details, you'll see it conducted the following steps to grow the partition (this screenshot is from the output HTML details if you save them):
Notice it ran cryptsetup -v resize 'sdb3_crypt' and lvm pvresize -v '/dev/mapper/sdb3_crypt' for you.
IMPORTANT: Once resizing the partition completes, if at any point from here on out you see this warning pop up, click the "Fix" button:
Libparted Warning
Not all of the space available to /dev/sdb appears to be used, you can fix the GPT to use all of the space (an extra 2906813952 blocks) or continue with the current setting?
Fix
Ignore
Now that you resized the partition, you'll see it has grown, as shown here:
If you mount this unlocked LUKS partition at this time (I have mount instructions further below if you're curious), however, and run df -h to see the disk usage, it will show that the mounted and decrypted disk has NOT yet grown. We'll fix that next.
3. Change out the disks
Assuming you want your cloned disk to be your new internal disk, make that hardware change now. Turn off the computer (shutdown from the live USB), take apart the computer, remove the old drive, and put in the newly-cloned drive as the primary drive inside the computer.
If you choose not to do that at this time, you should change the GUID or whatever its called of the new disk so it doesn't match the old disk (since you just did an exact clone). I don't remember those details so I'll let you Google if you really need to use both the old disk and the new disk at the same time in the same computer. If you're like me, however, and you're removing the old disk entirely, just continue on to the next step.
4. Grow the decrypted file system to fit the newly-resized partition it is in
With your newly-cloned drive now in your computer as the main disk, and the original drive NOT plugged in at all, boot back onto your live USB.
Again, open gparted to decrypt/unlock the drive, as described previously above:
Right-Click the LUKS-encrypted partition --> Open Encryption --> type password.
Now run this, where system-root is what I see as my most-logical option when I type sudo e2fsck -f /dev/mapper/ and then press Tab Tab to have it list out all valid auto-complete options.
# WARNING WARNING WARNING: do NOT run this cmd while this partition is mounted!
# It says that file system corruption WILL occur.
sudo e2fsck -f /dev/mapper/system-root
This e2fsck step is REQUIRED to be run before the resize2fs step, which comes next. e2fsck checks and fixes the newly-cloned and decrypted (unlocked) file system. Example output:
ubuntu@ubuntu:~$ sudo e2fsck -f /dev/mapper/system-root
e2fsck 1.45.5 (07-Jan-2020)
Pass 1: Checking inodes, blocks, and sizes
Inode 27012476 extent tree (at level 2) could be narrower. Optimize<y>? yes
Inode 27013750 extent tree (at level 2) could be narrower. Optimize<y>? yes
Inode 27191704 extent tree (at level 2) could be narrower. Optimize<y>? yes
Pass 1E: Optimizing extent trees
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/system-root: ***** FILE SYSTEM WAS MODIFIED *****
/dev/mapper/system-root: 4106785/31121408 files (0.5% non-contiguous), 114717248/124455936 blocks
Now run the resize2fs command to expand the unlocked file system to fill the full size of your newly-expanded LUKS-encrypted partition it is in:
# This command can be run with this partition mounted, but if you're following
# my instructions exactly right now, it will *not* be mounted at this time.
sudo resize2fs -p /dev/mapper/system-root
Example output if you did have it mounted when you ran it:
ubuntu@ubuntu:~$ sudo resize2fs -p /dev/mapper/system-root
resize2fs 1.45.5 (07-Jan-2020)
Filesystem at /dev/mapper/system-root is mounted on /home/ubuntu/mnt; on-line resizing required
old_desc_blocks = 60, new_desc_blocks = 211
The filesystem on /dev/mapper/system-root is now 441932800 (4k) blocks long.
Now the file system is expanded! Next we will mount it and run df -h to prove it.
5. Mount the decrypted (unlocked) partition and run df -h to prove its file system is expanded
We are still running in the live USB, and we have already unlocked the LUKS partition with our password using gparted. Let's mount it so we can read and access it if we want!
cd ~
mkdir mnt
# Mount `/dev/mapper/system-root` to the local ~/mnt directory
sudo mount /dev/mapper/system-root mnt
Now run df -h and you'll see this. Notice my disk is only 28% used now, with 1.2 TB available!
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root 1.7T 430G 1.2T 28% /home/ubuntu/mnt
Previously it was 97% used with only 14 GB available!
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root 467G 430G 14G 97% /home/ubuntu/mnt
Optionally, unmount the file system when done with:
sudo umount ~/mnt
6. Reboot and use your new disk as normal
Shut down the live USB and unplug it from your computer. Turn your computer back on to boot from the newly-cloned and now-expanded new internal drive. Type in your LUKS password when prompted, as an early part of the boot process. Log in like normal. You'll see your computer operating system and files are all intact and everything operates as if nothing was changed!
df -h, however, shows your new root (/) mount point has much more space!:
new df -h:
Notice I am 28% used with 1.2 TB available!
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root 1.7T 430G 1.2T 28% /
old df -h:
Notice I am 97% used with only 14 GB available!
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root 467G 430G 14G 97% /
Extra notes
To use gparted to "eject and remove" a LUKS file system, after unlocking it with your password, do the following:
- In gparted, right-click the partition and go to --> Deactivate.
- Then, right-click the partition again and go to "Close Encryption".
- Now you can safely unplug it, as far as I know.
I don't know what happened or why, but somewhere along the way gparted went from showing the true partition usage when unlocked to NOT showing the true partition usage when unlocked. It now shows full 100% usage even when unlocked. I think I did something weird. See the image below. Everything seems to work ok though, so I guess I can ignore it? If anyone knows why it does NOT show my true disk usage for this partition anymore, please let me know!
Keywords: expand encrypted LUKS partition: how to clone your Ubuntu OS or hard drive and expand your LUKS-encrypted hard drive / file system when done! ssd over-provisioning; clone ssd; clone hdd
