Open: cryptsetup out of memory ("Not enough available memory to open a keyslot.")

Question

I am working on my yocto distribution including cryptsetup in the 2.3.2 version

I am running such distribution on a board with 1 GB RAM and I am incurring in an "out of memory" error trying to open an encrypted partition that I am not able to properly debug. Any ideas?

My distro runs from an mSD with 3 partitions; the third one (30 MB) is the encrypted one.

I used the steps described on the ArchLinux guide to encrypt that partition, with ext3 instead of ext4

# cryptsetup -y -v luksFormat /dev/sda2
# cryptsetup open /dev/sda2 cryptroot
# mkfs.ext3 /dev/mapper/cryptroot

But trying to open that partition on my board raises an error:

cryptsetup --debug open /dev/mmcblk0p3  cryptroot
# cryptsetup 2.3.2 processing "cryptsetup --debug open /dev/mmcblk0p3 cryptroot"
# Running command open.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device /dev/mmcblk0p3.
# Trying to open and read device /dev/mmcblk0p3 with direct-io.
# Initialising device-mapper backend library.
# Trying to load any crypt type from device /dev/mmcblk0p3.
# Crypto backend (OpenSSL 1.1.1k  25 Mar 2021) initialized in cryptsetup library version 2.3.2.
# Detected kernel Linux 4.1.35-rt41 ppc.
# Loading LUKS2 header (repair disabled).
# Acquiring read lock for device /dev/mmcblk0p3.
# Opening lock resource file /run/cryptsetup/L_179:3
# Verifying lock handle for /dev/mmcblk0p3.
# Device /dev/mmcblk0p3 READ lock taken.
# Trying to read primary LUKS2 header at offset 0x0.
# Opening locked device /dev/mmcblk0p3
# Veryfing locked device handle (bdev)
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:43e122216ab19330fdfb6d2f9d7b586c4e5189884aef24be884e7159228e9ee5 (on-disk)
# Checksum:43e122216ab19330fdfb6d2f9d7b586c4e5189884aef24be884e7159228e9ee5 (in-memory)
# Trying to read secondary LUKS2 header at offset 0x4000.
# Reusing open ro fd on device /dev/mmcblk0p3
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:4ed9a44c22fde04c4b59a638c20eba6da3a13e591a6a1cfe7e0fec4437dc14cc (on-disk)
# Checksum:4ed9a44c22fde04c4b59a638c20eba6da3a13e591a6a1cfe7e0fec4437dc14cc (in-memory)
# Device size 32505856, offset 16777216.
# Device /dev/mmcblk0p3 READ lock released.
# Only 1 active CPUs detected, PBKDF threads decreased from 4 to 1.
# Not enough physical memory detected, PBKDF max memory decreased from 1048576kB to 255596kB.
# PBKDF argon2i, time_ms 2000 (iterations 0), max_memory_kb 255596, parallel_threads 1.
# Activating volume cryptroot using token -1.
# Interactive passphrase entry requested.
Enter passphrase for /dev/mmcblk0p3:
# Activating volume cryptroot [keyslot -1] using passphrase.
device-mapper: ioctl: 4.31.0-ioctl (2015-3-12) initialised: [email protected]
# dm version   [ opencount flush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# Detected dm-ioctl version 4.31.0.
# Device-mapper backend running with UDEV support enabled.
# dm status cryptroot  [ opencount noflush ]   [16384] (*1)
# Keyslot 0 priority 1 != 2 (required), skipped.
# Trying to open LUKS2 keyslot 0.
# Keyslot 0 (luks2) open failed with -12.
Not enough available memory to open a keyslot.
# Releasing crypt device /dev/mmcblk0p3 context.
# Releasing device-mapper backend.
# Closing read only fd for /dev/mmcblk0p3.
# Unlocking memory.
Command failed with code -3 (out of memory).

Answer 1

Vojtech Trefny already explained that this happens because of Argon2i requiring too much memory. Indeed, from man cryptsetup:

For PBKDF2, only time cost (number of iterations) applies. For Argon2i/id, there is also memory cost (memory required during the process of key derivation) and parallel cost (number of threads that run in parallel during the key derivation).

But, as long as you have access to hardware with sufficient memory¹, it is overkill to re-create the device.

Instead, using a computer with sufficient memory, just add a key to your encrypted device with pbkdf2,

cryptsetup luksAddKey -S 1 --pbkdf pbkdf2 /dev/sdxy

which assumes that the key slot 1 is free (you can find free key slots by inspecting cryptsetup luksDump /dev/sdxy).

Then, in your less powerful computer, unlock the device with

cryptsetup luksOpen -S 1 /dev/sdxy name

The -S 1 is essential, otherwise the more expensive key may be tried and the OOM killer triggered all the same.

^{1: Remember that you can always keep around a bootable USB with a minimal shell and core utilities (a Debian ISO cut it for me), and you will be able to boot it in most computers and access your encrypted volume.}

Answer 2

LUKS2 uses Argon2i key derivation function which is memory-hard -- meaning it requires a lot of memory to open the device to prevent (or at least make it harder) brute force attacks using GPUs. You can check how much memory you need to open your device using cryptsetup luksDump /dev/sda2, look for the line Memory: 755294 under Keyslots.

When creating the device, cryptsetup checks how much memory is available and adjusts the amount required for opening it accordingly, but if you did create the LUKS device from a different computer (for example when formatting the SD card on a desktop) or even on the same machine with more memory available, it's possible you simply don't have enough memory now. And we are talking only about RAM, swap is not used in this case.

I recommend re-creating the LUKS device with --pbkdf pbkdf2 to switch to the "old" (used to be default in LUKS1) key derivation function PBKDF2 which doesn't use extra memory. Alternatively you can also use --pbkdf-memory <num> to force lower amount of memory for the default Argon2i.

Answer 3

In addition to the top answer I wanted to drop this here. Using a machine with more memory you can reduce the memory requirements on the existing key with the following. Memory requirements are in KB. This is useful if you already have data on the disk.

cryptsetup luksConvertKey --key-file /path/to/keyfile --pbkdf-memory=10240 /dev/<device>

Answer 4

Ok, I just got into this same problem when trying to encrypt my recently acquired VPS. Actually in my case a got the same "Out of Memory" when running # cryptsetup luksFormat /dev/<Partition> for solving I recreated the partition table as follow: made a | /dev/sda1 ( EXT4, BOOT ) | /dev/sda2 ( LUKS, ROOT and SWAP ) | /dev/sda3 ( SWAP ) |.

There may be a way to change some config when making LUKS partition, but probably would get into a unsafer, even that just little bit.

Resulting disk partition table

Partition Table	/dev/sda1	/dev/sda2	/dev/sda3
GPT	boot	luks	swap
LVM	n/a	swap / root	n/a

So I've turned SWAP on over /dev/sda3 (#swapon /dev/sda3). Then run the command: cryptsetup luksFormat /dev/sda2 successfully .

Inside LUKS I decided to create a SWAP and later EXT4/ROOT. First I was thinking to place after SWAP but got into that, after I delete /dev/sda2 and expand LUKS, I would have to move the SWAP to the end of LUKS partition and then increase ROOT one. Placing SWAP at LUKS begin, I just have to increase root partition.

From now I'm understanding that I'll not get OOM without swap. I think I need the swap just for creating luks.

I already have system configured, my next steps to do delete /dev/sda3, expand /dev/sda2 over the free space left from /dev/sda3. Increase root LVM partition.