cryptsetup: verification in luksOpen is non-deterministic when reading the password from a file

Question

I'm debugging a weird behavior of cryptsetup:

Assume the correct password is stored in the file pw. I expected now that --test-passphrase would always succeed (i.e. printing no output) if it is passed in as stdin. But it turns out that it randomly fails:

# cryptsetup luksOpen --test-passphrase /dev/nvme0n1p2 < pw
# cryptsetup luksOpen --test-passphrase /dev/nvme0n1p2 < pw
No key available with this passphrase.
# cryptsetup luksOpen --test-passphrase /dev/nvme0n1p2 < pw
# cryptsetup luksOpen --test-passphrase /dev/nvme0n1p2 < pw
No key available with this passphrase.
# cryptsetup luksOpen --test-passphrase /dev/nvme0n1p2 < pw
# cryptsetup luksOpen --test-passphrase /dev/nvme0n1p2 < pw
No key available with this passphrase.
# cryptsetup luksOpen --test-passphrase /dev/nvme0n1p2 < pw
# cryptsetup luksOpen --test-passphrase /dev/nvme0n1p2 < pw
# cryptsetup luksOpen --test-passphrase /dev/nvme0n1p2 < pw
No key available with this passphrase.

I noticed it, since I constantly fail multiple times to unlock my partition at boot (in GRUB). First, I thought I'm mistyping it, but now I get the impression that it might be a bug in cryptsetup. I also fail to unlock it consistently later (not in GRUB), even if I'm copy-pasting the correct password.

Note that it also differs when I pass it through this (mostly equivalent) way:

# cat pw | cryptsetup luksOpen --test-passphrase /dev/nvme0n1p2
No key available with this passphrase.
# cat pw | cryptsetup luksOpen --test-passphrase /dev/nvme0n1p2
No key available with this passphrase.
# cat pw | cryptsetup luksOpen --test-passphrase /dev/nvme0n1p2
# cat pw | cryptsetup luksOpen --test-passphrase /dev/nvme0n1p2
No key available with this passphrase.
# cat pw | cryptsetup luksOpen --test-passphrase /dev/nvme0n1p2
No key available with this passphrase.
# cat pw | cryptsetup luksOpen --test-passphrase /dev/nvme0n1p2
No key available with this passphrase.

Here all but one attempt failed. While the other approach succeeds more often. That behavior is reproducible for me: it always fails more often.

# cryptsetup --version
cryptsetup 2.6.1 flags: UDEV BLKID KEYRING KERNEL_CAPI

# cryptsetup luksDump /dev/nvme0n1p2
LUKS header information
Version:        2
Epoch:          5
Metadata area:  16384 [bytes]
Keyslots area:  16744448 [bytes]
UUID:           2372e472-ef96-428f-b971-f68fb0c35b63
Label:          (no label)
Subsystem:      (no subsystem)
Flags:          (no flags)

Data segments:
  0: crypt
    offset: 16777216 [bytes]
    length: (whole device)
    cipher: aes-xts-plain64
    sector: 512 [bytes]

Keyslots:
  0: luks2
    Key:        512 bits
    Priority:   normal
    Cipher:     aes-xts-plain64
    Cipher key: 512 bits
    PBKDF:      argon2id
    Time cost:  13
    Memory:     1048576
    Threads:    4
    Salt:       ea b0 88 ... 
                f3 f9 72 ... 
    AF stripes: 4000
    AF hash:    sha256
    Area offset:32768 [bytes]
    Area length:258048 [bytes]
    Digest ID:  0
Tokens:
Digests:
  0: pbkdf2
    Hash:       sha256
    Iterations: 334367
    Salt:       f0 ac 44 ... 
                f3 6f d5 ... 
    Digest:     cd a8 ... 
                23 2a ... 

$ uname -a
Linux amd12 6.3.2-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 11 May 2023 16:40:42 +0000 x86_64 GNU/Linux

(OS: Arch Linux)

Eventually, I can always unlock, but I need many attempts, which is annoying. Looks like the verification code or the mechanism to read the input is flaky.

I wonder if it is a known problem (though I haven't found anything about it)? If not, is there a way to debug? Unfortunately, I saw no option to get any visual feedback (I think, it is considered a security flaw to reveal the password length).

Update: Just realized that there is a --debug option. Though the output of a successful and failing run are identical till the point where the computation happens. All headers and checksums in the debug log are the same.

Also, it shows the same behavior with cryptsetup 2.4.3 on a Linux Mint Live CD.

Answer 1

@frostschutz was correct. It turns out that the memory on my machine shows errors in a Memtest86+ run.

The most likely explanation is that the computation works or fails depending on what part of the RAM is being used. And Argon2 - now the default for key derivation - uses a lot of memory during the computation.

But it is not the fault of cryptsetup. Now, I tested again with only one memory block that passed at least one run of Memtest86+. It is no longer reproducible. Both versions (< pwand cat pw |) are now always passing.

Update: I did more testing with removing RAM slots. Interestingly, it works with individual slots, but in combination it becomes unreliable. The machine itself is new, but I saw that XMP (Extreme Memory Profile) was enabled. From what I read, it is a reasonable safe default on new hardware, but on my machine it seems to cause the issues. After disabling it, it works now.

Perhaps they should include Argon2 in memtest86+. On my system, it is great to detect memory issues.