1

I think I've got a dead system now, but I don't know how to recover.

I was updating from Buster to Bookworm with the standard method outlined here - https://wiki.debian.org/DebianUpgrade

After starting the upgrade, I had the error /usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory and the upgrade stopped. Quickly searched for the solution and found this answer to resolve it, so started to execute that.

When trying to run sudo cp -av lib/x86_64-linux-gnu/* /lib/x86_64-linux-gnu/, I had a strange sudo error, something along the lines of cannot escalate, is your account locked? I don't know the exact error. I assumed a quick reboot would solve it, but this is where the problem started.

Once the computer was booted, I attempted to SSH into it and kept getting connection refused. I run this headless most of the time, so this was obviously a bad sign. I took it down from where it normally is and plugged it into a monitor to figure out what was going on.

The machine booted fine, but at the login prompt whenever I typed my username I'd get an immediate Login incorrect before even being prompted for the password. I tried root and also got Login incorrect.

I attempted to boot into debian recovery, and was met with the same error. No matter what I do, I can't log in to the system.

Here is a screenshot of the log before the login prompt.

boot log

I've attempted to boot into rescue mode from a usb drive, and I've been able to get a prompt on the system. Trying journalctl just gives me "No journal files were found."

The only thing I can find that might be of note is in /var/log/auth.log, which is;

PAM unable to dlopen(pam_unix.so): /lib/security/pam_unix.so: cannot open shared object file: No such file or directory
PAM adding faulty module: pam_unix.so
FAILED LOGIN (1) on `/dev/tty1` for `root`, Authentication Failure

Does anyone know how I can recover the system from this point? I've got absolutely no idea what to do. All I can access is the USB rescue mode prompt, so the actions I have are relatively limited.

Thanks

2
  • 1
    Buster and Bookworm are two stable releases apart (there was Bullseye in between), so if you tried upgrading directly, that may have caused the problem. Commented Jan 3 at 9:54
  • I did, that's probably a good call. Didn't occur to me that it would cause issues
    – Jademalo
    Commented Jan 4 at 14:28

2 Answers 2

2

Having just recovered from this state, this is what I did;

  • Boot using the Debian Live installer, and go into recovery.
  • Load the existing installation in recovery
  • Mount all relevant drives, in my case /var and /usr weren't mounted so that casued a few issues.
  • Following information here,

Run

cd /tmp  
apt download libcrypt1  
dpkg-deb -x libcrypt1_1%3a4.4.33-2_amd64.deb  .  
cp -av lib/x86_64-linux-gnu/* /lib/x86_64-linux-gnu/
apt --fix-broken install
  • Reboot, login now works since the broken state of PAM has been resolved
  • finish sudo apt full-upgrade

This has my system back in a fully usable, functional, and upgraded state. It seems that it broke libcrypt during the upgrade, crashed out of the upgrade because of that, and then broke login due to libcrypt being broken.

As a note, I also had an issue where the kernel didn't upgrade due to aufs-dkms, but after removing that it upgraded fine. I was still on v3.16, apparently that had been an issue for a while!

3
  • 1
    Just had the exact same problem after a dist-upgrade failed. This solution worked for me. Thank you very much! Commented Feb 13 at 10:19
  • Glad it was able to help someone!
    – Jademalo
    Commented Feb 28 at 15:18
  • Sadly this isn't an option with a cloud based server.
    – Ken Ingram
    Commented Mar 21 at 5:48
2

So, somehow your libcrypto.so doesn't work with the rest of your system. Maybe an interrupted upgrade? Maybe it was something manually patched and excluded? Hard to guess.

Anyway, libcrypt is involved in most system tools (it contains, amongst other things, password hashing routines), so all the executables that would allow you to authenticate are currently dysfunctional.

Anyways, the solution is usually to boot a Debian live USB stick, and reinstall the package containing libcrypto.so into your dysfunctional main system. This involves mounting the original system's root (say, on /target), and then doing something like

apt-get download libpam-modules libcrypt1
sudo dpkg -i --root=/target libpam-modules*.deb librypt1*.deb
1
  • You are correct about the interrupted upgrade, I've just managed to recover. Thanks for the suggestion! I've posted another answer with my solution since I was thankfully able to get a command prompt in the main system via the recovery and didn't have to do it from another.
    – Jademalo
    Commented Jan 3 at 7:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .