apt broken after trying to update and install without internet

Question

On a debian buster machine without internet access, most of apt command (like apt install/remove/autoremove) show the following error :

The following packages have unmet dependencies:
 systemd : Depends: libsystemd0 (= 241-7~deb10u6) but 241-7~deb10u8 is to be installed
           Recommends: libpam-systemd but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

When trying to run apt --fix-broken install, I get this error :

root@debian10-h1:/var/cuda-repo-debian10-11-4-local# apt --fix-broken install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
 big list of packages
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  systemd
0 upgraded, 0 newly installed, 1 to remove and 1 not upgraded.
After this operation, 13.8 MB disk space will be freed.
Do you want to continue? [Y/n] Y
(Reading database ... 114294 files and directories currently installed.)
Removing systemd (241-7~deb10u6) ...
systemd is the active init system, please switch to another before removing systemd.
dpkg: error processing package systemd (--remove):
 installed systemd package pre-removal script subprocess returned error exit status 1
Errors were encountered while processing:
 systemd
E: Sub-process /usr/bin/dpkg returned an error code (1)

How I got there

I had to install a program on a debian buster server without internet access. Up until now, when I had to install something via apt, I'd run apt-get download $(apt-cache depends --recurse --no-recommends --no-suggests --no-conflicts --no-breaks --no-replaces --no-enhances <your-package-here> | grep "^\w" | sort -u) on a VM to download the .deb files from the packages and all dependencies, transfer the files to my target machine and then run dpkg -i *.deb to install there, which was working fine.

I had to install CUDA, so I followed the nvidia installation guide. At some point, I'm supposed to run those command :

wget https://developer.download.nvidia.com/compute/cuda/11.4.0/local_installers/cuda-repo-debian10-11-4-local_11.4.0-470.42.01-1_amd64.deb
sudo dpkg -i cuda-repo-debian10-11-4-local_11.4.0-470.42.01-1_amd64.deb
sudo apt-key add /var/cuda-repo-debian10-11-4-local/7fa2af80.pub
sudo add-apt-repository contrib
sudo apt-get update
sudo apt-get -y install cuda

I did the first 4 commands without any problem (the first on my VM with internet connection), but since I didn't have an internet connection, the 5th one failed :

root@debian10-h1:/home/user/installation# apt-get update
Get:1 file:/var/cuda-repo-debian10-11-4-local  InRelease
Ign:1 file:/var/cuda-repo-debian10-11-4-local  InRelease
Get:2 file:/var/cuda-repo-debian10-11-4-local  Release [564 B]
Get:2 file:/var/cuda-repo-debian10-11-4-local  Release [564 B]
Err:4 http://deb.debian.org/debian buster InRelease
  Temporary failure resolving 'deb.debian.org'
Reading package lists... Done
W: Failed to fetch http://deb.debian.org/debian/dists/buster/InRelease  Temporary failure resolving 'deb.debian.org'
W: Some index files failed to download. They have been ignored, or old ones used                                                                                                                                                              instead.

and the 6th too :

root@debian10-h1:/home/user/installation# apt-get -y install cuda
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 apt-utils : Depends: apt (= 1.8.2.2) but 1.8.2.3 is to be installed
 cuda : Depends: cuda-11-4 (>= 11.4.0) but it is not going to be installed
 libglib2.0-bin : Depends: libglib2.0-0 (= 2.58.3-2+deb10u2) but 2.58.3-2+deb10u
3 is to be installed
 libpython3.7 : Depends: libpython3.7-stdlib (= 3.7.3-2+deb10u2) but 3.7.3-2+deb10u3 is to be installed
 systemd : Depends: libsystemd0 (= 241-7~deb10u6) but 241-7~deb10u8 is to be installed
 udev : Depends: libudev1 (= 241-7~deb10u6) but 241-7~deb10u8 is to be installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

So I tried apt --fix-broken install :

root@debian10-h1:/home/user/installation# apt --fix-broken install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  big list of packages
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  big list of packages
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  init systemd-sysv (due to init)
0 upgraded, 0 newly installed, 58 to remove and 1 not upgraded.
After this operation, 364 MB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
?]n

I tried to do apt autoremove as suggested but I got the same E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). as before. Same with apt remove cuda. Thinking that it didn't work because of the problem during apt-get update, I tried to do the update via apt-offline, but once again was asked to use apt --fix-broken install :

root@debian10-h1:/home/user/installation# apt-offline set ./my.sig

Generating database of files that are needed for an update.

Generating database of file that are needed for operation upgrade
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
ERROR: FATAL: Something is wrong with the APT system

Since I didn't find anything else to do, I retried apt --fix-broken install but this time accepting to continue :

root@debian10-h1:/home/user/installation# apt --fix-broken install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  big list of packages
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  big list of packages
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  init systemd-sysv (due to init)
0 upgraded, 0 newly installed, 58 to remove and 1 not upgraded.
After this operation, 364 MB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
?]Yes, do as I say!
(Reading database ... 120959 files and directories currently installed.)

Lot of others thing were removed, I just kept the ones where I got a different message, can post the full list if needed.

Removing linux-image-amd64 (4.19+105+deb10u9) ...
Removing linux-image-4.19.0-14-amd64 (4.19.171-2) ...
W: Removing the running kernel
W: Last kernel image has been removed, so removing the default symlinks
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-4.19.0-14-amd64
/etc/kernel/postrm.d/zz-update-grub:
Generating grub configuration file ...
Found background image: /usr/share/images/desktop-base/desktop-grub.png
Adding boot menu entry for EFI firmware configuration
done
update-initramfs: deferring update (trigger activated)
Removing initramfs-tools (0.133+deb10u1) ...
Removing initramfs-tools-core (0.133+deb10u1) ...
Removing systemd (241-7~deb10u6) ...
systemd is the active init system, please switch to another before removing systemd.
dpkg: error processing package systemd (--remove):
 installed systemd package pre-removal script subprocess returned error exit status 1
Errors were encountered while processing:
 systemd
E: Sub-process /usr/bin/dpkg returned an error code (1)

What I already tried

• Followed https://askubuntu.com/a/1093253 : sudo apt-get clean don't show anything,

  root@debian10-h1:/home/user# dpkg --configure -a
  Processing triggers for mime-support (3.62) ...
  Processing triggers for hicolor-icon-theme (0.17-2) ...
  Processing triggers for libglib2.0-0:amd64 (2.58.3-2+deb10u3) ...
  Processing triggers for libc-bin (2.28-10) ...
  Processing triggers for man-db (2.8.5-2) ...
  Processing triggers for libreoffice-common (1:6.1.5-3+deb10u6) ...
  Processing triggers for dbus (1.12.20-0+deb10u1) ...
  Processing triggers for desktop-file-utils (0.23-4) ...
  

and sudo apt upgrade --fix-missing ask me to do apt --fix-broken install again.

• Followed https://askubuntu.com/a/1244984, but stat / /dev /var show that root is the owner of all 3 directories.

• Followed https://askubuntu.com/a/380701, but sudo fuser -v /var/cache/debconf/config.dat show nothing. (+ I don't think it's exactly the same problem).

I was wondering if anyone knew how to solve that problem, I can provide more informations if needed.

Answer 1

I’ll explain how the system ended up in this situation (at least, my theory), and how to avoid it in future; but first, let’s see how to fix it.

I suspect that your setup is too broken for apt-offline to be able to fix things in a reasonable number of steps. You would be better off downloading the first full DVD of the current point release, and using that. To do so, remove all the configured repositories, mount the DVD image on your server, and add it using apt-cdrom, with the --no-mount option so that apt-cdrom doesn’t try to manage the “DVD drive” itself (assuming you use an image).

This will provide you with an up-to-date source of packages which you can use to re-install the removed packages and upgrade the system so it’s in a consistent state. I don’t have a magic recipe for that; you’ll have to figure out what packages were removed. apt full-upgrade might help.

As for why the system ended up broken like this, I don’t think it’s related to CUDA, but rather to the technique you use for off-line installations. Downloading full dependencies trees, ignoring what’s already installed, is bound to end up downloading package sets which intersect with other dependencies trees which should be upgraded together. For example, many packages end up depending on libsystemd0, but their dependencies trees don’t include systemd; installing such a package set results in a mismatched systemd and libsystemd0, which explains the first error in your question. dpkg -i can allow such mismatches to occur, but apt doesn’t like them and they need to be fixed before it can do anything sensible; this is what apt install --fix-broken repairs, but it needs a package repository...

To avoid this, the installed set of packages has to be kept globally consistent. For a disconnected system, apt-offline is the best approach, but it has to be used all the time, without apt-cache depends tricks. For a partially-connected system, i.e. one which is on a network but without direct Internet access, a network-local package cache can be a better solution; see for example apt-cacher-ng.