3

Apparently on a production server there has been some problem and someone has deleted the contents of the /var folder.

This has caused several errors that I have been solving with various services as a web server.

The problem I'm having with apt, problem that doesn't allow me to update, remove or install packages.

for example I want to reinstall the database server mariadb-server, but it tells me that the package is not installed in the system (this is false, i installed it personally):

root# apt remove mariadb-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package 'mariadb-server' is not installed, so not removed
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
 chkconfig : Depends: perl but it is not going to be installed
             Recommends: insserv but it is not going to be installed
 libboost-chrono1.50.0 : Depends: libgcc1 (>= 1:4.4.0) but it is not going to be installed
                         Depends: libstdc++6 (>= 4.4.0) but it is not going to be installed
 libboost-system1.50.0 : Depends: libgcc1 (>= 1:4.4.0) but it is not going to be installed
                         Depends: libstdc++6 (>= 4.4.0) but it is not going to be installed
 libboost-thread1.50.0 : Depends: libgcc1 (>= 1:4.4.0) but it is not going to be installed
                         Depends: libstdc++6 (>= 4.6) but it is not going to be installed
 libc6 : Depends: libgcc1 but it is not going to be installed
 libprotobuf-lite7 : Depends: libgcc1 (>= 1:4.4.0) but it is not going to be installed
                     Depends: libstdc++6 (>= 4.4.0) but it is not going to be installed
                     Depends: zlib1g (>= 1:1.1.4) but it is not going to be installed
 oracle-java8-jdk : Depends: libasound2 (>= 1.0.16)
                    Depends: libgcc1 (>= 1:4.4.0) but it is not going to be installed
                    Depends: libx11-6 but it is not going to be installed
                    Depends: libxext6 but it is not going to be installed
                    Depends: libxi6 but it is not going to be installed
                    Depends: libxrender1 but it is not going to be installed
                    Depends: libxtst6 but it is not going to be installed
                    Recommends: netbase but it is not going to be installed
 redis-server : Depends: init-system-helpers (>= 1.18~) but it is not going to be installed
                Depends: libjemalloc1 (>= 2.1.1) but it is not going to be installed
                Depends: adduser but it is not going to be installed
 redis-tools : Depends: libjemalloc1 (>= 2.1.1) but it is not going to be installed
 watchdog : Depends: debconf (>= 0.5) but it is not going to be installed or
                     debconf-2.0
            Depends: init-system-helpers (>= 1.18~) but it is not going to be installed
            Depends: makedev (>= 2.3.1-24) but it is not going to be installed or
                     udev but it is not going to be installed
            Depends: lsb-base (>= 3.2-14) but it is not going to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).

if I launch the command that tells me to resolve unmet dependencies:

root# apt-get -f install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following extra packages will be installed:
  adduser apt apt-utils base-passwd ca-certificates coreutils debconf debconf-i18n debianutils dpkg e2fslibs e2fsprogs gnupg gnupg-curl gpgv ifupdown init-system-helpers initscripts insserv iproute2 isc-dhcp-client isc-dhcp-common
  krb5-locales libacl1 libalgorithm-c3-perl libapt-inst1.5 libapt-pkg4.12 libarchive-extract-perl libasound2 libasound2-data libatm1 libattr1 libaudit-common libaudit1 libblkid1 libbz2-1.0 libcgi-fast-perl libcgi-pm-perl
  libclass-c3-perl libclass-c3-xs-perl libcomerr2 libcpan-meta-perl libcurl3-gnutls libdata-optlist-perl libdata-section-perl libdb5.3 libdebconfclient0 libdns-export100 libfcgi-perl libffi6 libgcc1 libgcrypt20 libgdbm3 libgmp10
  libgnutls-deb0-28 libgpg-error0 libgpm2 libgssapi-krb5-2 libhogweed2 libidn11 libirs-export91 libisc-export95 libisccfg-export90 libjemalloc1 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libldap-2.4-2 liblocale-gettext-perl
  liblog-message-perl liblog-message-simple-perl liblzma5 libmodule-build-perl libmodule-pluggable-perl libmodule-signature-perl libmount1 libmro-compat-perl libncurses5 libnettle4 libp11-kit0 libpackage-constants-perl libpam-modules
  libpam-modules-bin libpam0g libparams-util-perl libpcre3 libpng12-0 libpod-latex-perl libpod-readme-perl libreadline6 libregexp-common-perl librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libselinux1 libsemanage-common
  libsemanage1 libsepol1 libslang2 libsmartcols1 libsoftware-license-perl libss2 libssh2-1 libssl1.0.0 libstdc++6 libsub-exporter-perl libsub-install-perl libsystemd0 libtasn1-6 libterm-ui-perl libtext-charwidth-perl libtext-iconv-perl
  libtext-soundex-perl libtext-template-perl libtext-wrapi18n-perl libtinfo5 libusb-0.1-4 libustr-1.0-1 libuuid1 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxext6 libxi6 libxrender1 libxtables10 libxtst6 lsb-base makedev netbase
  openssl passwd perl perl-base perl-modules psmisc raspbian-archive-keyring readline-common redis-server redis-tools rename sensible-utils startpar sysv-rc sysvinit-utils tar tzdata util-linux uuid-runtime x11-common zlib1g
Suggested packages:
  aptitude synaptic wajig dpkg-dev apt-doc python-apt debconf-doc debconf-utils whiptail dialog gnome-utils libterm-readline-gnu-perl libgtk2-perl libnet-ldap-perl libqtgui4-perl libqtcore4-perl gpart parted fuse2fs e2fsck-static
  gnupg-doc libpcsclite1 parcimonie xloadimage imagemagick eog ppp rdnssd net-tools bootchart2 iproute2-doc resolvconf avahi-autoipd libasound2-plugins alsa-utils rng-tools gnutls-bin gpm krb5-doc krb5-user libpam-doc
  libsasl2-modules-otp libsasl2-modules-ldap libsasl2-modules-sql libsasl2-modules-gssapi-mit libsasl2-modules-gssapi-heimdal perl-doc make libb-lint-perl libcpanplus-dist-build-perl libcpanplus-perl libfile-checktree-perl
  libobject-accessor-perl readline-doc bum bootlogd sash bzip2 ncompress xz-utils tar-scripts dosfstools kbd console-tools util-linux-locales
Recommended packages:
  libarchive-tar-perl
The following NEW packages will be installed:
  adduser apt apt-utils base-passwd ca-certificates coreutils debconf debconf-i18n debianutils dpkg e2fslibs e2fsprogs gnupg gnupg-curl gpgv ifupdown init-system-helpers initscripts insserv iproute2 isc-dhcp-client isc-dhcp-common
  krb5-locales libacl1 libalgorithm-c3-perl libapt-inst1.5 libapt-pkg4.12 libarchive-extract-perl libasound2 libasound2-data libatm1 libattr1 libaudit-common libaudit1 libblkid1 libbz2-1.0 libcgi-fast-perl libcgi-pm-perl
  libclass-c3-perl libclass-c3-xs-perl libcomerr2 libcpan-meta-perl libcurl3-gnutls libdata-optlist-perl libdata-section-perl libdb5.3 libdebconfclient0 libdns-export100 libfcgi-perl libffi6 libgcc1 libgcrypt20 libgdbm3 libgmp10
  libgnutls-deb0-28 libgpg-error0 libgpm2 libgssapi-krb5-2 libhogweed2 libidn11 libirs-export91 libisc-export95 libisccfg-export90 libjemalloc1 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libldap-2.4-2 liblocale-gettext-perl
  liblog-message-perl liblog-message-simple-perl liblzma5 libmodule-build-perl libmodule-pluggable-perl libmodule-signature-perl libmount1 libmro-compat-perl libncurses5 libnettle4 libp11-kit0 libpackage-constants-perl libpam-modules
  libpam-modules-bin libpam0g libparams-util-perl libpcre3 libpng12-0 libpod-latex-perl libpod-readme-perl libreadline6 libregexp-common-perl librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libselinux1 libsemanage-common
  libsemanage1 libsepol1 libslang2 libsmartcols1 libsoftware-license-perl libss2 libssh2-1 libssl1.0.0 libstdc++6 libsub-exporter-perl libsub-install-perl libsystemd0 libtasn1-6 libterm-ui-perl libtext-charwidth-perl libtext-iconv-perl
  libtext-soundex-perl libtext-template-perl libtext-wrapi18n-perl libtinfo5 libusb-0.1-4 libustr-1.0-1 libuuid1 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxext6 libxi6 libxrender1 libxtables10 libxtst6 lsb-base makedev netbase
  openssl passwd perl perl-base perl-modules psmisc raspbian-archive-keyring readline-common rename sensible-utils startpar sysv-rc sysvinit-utils tar tzdata util-linux uuid-runtime x11-common zlib1g
The following packages will be upgraded:
  redis-server redis-tools
2 upgraded, 153 newly installed, 0 to remove and 3 not upgraded.
11 not fully installed or removed.
Need to get 37.1 MB of archives.
After this operation, 120 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://archive.raspberrypi.org/debian/ jessie/main libasound2-data all 1.0.28-1+rpi3 [65.3 kB]
Get:2 http://archive.raspberrypi.org/debian/ jessie/main libasound2 armhf 1.0.28-1+rpi3 [320 kB]
Get:3 http://mirrordirector.raspbian.org/raspbian/ jessie/main libgcc1 armhf 1:4.9.2-10+deb8u2 [39.5 kB]
Get:4 http://archive.raspberrypi.org/debian/ jessie/main x11-common all 1:7.7+16 [251 kB]
[...]
Get:152 http://mirrordirector.raspbian.org/raspbian/ jessie/main libterm-ui-perl all 0.42-1 [19.1 kB]
Get:153 http://mirrordirector.raspbian.org/raspbian/ jessie/main libtext-soundex-perl armhf 3.4-1+b2 [13.3 kB]
Get:154 http://mirrordirector.raspbian.org/raspbian/ jessie/main psmisc armhf 22.21-2 [117 kB]
Get:155 http://mirrordirector.raspbian.org/raspbian/ jessie/main rename all 0.20-3 [12.4 kB]
Fetched 37.1 MB in 2min 29s (249 kB/s)
Reading changelogs... Done
E: Cannot get debconf version. Is debconf installed?
debconf: apt-extracttemplates failed: No such file or directory
Extracting templates from packages: 19%E: Cannot get debconf version. Is debconf installed?
debconf: apt-extracttemplates failed: No such file or directory
Extracting templates from packages: 38%E: Cannot get debconf version. Is debconf installed?
debconf: apt-extracttemplates failed: No such file or directory
Extracting templates from packages: 58%E: Cannot get debconf version. Is debconf installed?
debconf: apt-extracttemplates failed: No such file or directory
Extracting templates from packages: 77%E: Cannot get debconf version. Is debconf installed?
debconf: apt-extracttemplates failed: No such file or directory
Extracting templates from packages: 96%E: Cannot get debconf version. Is debconf installed?
debconf: apt-extracttemplates failed: No such file or directory
Extracting templates from packages: 100%
dpkg: regarding .../libgcc1_1%3a4.9.2-10+deb8u2_armhf.deb containing libgcc1:armhf, pre-dependency problem:
 libgcc1 pre-depends on multiarch-support
  multiarch-support is unpacked, but has never been configured.

dpkg: error processing archive /var/cache/apt/archives/libgcc1_1%3a4.9.2-10+deb8u2_armhf.deb (--unpack):
 pre-dependency problem - not installing libgcc1:armhf
Errors were encountered while processing:
 /var/cache/apt/archives/libgcc1_1%3a4.9.2-10+deb8u2_armhf.deb

E: Sub-process /usr/bin/dpkg returned an error code (1)

Here is proof that mariadb-server is installed (the error is due to the removal of /var, so I want to reinstall it):

root# mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")

As you can see, both apt and mariadb-server throw me errors with missing content in the /var directory. So I've entered a loop where I don't know how to proceed to solve this problem.

  • A few quick questions, is it possible to backup your data/applications and start over? And if not, Do you have a second raspberry pi or capability to install Raspbian in a virtual machine? And if you need access to the contents of that mariadb server I hope you had backups.... – kemotep Nov 26 '19 at 15:11
  • I have no physical access to this or other similar Raspberrys, as they belong to an extensive network with specific tasks in different countries. I do have logical access (via SSH) to other similar raspys that perform the same task in other sub-networks – balon Nov 26 '19 at 15:16
  • Okay, so do you just need to restore a missing /var? I feel comfortable enough walking you through the process of doing that. If you have backups this will be much easier but you can copy a debootstrapped /var over via scp without too much trouble. If you require the mariadb database back and you do not have backups then you need to power off the device as soon as possible and pull its sd card to attempt to use a data recovery solution to restore that data. – kemotep Nov 26 '19 at 15:22
  • Would there be no problem copying the /var directory from one computer to another despite being similar? In that case I would only have to copy the content. The current database data is not important as they are sent to a backup database server. – balon Nov 26 '19 at 15:30
  • There may be some issues, I will spend some time researching this before I post an answer to clear those up. Using debootstrap to create a "clean" /var to copy over should work. As long as the working raspbian system you have has the disk space to install another instance of raspbian and can briefly enable root ssh access from this system to the broken system to copy /var over it should work. – kemotep Nov 26 '19 at 15:34
4

I will be referencing the following article and Debian Wiki post for my answer.

First, on a working Raspbian system, create a debootstrap environment. Following the instructions from the Debian Wiki you would only need to complete the following:

mkdir /debootstrap
debootstrap stable /debootstrap http://deb.debian.org/debian/

Now, you are using Raspbian so you may need to add the correct gpg-key, as pointed out here, and reference the Raspbian Deb Mirror. Something like this: http://archive.raspbian.org/raspbian. I am going to include a link to a cross-platform build guide encase you are doing this on an amd64 system instead of a Raspberry Pi.

IMPORTANT NOTE: if you are prompted, DO NOT install grub to the MBR! This might break your existing install!

Next, on the broken system get a list of all installed programs. Initially I suggested using dpkg --get-selections, however as user A.B. points out this will fail. dpkg references various /var directories to work and with your /var missing this will of course fail. User A.B. points out that you can reference the contents of /usr/share/doc/ and related symlinks. I would start by see what is available and try building an installed.txt with your results.

cd /usr/share/doc && for i in *; do echo $i install >> /home/user/installed.txt; done

Compare the results of this installed.txt with what you expect. Another place to check would be /usr/share/man. This directory has manpages but will only be accurate to packages that install manpages. If you have a known working Raspberry Pi to also compare against, and it is configured the same as the broken Raspberry Pi, you can build installed.txt with the output of that Raspberry Pi's dpkg --get-selections.

Copy installed.txt from the missing /var host to the fixing /var host. scp or rysnc will work here but at this point you do need to confirm that you have a working ssh configuration between the broken host and the host doing the fixing. Without a working ssh configuration between these two hosts, you will be unable to copy over /var. Any ssh issues need to be solved before moving on.

Now, move this file into the /deboostrap environment and prepare to install all the same software inside this chroot.

cp installed.txt /debootstrap/
chroot /bootstrap/
mount -t proc none /proc
dpkg --set-selections < installed.txt

If need be, make sure your sources.list matches. debootstrap by default only includes main.

apt update
apt install dselect
dselect update
apt-get dselect-upgrade

If you were unable to build a good installed.txt

You may need to simply manually install all necessary packages in your debootstrap chroot. Here is where good setup documentation or infrastructure as code comes in handy. Run apt install package1 package2 package3 ... packageN in the chroot to build your /var to as close as possible match the original /var. Instead of installing dselect you should follow these steps:

chroot /bootstrap/
mount -t proc none /proc
apt update
apt install package1 package2 package3 ... packageN

At this point your debootstrap environment should match that of your target, broken host. Here is where you need to use rsync to copy /var over. rsync -A should be enough. As pointed out in my link, you may need to allow PermitRootLogin without-password in your sshd_config for rsync to work.

I will also just directly quote the last steps and considerations Pete Donnell at Alephnull.uk had to run:

...I had to reinstall the mariadb-server packages on the broken server using apt-get install --reinstall mariadb-server mariadb-server-10.1 mariadb-server-core-10.1.

The next step is to restore your user data. This will be specific to the structure that you use, so I can’t help with instructions for that. Once you’ve done that, check the permissions of the files in /var/lib and /var/log against a working server (or perhaps your bootstrap) to check that they are correct. It’s quite likely that the user IDs of the system accounts will be different between the server and the bootstrap environment.

Now you should check the various services that are installed, using service --status-all. Not all of the services should necessarily be enabled, again I recommend comparing against a working server. Try to restart any services that aren't currently running but should be. If all the permissions are correct and the relevant user data (if any) has been restored then the service should start successfully. If it doesn’t, check the systemd status with systemctl status <service-name>, the systemd logs with journalctl -xe and the service’s log files in /var/log. Those should give you enough information to track down any remaining problems.

I highly recommend you spend time comparing the debootstrap environment and your broken host as well if you are fortunate enough to have a second working host. You want to confirm that all the permissions are right and services are running properly. Only once you have done this, I would reboot the host and confirm that everything came back.

Please thoroughly read through every link I have provided before trying any commands. If there are any mistakes, misconceptions, or issues with my post please let me know and I will work to correct them.

Best of Luck!

| improve this answer | |
  • 1
    I expect dpkg --get-selections to fail since it depends on informations in /var/lib/dpkg which disappeared. Using the content of /usr/share/doc/ (including symlinks) to build installed.txt may be an alternative even if it might be slightly inaccurate. – A.B Nov 26 '19 at 21:32
  • 1
    @A.B that is a good point, thank you. When I have time I will adapt the answer accordingly. – kemotep Nov 26 '19 at 22:39
  • 1
    @balon I have updated the answer to include A.B's suggestions. Without being able to use dpkg on a comparable system you will likely need to manually install all packages to build up /var. – kemotep Nov 27 '19 at 13:28
  • 2
    testing on a small system with ~ 1100 packages I get < 10 differences, with most of them because of status hold or deinstall. By the way the file should be built with something like (cd /usr/share/doc && for i in *; do echo $i install; done) : there has to be the word install at the end for the dpkg --set-selections input – A.B Nov 27 '19 at 13:47
  • 2
    Thanks for your help! I have finally solved it by following the steps of this response but only by copying the content of /var/lib/dpkg, this has fixed dpkg and I have been able to restore everything as usual. – balon Nov 27 '19 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.