/var/lib/dpkg/ disappeared after fsck

Question

All apt-like commands failed to create the lock file, because /var/lib/dpkg/ doesn't exist. Also, /lost+found/ has contents:

pi@pi-top:~ $ sudo ls -al /lost+found/
total 102456
drwx------ 11 root root      16384 Apr  3 16:26 .
drwxr-xr-x 23 root root       4096 May  5 17:00 ..
-rw-------  1 root root  104857600 Apr  3 16:30 #29025
lrwxrwxrwx  1 root root          9 Mar 29 10:05 #87867 -> /run/lock
lrwxrwxrwx  1 root root          4 Mar 29 10:05 #87868 -> /run
drwxr-xr-x  2 root root       4096 May  5 10:35 #89863
drwxr-xr-x 12 root root       4096 Apr  3 16:41 #89864
drwxr-xr-x 44 root root       4096 Apr  3 16:30 #89865
drwxrwsr-x  2 root staff      4096 Mar 12 14:03 #89866
drwxr-xr-x  6 root root       4096 May  5 16:30 #89867
drwxrwsr-x  2 root mail       4096 Mar 29 10:05 #89868
drwxr-xr-x  2 root root       4096 Mar 29 10:05 #89869
drwxr-xr-x  5 root root       4096 Mar 29 10:32 #89870
drwxrwxrwt  3 root root       4096 May  5 16:31 #89871

Lots of /var/lib/ is also missing, though the system isn't showing any other symptoms. Is it possible to recover the system (or at least dpkg)? If so, how?

Answer 1

I'm sorry, but at this stage, you're probably better off restoring from a clean backup. When fsck places so many directories in /lost+found, that's a sign that there's been a lot of corruption. It's quite possible that there's more corruption, but because it's in the file contents rather than in metadata, fsck has no way to know.

When restoring from a backup, make sure that it's a clean backup. The corruption may have started before it was detected.

The only way to identify what the files in lost+found are is to look at them and figure it out. There is no systematic way. If there was one, fsck would do it.

Looking at the contents you show for /lost+found, it looks like the /var directory got damaged. You can try to repair it by creating /var and moving the appropriate entries in /lost+found to /var.

# Running as root, of course
umask 022
mkdir /var
mv /lost+found/\#87867 /var/lock
mv /lost+found/\#87868 /var/run
mv /lost+found/\#89866 /var/local
mv /lost+found/\#89868 /var/mail
…

I figured out the entries above from the metadata (ownership and symlink targets). You can figure more out by looking at the directory contents. Compare with an existing system installation (preferably the same distribution or at least something close, but the processor architecture doesn't matter).

/var/lib may be #89865 because it tends to have a lot of subdirectories, but that's only a guess. It could be from another part of the system altogether.

Don't focus on recovering /var/lib/dpkg and ignore the rest. The lack of /var/lib/dpkg is just the first symptom you noticed.

On a PC, I'd suggest to do a RAM test (with Memtest86+ which is available as a package on most distributions and is installed by default at least on Ubuntu). On a Raspberry Pi, if your system is on an SD card, I recommend replacing the SD card: SD cards are the least reliable part of the system, and if you keep using it, it'll probably keep corrupting your data.