1

Something has corrupted my ext4 partition on my debian server. The server had an uptime of about 1700 days, and unfortunately the UPS did not have enough battery to last a whole day power outage.

I was running Debian sid on it (so stupid), and about the time systemd came into the picture i stopped doing apt-get dist-upgrade's. (more stupid)

After the power outage, it booted into systemd emergency mode, and it was not possible to mount the drive at all. (/dev/sdb1 not found). I tried apt-get dist-upgrade to see if it would fix the problem, but it failed after a while, and after rebooting I could not log in anymore because systemd failed. After a while I found sysvinit in the bootmenu, and could then boot into linux. (Should've done that first! doh)

The partition that's become corrupted is on a 750 GB harddrive and is used for /home. There should be about 100 GB of used space, but it's reported as 968 MB.

And somehow, the root directory structure for / (/dev/sda1) has been copied to /home (/dev/sdc1) ???

root@server:/home# ls -lah
ls: cannot access 'run': Input/output error
ls: cannot access 'sys': Input/output error
ls: cannot access 'etc': Input/output error
total 92K
drwxr-xr-x 23 root    root    4.0K Jan 14  2013 .
drwxr-xr-x 23 root    root    4.0K Feb 11 14:23 ..
drwxr-xr-x  2 user2   user2   4.0K Nov  2  2010 bin
drwxr-xr-x  2 user1   user1   4.0K Feb  3  2013 boot
drwxrwxr-x  3 user2   user2   4.0K Aug 19  2005 dev
d?????????  ? ?       ?          ?            ? etc
drwxr-xr-x  2 user1   user1   4.0K Jan 15  2013 home
lrwxrwxrwx  1 root    root      30 Jan  3  2013 initrd.img -> /boot/initrd.img-3.2.0-4-amd64
drwxr-xr-x 14 root    root    4.0K Jan  9  2013 lib
drwxr-xr-x  2 root    root    4.0K Jan  3  2013 lib64
drwx------  2 root    root     16K Jan  3  2013 lost+found
drwxr-xr-x  2 user1   user1   4.0K May 31  2010 media
drwxr-xr-x  6 user1   user1   4.0K Jan 15  2013 mnt
drwxr-xr-x  2 user1   user1   4.0K Feb  5  2006 opt
drwxr-xr-x  2 user1   user1   4.0K Feb  3  2013 proc
drwxr-xr-x  2 user1   user1   4.0K Feb  3  2013 root
d?????????  ? ?       ?          ?            ? run
drwxr-xr-x  4 user3   user3   4.0K Mar 26  2013 sbin
drwxr-xr-x  2 user1   user1   4.0K Jan 15  2013 selinux
drwxr-xr-x  2 user1   user1   4.0K Feb  3  2013 srv
d?????????  ? ?       ?          ?            ? sys
drwxr-xr-x  3 user4   user4   4.0K Nov 27 23:05 tmp
drwx------  2 user5   user5   4.0K Aug 17  2009 usr
drwx--x--x 13 user2   user2   4.0K Jan 12  2012 var
lrwxrwxrwx  1 root    root      26 Jan  3  2013 vmlinuz -> boot/vmlinuz-3.2.0-4-amd64

(usernames changed)

This looks identical to / (/dev/sda1).

But inside these folders are files which shouldn't be where they are, but are readable. For example, inside "opt" there are jpegs from somewhere in my own userdir. Inside "var" are files and folders from one user almost completely intact with correct directory structure and filenames, just files missing on at least one place.

I did a dd copy of the partition to another harddrive, and then tried fsck on it, it apparently "fixed" a lot of things, but then it wasn't possible to mount it any longer. "wrong fs type, bad option, bad superblock on /dev/sdc1, missing codepage or helper program, or other error."

I have no idea how this happened, or how to fix it, and I don't have high expectations that it will be possible. This is the weirdest thing I have seen in linux, and I mostly want to know how it happened. If I can recover all files though, that would make me a very happy man!

I know about PhotoRec, and plan on using it if I can't repair the original directory structure.

0

First, as a backup, make a copy of the partition with dd, gddrescue, or some other imaging util, (even cp works).

Given that's a v3.2 kernel from 2013, try rebooting from a newer distro/kernel, then run its newer fsck on the afflicted partition. The newer distros' file utils have had five years of bugfixes and improvements, and that may help.

  • 2
    I'm unfamiliar with procedures in situations like these, but wouldn't it be an idea if the OP first tried to make an image of the affected filsystems before attempting to recover? This may not be practically possible, but nonetheless, it would preserve what they have before possibly destroying it completely... – Kusalananda Feb 11 '18 at 20:58
  • @Kusalananda Good idea! Clone it first, and then try agc's suggestion. Clonezilla or dd. – Mike Waters Feb 11 '18 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.