Something has corrupted my ext4 partition on my debian server. The server had an uptime of about 1700 days, and unfortunately the UPS did not have enough battery to last a whole day power outage.
I was running Debian sid on it (so stupid), and about the time systemd came into the picture i stopped doing apt-get dist-upgrade's. (more stupid)
After the power outage, it booted into systemd emergency mode, and it was not possible to mount the drive at all. (/dev/sdb1 not found). I tried apt-get dist-upgrade to see if it would fix the problem, but it failed after a while, and after rebooting I could not log in anymore because systemd failed. After a while I found sysvinit in the bootmenu, and could then boot into linux. (Should've done that first! doh)
The partition that's become corrupted is on a 750 GB harddrive and is used for /home. There should be about 100 GB of used space, but it's reported as 968 MB.
And somehow, the root directory structure for / (/dev/sda1) has been copied to /home (/dev/sdc1) ???
root@server:/home# ls -lah
ls: cannot access 'run': Input/output error
ls: cannot access 'sys': Input/output error
ls: cannot access 'etc': Input/output error
total 92K
drwxr-xr-x 23 root root 4.0K Jan 14 2013 .
drwxr-xr-x 23 root root 4.0K Feb 11 14:23 ..
drwxr-xr-x 2 user2 user2 4.0K Nov 2 2010 bin
drwxr-xr-x 2 user1 user1 4.0K Feb 3 2013 boot
drwxrwxr-x 3 user2 user2 4.0K Aug 19 2005 dev
d????????? ? ? ? ? ? etc
drwxr-xr-x 2 user1 user1 4.0K Jan 15 2013 home
lrwxrwxrwx 1 root root 30 Jan 3 2013 initrd.img -> /boot/initrd.img-3.2.0-4-amd64
drwxr-xr-x 14 root root 4.0K Jan 9 2013 lib
drwxr-xr-x 2 root root 4.0K Jan 3 2013 lib64
drwx------ 2 root root 16K Jan 3 2013 lost+found
drwxr-xr-x 2 user1 user1 4.0K May 31 2010 media
drwxr-xr-x 6 user1 user1 4.0K Jan 15 2013 mnt
drwxr-xr-x 2 user1 user1 4.0K Feb 5 2006 opt
drwxr-xr-x 2 user1 user1 4.0K Feb 3 2013 proc
drwxr-xr-x 2 user1 user1 4.0K Feb 3 2013 root
d????????? ? ? ? ? ? run
drwxr-xr-x 4 user3 user3 4.0K Mar 26 2013 sbin
drwxr-xr-x 2 user1 user1 4.0K Jan 15 2013 selinux
drwxr-xr-x 2 user1 user1 4.0K Feb 3 2013 srv
d????????? ? ? ? ? ? sys
drwxr-xr-x 3 user4 user4 4.0K Nov 27 23:05 tmp
drwx------ 2 user5 user5 4.0K Aug 17 2009 usr
drwx--x--x 13 user2 user2 4.0K Jan 12 2012 var
lrwxrwxrwx 1 root root 26 Jan 3 2013 vmlinuz -> boot/vmlinuz-3.2.0-4-amd64
(usernames changed)
This looks identical to / (/dev/sda1).
But inside these folders are files which shouldn't be where they are, but are readable. For example, inside "opt" there are jpegs from somewhere in my own userdir. Inside "var" are files and folders from one user almost completely intact with correct directory structure and filenames, just files missing on at least one place.
I did a dd copy of the partition to another harddrive, and then tried fsck on it, it apparently "fixed" a lot of things, but then it wasn't possible to mount it any longer. "wrong fs type, bad option, bad superblock on /dev/sdc1, missing codepage or helper program, or other error."
I have no idea how this happened, or how to fix it, and I don't have high expectations that it will be possible. This is the weirdest thing I have seen in linux, and I mostly want to know how it happened. If I can recover all files though, that would make me a very happy man!
I know about PhotoRec, and plan on using it if I can't repair the original directory structure.
