A little backstory because I'm not sure what's relevant...

I set up a file server at a school here in West Africa where they teach computer classes, ESL, and a few other things. Though it's used heavily every day, the server has not been actively managed by anyone for several years. I'm pretty sure it was CentOS 6.X (probably 0). The disk system is five WD Black 2TB drives. The partitions are as follows:

1) boot partition, mdadm RAID 1, two active disks, three spares 2) swap, RAID 1, two active disks, three spares 3) / partition, RAID 1, two active disks, three spares 4-8) several partitions for FS storage, all RAID 5, four active disks, one spare - running LVM on top

The school has very unstable power, and until recently the APC rackmounted units were doing a great job. However, batteries were recently replaced with subpar ones (what's available here) and are now unstable. The CMOS battery on the server MB (3210SHLX) also died.

Also or because of the power instability, one of the five Blacks died. No worries. I swapped the spare from the shelf in and started fdisk'ing and readding the new partitions to the various RAID arrays. Rebuilding went fine, and this configuration worked fine for a few days.

When I was called back a week later, the box wouldn't boot. At that point I found the dead CMOS battery and realized that the BIOS defaults to IDE rather than AHCI mode which only exposes a few of the drives, causing the RAID arrays to try to mount degraded and failing to do so.

Using systemrescueCD I got the arrays put back together and resynced - all except the / device. An mdadm -E of any of the member partitions (sd[abcde]2) showed two interesting things: the UUID of all of them is just zeros, and in the bottom of the listing, they all show each other as members of the array, but show all five disks as spares.

So, I tried reassembling the array in the vane hope that it would restart, but (I'm assuming since the UUID's are zeroed) it failed to reassemble.

Based on the advice of another page I found somewhere, I then tried to create the RAID 1 array using only sde2 and the other disk missing, then tried mounting that device. Doing so produces an NTFS mounting error (despite the underlying partition obviously being ext(3?). I thought that might've been a fluke since sde was the disk I'd recently swapped in, and thought that if it had always been a spare, maybe the data had never been synced to it. I tried the same with sdd and then sdc, same errors.

At that point I decided to stop - probably should've done that earlier. So, now I have two partitions left that I haven't touched, sda2 and sdb2, and I hope one of those two was one of the active members of the array before it was disbanded (still don't know how that part happened.)

It's the OS partition, and I would be content to just do a fresh install to it, except that I'm an idiot and didn't back up my conf files or samba's secrets.tdb file - and it's the domain controller for 40 clients. So reinstalling and losing that will require rebuilding those machines' images and rejoining the new domain. I'd rather not if I can get around it.

All of the other partitions are now working fine in rescue, fully synced with spares and running great.

I'm hoping for some advice on what to try next to hopefully recover the / partition from the two virgin partitions, one of which hopefully contains a copy of the array data. Even an old copy would be just fine.

Thanks in advance!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.