Yesterday I replaced one of the HDDs in my remote server. The setup is simple - Debian Linux (testing), software RAID 5, LVM with 4 partitions on it, home, var, usr, and swap.
I pulled the old disk, popped the new one in, saw it was recognised, added a partition and added it to the array, and it started rebuilding.
I noticed this morning that I'd accidentally added the wrong device to the array.... /dev/sdf instead of /sdf1. I've had this before and it buggers up when booting. So I failed the disk, removed it form the array, then repartitioned it (as it had been blatted), then re-added it to the array.
It jumped quickly to 10% rebuilt, and then was adamant it had completed when I next checked 20 or so minutes later. "That was quick", I thought.
Then I got a call a few hours later - services were down. I'd left myself logged in so I knew it was still running, but the logs were full of ext3 errors about going past the end of the device on my /var partition.
I've closed off all processes so I could unmount it for a fsck, but it's adamant I can't:
:~# fsck /dev/mapper/main-variable
fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
/dev/mapper/main-variable is in use.
e2fsck: Cannot continue, aborting.
:~# umount /dev/mapper/main-variable
umount: /dev/mapper/main-variable: not mounted
:~#
I can reboot it and let fsck do it's job...but I don't think it'll come back up. And I worry if the fsck is going to make things worse.
Aside from getting the partition wrong originally, did I do something wrong or is this just coincidence and it's bad RAM? I swear I did this last time and there was no corruption, and I (perhaps naively) thought that md would verify the integrity of the array when rebuilding...so I'm assuming it's not that.
I've started getting lots of segfault messages whilst trying to fix this - I don't know if that's due to the swap being bad (even though I've now taken it offline), or perhaps this is the coincidence - perhaps some memory has suddenly failed and that's screwed up the array during the rebuild.
Problem is, I have no physical access to the server for a few weeks now.
