Strange issue. After an extended power outage (transformer exploded!) which outlasted the UPS, a server running Slackware 13 crashed. This machine has been quietly serving accounting software and file storage in a multiuser environment since December of 2009 with NO ISSUES!
This machine has two 500GB SATA disks set up using LVM on top of software RAID1 (all done by slackware installer). When power is applied, the initrd image
loads properly, but the boot fails when it's time to mount the root filesystem.
"Failed to mount /dev/Volume00/RootVol on /mnt. No such device or directory."
RootVol is a logical volume in volume group Volume00. There are 3 other LV filesystems plus swap in this volume group. I can mount any of them from the boot#_ prompt
and they appear fine. However, RootVol, the root filesystem, does not show up at all... /dev/mapper/Volume00
lists the other 4 but RootVol is missing.
So, I booted up a linux rescue cd. The RAID1 volume /dev/md0
comes up, and lo and behold, I can see, and mount, /dev/Volume00/RootVol
.
Now, /proc/mdstat
shows that /dev/md0
is running degraded, using only device /dev/dm-1
. I'm not sure if this means that I'm down a disk... I am accustomed to seeing actual hard drives in /proc/mdstat... /dev/sda1, /dev/sda2, etc. on non-LVM arrays. How can I determine which disk has failed? How do /dev/dm-x
devices relate to /dev/sdx
devices? And how can I get the machine to boot from the remaining disk, which (appears to) be fine. The two hard drives used in the array are /dev/sda and /dev/sdb, entire disks.
Can anyone point me to a tutorial on /dev/dm-x devices? I see they are LVM devices from the dmsetup documentation but don't want to start messing with anything without a good understanding of what I'm doing. I thought RAID was built on physical devices, then volume groups built on raid devices... I cannot fathom why there would be LVM devices showing as elements of a disk array but I'm sure there's a damned good reason... or a fundamental misunderstanding on my part.
Here's some info:
Output of cat /dev/mdstat is:
root@sysresccd /mnt/rootvol/etc % cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 dm-1[0]
488287488 blocks [2/1] [U_]
unused devices: <none>
Not what I expected. My two hard drives are /dev/sda and /dev/sdb, no errors in /var/log/messages about them although I have no ability to tweak loglevels in the Live CD version I am running. Why does /proc/mdstat not show actual devices? What are /dev/dm-# devices?
Output of mdadm --detail /dev/md0 is:
root@sysresccd /mnt/rootvol/etc % dmadm -D /dev/md0
zsh: correct 'dmadm' to 'mdadm' [nyae]? y
/dev/md0:
Version : 0.90
Creation Time : Thu Dec 3 11:53:48 2009
Raid Level : raid1
Array Size : 488287488 (465.67 GiB 500.01 GB)
Used Dev Size : 488287488 (465.67 GiB 500.01 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Sun Jul 10 12:00:57 2016
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
UUID : a89cbdf5:f83cf3f7:dcc86dce:213c81b2
Events : 0.38
Number Major Minor RaidDevice State
0 253 1 0 active sync /dev/dm-1
2 0 0 2 removed
I am guessing that either a) I have a failed disk or b) the array /dev/md0 is not synched, maybe thinks a disk has failed?
At any rate, the machine definitely will not boot from this state, and I can't figure out which, if any, of my hard disks are the problem, nor how to fix this mess. This is a production server with full backups... I could rebuild it, but really would rather not as it's a pretty tedious process... there's nothing wrong with the data nor, I'm guessing, either of the disks.
There is no mdadm.conf
.
fdisk -l
shows both disks as Linux Raid Autodetect, everything looks normal.
Output of dmsetup
is (???'s inserted by me):
root@sysresccd /mnt/rootvol/etc % dmsetup ls
isw_bfdbfijegh_Volume01 (253:1)
isw_bfdbfijegh_Volume0 (253:0)
Volume00-MediaVol (253:9)
Volume00-RootSnap-cow (253:4) <-- ??? --- "cow" --- ???
Volume00-XplrVol (253:7)
Volume00-RootSnap (253:5)
Volume00-SwapVol (253:8)
Volume00-RootVol (253:3)
Volume00-RootVol-real (253:2) <--- ??? --- "real" --- ???
Volume00-HomeVol (253:6)
I posted the info below as a reply, but it was suggested I edit this original post and add it in so here it is:
Here is dmsetup
table:
root@sysresccd /mnt/usbhdd1/server_backup/oldroot % dmsetup table
isw_bfdbfijegh_Volume01: 0 976575222 linear 253:0 63
isw_bfdbfijegh_Volume0: 0 976767240 mirror core 2 131072 nosync 2 8:0 0 8:16 0 1 handle_errors
Volume00-MediaVol: 0 209715200 linear 9:127 134218112
Volume00-RootSnap-cow: 0 2097152 linear 9:127 385876352
Volume00-XplrVol: 0 41943040 linear 9:127 83886464
Volume00-RootSnap: 0 20971520 snapshot 253:2 253:4 P 8
Volume00-SwapVol: 0 8388608 linear 9:127 125829504
Volume00-RootVol: 0 20971520 snapshot-origin 253:2
Volume00-RootVol-real: 0 20971520 linear 9:127 384
Volume00-HomeVol: 0 62914560 linear 9:127 20971904
Volume00-HomeVol: 62914560 41943040 linear 9:127 343933312
And here is mdadm -E
for each hard drive.
root@sysresccd /mnt/usbhdd1/server_backup/oldroot % mdadm -E /dev/sda
mdmon: /dev/sda is not attached to Intel(R) RAID controller.
mdmon: /dev/sda is not attached to Intel(R) RAID controller.
/dev/sda:
Magic : Intel Raid ISM Cfg Sig.
Version : 1.1.00
Orig Family : 5b4a335b
Family : 5b4a335b
Generation : 0000000a
Attributes : All supported
UUID : 15980595:45ac18ac:22467c90:56138fde
Checksum : c51f833a correct
MPB Sectors : 1
Disks : 2
RAID Devices : 1
Disk00 Serial : WD-WCASY8349657
State : active
Id : 00000000
Usable Size : 976767240 (465.76 GiB 500.10 GB)
[Volume0]:
UUID : 4b19b799:c2a34a28:0bcca120:36d318a0
RAID Level : 1
Members : 2
Slots : [UU]
Failed disk : none
This Slot : 0
Array Size : 976766976 (465.76 GiB 500.10 GB)
Per Dev Size : 976767240 (465.76 GiB 500.10 GB)
Sector Offset : 0
Num Stripes : 3815496
Chunk Size : 64 KiB
Reserved : 0
Migrate State : idle
Map State : uninitialized
Dirty State : clean
Disk01 Serial : WD-WCASY8288673
State : active
Id : 00010000
Usable Size : 976767240 (465.76 GiB 500.10 GB)
root@sysresccd /mnt/usbhdd1/server_backup/oldroot % mdadm -E /dev/sdb
mdmon: /dev/sdb is not attached to Intel(R) RAID controller.
mdmon: /dev/sdb is not attached to Intel(R) RAID controller.
/dev/sdb:
Magic : Intel Raid ISM Cfg Sig.
Version : 1.1.00
Orig Family : 5b4a335b
Family : 5b4a335b
Generation : 0000000a
Attributes : All supported
UUID : 15980595:45ac18ac:22467c90:56138fde
Checksum : c51f833a correct
MPB Sectors : 1
Disks : 2
RAID Devices : 1
Disk01 Serial : WD-WCASY8288673
State : active
Id : 00010000
Usable Size : 976767240 (465.76 GiB 500.10 GB)
[Volume0]:
UUID : 4b19b799:c2a34a28:0bcca120:36d318a0
RAID Level : 1
Members : 2
Slots : [UU]
Failed disk : none
This Slot : 1
Array Size : 976766976 (465.76 GiB 500.10 GB)
Per Dev Size : 976767240 (465.76 GiB 500.10 GB)
Sector Offset : 0
Num Stripes : 3815496
Chunk Size : 64 KiB
Reserved : 0
Migrate State : idle
Map State : uninitialized
Dirty State : clean
Disk00 Serial : WD-WCASY8349657
State : active
Id : 00000000
Usable Size : 976767240 (465.76 GiB 500.10 GB)
Wouldn't initrd
already have --assemble in it? The array is up even when I can't mount the root volume... the other 3 volumes I CAN mount and they're in the same VG on the same RAID1 array... why can I mount /dev/Volume00/RootVol
when booted from rescue CD vs. booting from disk?
dmsetup table
and also,mdadm -E /dev/sd[ab]*
. I'd guess the recovery disk has done something weird (read: wrong) in how it set up LVM and RAID. I'd guess anmdadm --assemble «something»
from the initrd would work.../dev/dm-*
is Device Mapper, which is the underlying technology behind LVM, and several other things. "cow" (= copy-on-write) and "real" are from having an LVM snapshot.vgchange -ay
and see what happens. Please also add output ofpvdisplay -m
andlvs
. But my recommendation is: First do a backup of both disks usingddrescue
(sometimes calledgddrescue
). You can attach storage for this usingsshfs
(if you have a stable network connection) or do it the other way round (mount the/dev
from the server viasshfs
and do theddrescue
on the other side, but this is slower and less intuitive).