4

Strange issue. After an extended power outage (transformer exploded!) which outlasted the UPS, a server running Slackware 13 crashed. This machine has been quietly serving accounting software and file storage in a multiuser environment since December of 2009 with NO ISSUES!

This machine has two 500GB SATA disks set up using LVM on top of software RAID1 (all done by slackware installer). When power is applied, the initrd image loads properly, but the boot fails when it's time to mount the root filesystem.

"Failed to mount /dev/Volume00/RootVol on /mnt.  No such device or directory."

RootVol is a logical volume in volume group Volume00. There are 3 other LV filesystems plus swap in this volume group. I can mount any of them from the boot#_ prompt and they appear fine. However, RootVol, the root filesystem, does not show up at all... /dev/mapper/Volume00 lists the other 4 but RootVol is missing.

So, I booted up a linux rescue cd. The RAID1 volume /dev/md0 comes up, and lo and behold, I can see, and mount, /dev/Volume00/RootVol.

Now, /proc/mdstat shows that /dev/md0 is running degraded, using only device /dev/dm-1. I'm not sure if this means that I'm down a disk... I am accustomed to seeing actual hard drives in /proc/mdstat... /dev/sda1, /dev/sda2, etc. on non-LVM arrays. How can I determine which disk has failed? How do /dev/dm-x devices relate to /dev/sdx devices? And how can I get the machine to boot from the remaining disk, which (appears to) be fine. The two hard drives used in the array are /dev/sda and /dev/sdb, entire disks.

Can anyone point me to a tutorial on /dev/dm-x devices? I see they are LVM devices from the dmsetup documentation but don't want to start messing with anything without a good understanding of what I'm doing. I thought RAID was built on physical devices, then volume groups built on raid devices... I cannot fathom why there would be LVM devices showing as elements of a disk array but I'm sure there's a damned good reason... or a fundamental misunderstanding on my part.

Here's some info:

Output of cat /dev/mdstat is:

root@sysresccd /mnt/rootvol/etc % cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4]         [raid10] 
md0 : active raid1 dm-1[0]
488287488 blocks [2/1] [U_]

unused devices: <none>

Not what I expected. My two hard drives are /dev/sda and /dev/sdb, no errors in /var/log/messages about them although I have no ability to tweak loglevels in the Live CD version I am running. Why does /proc/mdstat not show actual devices? What are /dev/dm-# devices?

Output of mdadm --detail /dev/md0 is:

root@sysresccd /mnt/rootvol/etc % dmadm -D /dev/md0
zsh: correct 'dmadm' to 'mdadm' [nyae]? y
/dev/md0:
Version : 0.90
Creation Time : Thu Dec 3 11:53:48 2009
Raid Level : raid1
Array Size : 488287488 (465.67 GiB 500.01 GB)
Used Dev Size : 488287488 (465.67 GiB 500.01 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Sun Jul 10 12:00:57 2016
State : clean, degraded 
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

UUID : a89cbdf5:f83cf3f7:dcc86dce:213c81b2
Events : 0.38

Number Major Minor RaidDevice State
0 253 1 0 active sync /dev/dm-1
2 0 0 2 removed

I am guessing that either a) I have a failed disk or b) the array /dev/md0 is not synched, maybe thinks a disk has failed?

At any rate, the machine definitely will not boot from this state, and I can't figure out which, if any, of my hard disks are the problem, nor how to fix this mess. This is a production server with full backups... I could rebuild it, but really would rather not as it's a pretty tedious process... there's nothing wrong with the data nor, I'm guessing, either of the disks.

There is no mdadm.conf.

fdisk -l shows both disks as Linux Raid Autodetect, everything looks normal.

Output of dmsetup is (???'s inserted by me):

root@sysresccd /mnt/rootvol/etc % dmsetup ls 
isw_bfdbfijegh_Volume01 (253:1)
isw_bfdbfijegh_Volume0  (253:0)
Volume00-MediaVol   (253:9)
Volume00-RootSnap-cow   (253:4) <-- ??? --- "cow" --- ???
Volume00-XplrVol    (253:7)
Volume00-RootSnap   (253:5)
Volume00-SwapVol    (253:8)
Volume00-RootVol    (253:3)
Volume00-RootVol-real   (253:2) <--- ??? --- "real" --- ???
Volume00-HomeVol    (253:6)

I posted the info below as a reply, but it was suggested I edit this original post and add it in so here it is:

Here is dmsetup table:

root@sysresccd /mnt/usbhdd1/server_backup/oldroot % dmsetup table
isw_bfdbfijegh_Volume01: 0 976575222 linear 253:0 63
isw_bfdbfijegh_Volume0: 0 976767240 mirror core 2 131072 nosync 2 8:0 0    8:16 0 1 handle_errors
Volume00-MediaVol: 0 209715200 linear 9:127 134218112
Volume00-RootSnap-cow: 0 2097152 linear 9:127 385876352
Volume00-XplrVol: 0 41943040 linear 9:127 83886464
Volume00-RootSnap: 0 20971520 snapshot 253:2 253:4 P 8
Volume00-SwapVol: 0 8388608 linear 9:127 125829504
Volume00-RootVol: 0 20971520 snapshot-origin 253:2
Volume00-RootVol-real: 0 20971520 linear 9:127 384
Volume00-HomeVol: 0 62914560 linear 9:127 20971904
Volume00-HomeVol: 62914560 41943040 linear 9:127 343933312

And here is mdadm -E for each hard drive.

root@sysresccd /mnt/usbhdd1/server_backup/oldroot % mdadm -E /dev/sda
mdmon: /dev/sda is not attached to Intel(R) RAID controller.
mdmon: /dev/sda is not attached to Intel(R) RAID controller.
/dev/sda:
          Magic : Intel Raid ISM Cfg Sig.
        Version : 1.1.00
    Orig Family : 5b4a335b
         Family : 5b4a335b
     Generation : 0000000a
     Attributes : All supported
           UUID : 15980595:45ac18ac:22467c90:56138fde
       Checksum : c51f833a correct
    MPB Sectors : 1
          Disks : 2
   RAID Devices : 1

  Disk00 Serial : WD-WCASY8349657
          State : active
             Id : 00000000
    Usable Size : 976767240 (465.76 GiB 500.10 GB)

[Volume0]:
           UUID : 4b19b799:c2a34a28:0bcca120:36d318a0
     RAID Level : 1
        Members : 2
          Slots : [UU]
    Failed disk : none
      This Slot : 0
     Array Size : 976766976 (465.76 GiB 500.10 GB)
   Per Dev Size : 976767240 (465.76 GiB 500.10 GB)
  Sector Offset : 0
    Num Stripes : 3815496
     Chunk Size : 64 KiB
       Reserved : 0
  Migrate State : idle
      Map State : uninitialized
    Dirty State : clean

  Disk01 Serial : WD-WCASY8288673
          State : active
             Id : 00010000
    Usable Size : 976767240 (465.76 GiB 500.10 GB)


root@sysresccd /mnt/usbhdd1/server_backup/oldroot % mdadm -E /dev/sdb
mdmon: /dev/sdb is not attached to Intel(R) RAID controller.
mdmon: /dev/sdb is not attached to Intel(R) RAID controller.
/dev/sdb:
          Magic : Intel Raid ISM Cfg Sig.
        Version : 1.1.00
    Orig Family : 5b4a335b
         Family : 5b4a335b
     Generation : 0000000a
     Attributes : All supported
           UUID : 15980595:45ac18ac:22467c90:56138fde
       Checksum : c51f833a correct
    MPB Sectors : 1
          Disks : 2
   RAID Devices : 1

  Disk01 Serial : WD-WCASY8288673
          State : active
             Id : 00010000
    Usable Size : 976767240 (465.76 GiB 500.10 GB)

[Volume0]:
           UUID : 4b19b799:c2a34a28:0bcca120:36d318a0
     RAID Level : 1
        Members : 2
          Slots : [UU]
    Failed disk : none
      This Slot : 1
     Array Size : 976766976 (465.76 GiB 500.10 GB)
   Per Dev Size : 976767240 (465.76 GiB 500.10 GB)
  Sector Offset : 0
    Num Stripes : 3815496
     Chunk Size : 64 KiB
       Reserved : 0
  Migrate State : idle
      Map State : uninitialized
    Dirty State : clean

  Disk00 Serial : WD-WCASY8349657
          State : active
             Id : 00000000
    Usable Size : 976767240 (465.76 GiB 500.10 GB)

Wouldn't initrd already have --assemble in it? The array is up even when I can't mount the root volume... the other 3 volumes I CAN mount and they're in the same VG on the same RAID1 array... why can I mount /dev/Volume00/RootVol when booted from rescue CD vs. booting from disk?

| improve this question | |
  • Could we get dmsetup table and also, mdadm -E /dev/sd[ab]*. I'd guess the recovery disk has done something weird (read: wrong) in how it set up LVM and RAID. I'd guess an mdadm --assemble «something» from the initrd would work... – derobert Jul 11 '16 at 16:16
  • And yes, /dev/dm-* is Device Mapper, which is the underlying technology behind LVM, and several other things. "cow" (= copy-on-write) and "real" are from having an LVM snapshot. – derobert Jul 11 '16 at 16:17
  • 2
    Dave, it also appears that you have two unregistered accounts; I approved your edit but you'll want to confirm one of them then ask for them to be merged – Jeff Schaller♦ Jul 11 '16 at 20:44
  • 1
    Try vgchange -ay and see what happens. Please also add output of pvdisplay -m and lvs. But my recommendation is: First do a backup of both disks using ddrescue (sometimes called gddrescue). You can attach storage for this using sshfs (if you have a stable network connection) or do it the other way round (mount the /dev from the server via sshfs and do the ddrescue on the other side, but this is slower and less intuitive). – Tino Jul 12 '16 at 7:19
  • 1
    Also, your drives are 6.5 years old - overdue for replacement. The smallest HDD replacement you can get these days is 1TB for around $60-$100. If you don't need any more storage space but would benefit from more I/O performance, replace them with SATA3 SSDs - from $200-$300 for 500GB. 1TB SSDs start from around $400. BTW, LVM can do RAID-1 by itself, no need to layer it on top of mdadm. When you rebuild (and you should rebuild, because your drives are too old to be trusted), you should use just LVM and not bother with mdadm. – cas Jul 12 '16 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.