2

Lately I bit the bullet and upgrade my OS from Fedora 11 to Fedora 15, and I've been trying very hard to figure out why Fedora 15 can't see the Raid setup that I created in Fedora 11. I think I must have missed something so I resort to group wisdom here.

When I upgraded, I used a new boot drive for Fedora 15, so I can physically swap the boot drives and boot into either Fedora 11 or 15. Fedora 11 can still see the Raid and everything works. Fedora 15 shows something very strange.

[edited to add the output of the request of @psusi ]

On Fedora 11

I had a regular boot drive (/dev/sda) and an lvm built on raid 5 (/dev/sdb, /dev/sdc, /dev/sdd).

Specifically, the raid disk /dev/md/127_0 is built from /dev/sdb1, /dev/sdc1, /dev/sdd1, where each partition takes the whole disk space.

The volume group of the boot drive (/dev/vg_localhost/) is irrelevant. The volume group that I created onto the raid disk is called /dev/lvm-tb-storage/.

The following is the settings that I got from the system (mdadm, pvscan, lvscan, etc.)

[root@localhost ~]# cat /etc/mdadm.conf 

[root@localhost ~]# pvscan
  PV /dev/md127   VG lvm-tb-storage   lvm2 [1.82 TB / 0    free]
  PV /dev/sda5    VG vg_localhost        lvm2 [61.44 GB / 0    free]
  Total: 2 [1.88 TB] / in use: 2 [1.88 TB] / in no VG: 0 [0   ]

[root@localhost ~]# lvscan
  ACTIVE            '/dev/lvm-tb-storage/tb' [1.82 TB] inherit
  ACTIVE            '/dev/vg_localhost/lv_root' [54.68 GB] inherit
  ACTIVE            '/dev/vg_localhost/lv_swap' [6.77 GB] inherit

[root@localhost ~]# vgdisplay
  --- Volume group ---
  VG Name               lvm-tb-storage
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1.82 TB
  PE Size               4.00 MB
  Total PE              476839
  Alloc PE / Size       476839 / 1.82 TB
  Free  PE / Size       0 / 0   
  VG UUID               wqIXsb-KRZQ-eRnH-JvuP-VdHk-XJTG-DSWimc

  --- Volume group ---
  VG Name               vg_localhost
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               61.44 GB
  PE Size               4.00 MB
  Total PE              15729
  Alloc PE / Size       15729 / 61.44 GB
  Free  PE / Size       0 / 0   
  VG UUID               IVIpCV-C4qg-Lii7-zwkz-P3si-MXAZ-WYUSe6

[root@localhost ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "lvm-tb-storage" using metadata type lvm2
  Found volume group "vg_localhost" using metadata type lvm2

[root@localhost ~]# mdadm --detail --scan
ARRAY /dev/md/127_0 metadata=0.90 UUID=bebfd467:cb6700d9:29bdc0db:c30228ba

[root@localhost ~]# ls -al /dev/md
total 0
drwxr-xr-x.  2 root root   60 2011-09-13 03:14 .
drwxr-xr-x. 19 root root 5180 2011-09-13 03:15 ..
lrwxrwxrwx.  1 root root    8 2011-09-13 03:14 127_0 -> ../md127

[root@localhost ~]# mdadm --detail /dev/md/127_0 
/dev/md/127_0:
        Version : 0.90
  Creation Time : Wed Nov  5 18:26:25 2008
     Raid Level : raid5
     Array Size : 1953134208 (1862.65 GiB 2000.01 GB)
  Used Dev Size : 976567104 (931.33 GiB 1000.00 GB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 127
    Persistence : Superblock is persistent

    Update Time : Tue Sep 13 03:28:51 2011
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : bebfd467:cb6700d9:29bdc0db:c30228ba
         Events : 0.671154

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       49        1      active sync   /dev/sdd1
       2       8       33        2      active sync   /dev/sdc1

[root@localhost ~]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md127 : active raid5 sdb1[0] sdc1[2] sdd1[1]
      1953134208 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

unused devices: <none>

[root@localhost ~]# mdadm --examine /dev/sdb1
/dev/sdb1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : bebfd467:cb6700d9:29bdc0db:c30228ba
  Creation Time : Wed Nov  5 18:26:25 2008
     Raid Level : raid5
  Used Dev Size : 976567104 (931.33 GiB 1000.00 GB)
     Array Size : 1953134208 (1862.65 GiB 2000.01 GB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 127

    Update Time : Tue Sep 13 03:29:50 2011
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0
       Checksum : f1ddf826 - correct
         Events : 671154

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8       17        0      active sync   /dev/sdb1

   0     0       8       17        0      active sync   /dev/sdb1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       33        2      active sync   /dev/sdc1

[root@localhost ~]# fdisk -lu 2>&1
Disk /dev/dm-0 doesn't contain a valid partition table
Disk /dev/dm-1 doesn't contain a valid partition table
Disk /dev/md127 doesn't contain a valid partition table
Disk /dev/dm-2 doesn't contain a valid partition table

Disk /dev/sda: 250.0 GB, 250000000000 bytes
255 heads, 63 sectors/track, 30394 cylinders, total 488281250 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x00000080

  Device Boot      Start         End      Blocks   Id  System
/dev/sda1              63      610469      305203+  83  Linux
/dev/sda2          610470   359004554   179197042+  83  Linux
/dev/sda3   *   359004555   359414154      204800   83  Linux
/dev/sda4       359422245   488279609    64428682+   5  Extended
/dev/sda5       359422308   488278371    64428032   8e  Linux LVM

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0xb03e1980

  Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              63  1953134504   976567221   da  Non-FS data

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x7db522d5

  Device Boot      Start         End      Blocks   Id  System
/dev/sdc1              63  1953134504   976567221   da  Non-FS data

Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x20af5840

  Device Boot      Start         End      Blocks   Id  System
/dev/sdd1              63  1953134504   976567221   da  Non-FS data

Disk /dev/dm-0: 58.7 GB, 58707673088 bytes
255 heads, 63 sectors/track, 7137 cylinders, total 114663424 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x00000000


Disk /dev/dm-1: 7264 MB, 7264534528 bytes
255 heads, 63 sectors/track, 883 cylinders, total 14188544 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x00000000


Disk /dev/md127: 2000.0 GB, 2000009428992 bytes
2 heads, 4 sectors/track, 488283552 cylinders, total 3906268416 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x00000000


Disk /dev/dm-2: 2000.0 GB, 2000007725056 bytes
255 heads, 63 sectors/track, 243153 cylinders, total 3906265088 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x00000000

The kernel boot parameter that I have

kernel /vmlinuz-2.6.30.10-105.2.23.fc11.x86_64 ro root=/dev/mapper/vg_localhost-lv_root rhgb quiet

On Fedora 15

I installed Fedora 15 on a new boot drive, on which the installation program also created an lvm (/dev/vg_20110912a/) for me, but again that's irrelevant.

Under Fedora 15, lvm, pvscan, vgscan see nothing but the irrelevant boot drive. mdadm, however, shows something very strange -- the original raid is broken into three raid, and the combination is very puzzling.

[root@localhost ~] # cat /etc/mdadm.conf
# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all

[root@localhost ~]# pvscan
  PV /dev/sda2   VG vg_20110912a   lvm2 [59.12 GiB / 0  free]
  Total: 1 [59.12 GiB] / in use: 1 [59.12 GiB] / in no VG: 0 [0   ]

[root@localhost ~]# lvscan
  ACTIVE            '/dev/vg_20110912a/lv_home' [24.06 GiB] inherit
  ACTIVE            '/dev/vg_20110912a/lv_swap' [6.84 GiB] inherit
  ACTIVE            '/dev/vg_20110912a/lv_root' [28.22 GiB] inherit

[root@localhost ~]# vgdisplay
  --- Volume group ---
  VG Name               vg_20110912a
  System ID          
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               59.12 GiB
  PE Size               32.00 MiB
  Total PE              1892
  Alloc PE / Size       1892 / 59.12 GiB
  Free  PE / Size       0 / 0   
  VG UUID               8VRJyx-XSQp-13mK-NbO6-iV24-rE87-IKuhHH

[root@localhost ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "vg_20110912a" using metadata type lvm2

[root@localhost ~]# mdadm --detail --scan
ARRAY /dev/md/0_0 metadata=0.90 UUID=153e151b:8c717565:fd59f149:d2ea02c9
ARRAY /dev/md/127_0 metadata=0.90 UUID=bebfd467:cb6700d9:29bdc0db:c30228ba

[root@localhost ~]# ls -l /dev/md
total 4
lrwxrwxrwx. 1 root root   8 Sep 13 02:39 0_0 -> ../md127
lrwxrwxrwx. 1 root root  10 Sep 13 02:39 0_0p1 -> ../md127p1
lrwxrwxrwx. 1 root root   8 Sep 13 02:39 127_0 -> ../md126
-rw-------. 1 root root 120 Sep 13 02:39 md-device-map

[root@localhost ~]# cat /dev/md/md-device-map
md126 0.90 bebfd467:cb6700d9:29bdc0db:c30228ba /dev/md/127_0
md127 0.90 153e151b:8c717565:fd59f149:d2ea02c9 /dev/md/0_0

[root@localhost ~]# mdadm --detail /dev/md/0_0
/dev/md/0_0:
        Version : 0.90
  Creation Time : Tue Nov  4 21:45:19 2008
    Raid Level : raid5
    Array Size : 976762496 (931.51 GiB 1000.20 GB)
  Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 127
    Persistence : Superblock is persistent

    Update Time : Wed Nov  5 09:04:28 2008
        State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

        Layout : left-symmetric
    Chunk Size : 64K

        UUID : 153e151b:8c717565:fd59f149:d2ea02c9
        Events : 0.2202

    Number   Major   Minor   RaidDevice State
    0       8       48      0   active sync   /dev/sdd
    1       8       16      1   active sync   /dev/sdb

[root@localhost ~]# mdadm --detail /dev/md/127_0
/dev/md/127_0:
        Version : 0.90
  Creation Time : Wed Nov  5 18:26:25 2008
    Raid Level : raid5
    Array Size : 1953134208 (1862.65 GiB 2000.01 GB)
  Used Dev Size : 976567104 (931.33 GiB 1000.00 GB)
   Raid Devices : 3
  Total Devices : 2
Preferred Minor : 126
    Persistence : Superblock is persistent

    Update Time : Tue Sep 13 00:39:51 2011
        State : clean, degraded
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

        Layout : left-symmetric
    Chunk Size : 64K

        UUID : bebfd467:cb6700d9:29bdc0db:c30228ba
        Events : 0.671154

    Number   Major   Minor   RaidDevice State
    0   259     0       0   active sync   /dev/md/0_0p1
    1       0       0       1   removed
    2       8       33      2   active sync   /dev/sdc1

[root@localhost ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md126 : active (auto-read-only) raid5 md127p1[0] sdc1[2]
    1953134208 blocks level 5, 64k chunk, algorithm 2 [3/2] [U_U]

md127 : active (auto-read-only) raid5 sdb[1] sdd[0]
    976762496 blocks level 5, 64k chunk, algorithm 2 [2/2] [UU]

unused devices: <none>

[root@localhost ~]# mdadm --examine /dev/sdb1
/dev/sdb1:
        Magic : a92b4efc
        Version : 0.90.00
        UUID : bebfd467:cb6700d9:29bdc0db:c30228ba
  Creation Time : Wed Nov  5 18:26:25 2008
    Raid Level : raid5
  Used Dev Size : 976567104 (931.33 GiB 1000.00 GB)
    Array Size : 1953134208 (1862.65 GiB 2000.01 GB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 127

    Update Time : Tue Sep 13 00:39:51 2011
        State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0
    Checksum : f1ddd04f - correct
        Events : 671154

        Layout : left-symmetric
    Chunk Size : 64K

    Number   Major   Minor   RaidDevice State
this    0       8       17      0   active sync   /dev/sdb1

   0    0       8       17      0   active sync   /dev/sdb1
   1    1       8       49      1   active sync   /dev/sdd1
   2    2       8       33      2   active sync   /dev/sdc1

[root@localhost ~]# fdisk -lu 2>&1
Disk /dev/mapper/vg_20110912a-lv_swap doesn't contain a valid partition table
Disk /dev/mapper/vg_20110912a-lv_root doesn't contain a valid partition table
Disk /dev/md127 doesn't contain a valid partition table
Disk /dev/mapper/vg_20110912a-lv_home doesn't contain a valid partition table

Disk /dev/sda: 64.0 GB, 64023257088 bytes
255 heads, 63 sectors/track, 7783 cylinders, total 125045424 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0001aa2f

  Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1026047      512000   83  Linux
/dev/sda2         1026048   125044735    62009344   8e  Linux LVM

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xb03e1980

  Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              63  1953134504   976567221   da  Non-FS data

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x7db522d5

  Device Boot      Start         End      Blocks   Id  System
/dev/sdc1              63  1953134504   976567221   da  Non-FS data

Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x20af5840

  Device Boot      Start         End      Blocks   Id  System
/dev/sdd1              63  1953134504   976567221   da  Non-FS data

Disk /dev/mapper/vg_20110912a-lv_swap: 7348 MB, 7348420608 bytes
255 heads, 63 sectors/track, 893 cylinders, total 14352384 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_20110912a-lv_root: 30.3 GB, 30299652096 bytes
255 heads, 63 sectors/track, 3683 cylinders, total 59179008 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/md127: 2000.0 GB, 2000009428992 bytes
2 heads, 4 sectors/track, 488283552 cylinders, total 3906268416 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 131072 bytes
Disk identifier: 0x00000000


Disk /dev/md126: 1000.2 GB, 1000204795904 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953524992 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disk identifier: 0x20af5840

     Device Boot      Start         End      Blocks   Id  System
/dev/md126p1              63  1953134504   976567221   da  Non-FS data
Partition 1 does not start on physical sector boundary.

Disk /dev/mapper/vg_20110912a-lv_home: 25.8 GB, 25836912640 bytes
255 heads, 63 sectors/track, 3141 cylinders, total 50462720 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

My kernel boot parameter:

kernel /vmlinuz-2.6.40.4-5.fc15.x86_64 ro root=/dev/mapper/vg_20110912a-lv_root rd_LVM_LV=vg_20110912a/lv_root rd_LVM_LV=vg_20110912a/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us rhgb quiet rdblacklist=nouveau nouveau.modeset=0 nodmraid

The last mdadm --examine /dev/sdb1 shows exactly the same result as in Fedora 11, but I don't understand why mdadm --detail /dev/md/0_0 shows only /dev/sdb and /dev/sdd, and mdadm --detail /dev/md/127_0 shows /dev/sdc1 and /dev/md/0_0p1 .

Since mdadm --examine /dev/sdb1 shows the correct result, Fedora 15 is able to access the raid somehow, but I am not sure what to do. Shall I create/assembly a new raid /dev/md2 and hope that the lvm that I had created will magically show up?

Thank you in advance.

Improve this question
2
  • It appears that mdadm can't decide whether the raid superblock is in the partition, or the whole disk. Please add the output of fdisk -lu for the raid drives, and see if they are different in fedora 11 vs 15.
    – psusi
    Sep 13, 2011 at 15:12
  • Thanks, @psusi. The output of fdisk added to the original post (see the end of each section). However, I need some help decipher them.
    – Tsan-Kuang Lee
    Sep 14, 2011 at 6:11

1 Answer 1

Reset to default
1

It looks like you have some old crufty raid superblocks laying around. The array you were using had 3 disks and the uuid of bebfd467:cb6700d9:29bdc0db:c30228ba and was created on Nov 5, 2008. Fedora 15 has recognized another raid array that has only two disks and was created the day before, using the whole disks instead of the first partition. Fedora 15 seems to have activated that old raid array, and then tried to use that array as one of the components in the correct array, which is causing a mess.

I think you need to blow away the old, bogus superblocks:

mdadm --zero-superblock /dev/sdb /dev/sdd

You do have a current backup right? ;)

Improve this answer
3
  • Thank you very much for your response, @psusi. You do read detail carefully. Since I really don't have enough extra space to backup this much data (that's why I had Raid 5 in the first place so I have redudency), I'd like to make sure that this is the risk worth taking. After I zero the superblock of the two drives, what shall I do? Does it make them recognizable by Fedora 15? What I still don't get is how come when I specify the partition, mdadm --examine /dev/sdb1, mdadm recognize the original 3-partition raid, but not automatically. Is this why you asked me to check the superblock?
    – Tsan-Kuang Lee
    Sep 14, 2011 at 4:47
  • @Tsan-Kuang Lee, zapping the bogus superblocks and rebooting should fix everything. Also raid is not a backup solution. Raid protects against a single drive failure, there are still many ways, including plain old accidental deletion, that you can lose your data, and so you still need to back it up if you don't want to lose it. Raid is for keeping the server online when a drive dies, not protecting your data. Its about uptime, not safety.
    – psusi
    Sep 14, 2011 at 13:14
  • Thank you very much, @psusi. I am waiting to get a bigger drive to backup before I zap the superblocks, in the meantime, the problem is solved by manually editing /etc/mdadm.conf # mdadm.conf written out by anaconda MAILADDR root AUTO +imsm +1.x -all DEVICE /dev/sdb1 /dev/sdc1 /dev/sdd1 ARRAY /dev/md0 level=raid5 num-devices=3 UUID=bebfd467:cb6700d9:29bdc0db:c30228ba This temporarily solved my problem. I will do the zap after I manage to backup all the data. And thank you for the wisdom on backup. That makes a lot sense. I very much appreciate your help!
    – Tsan-Kuang Lee
    Sep 16, 2011 at 6:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .