3

I just upgraded my box to 14.04 and on bootup I got a message, that my RAID devices can't be mounted automatically so I chose to skip it and work through it after boot time. Now booted up, A mount -a gives me:

$ sudo mount -a
[sudo] password for reg: 
mount: wrong fs type, bad option, bad superblock on /dev/md0,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

mount: wrong fs type, bad option, bad superblock on /dev/md1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

$

while my mdadm.conf looks like:

$ cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# next line added by reg 09/01/2009
ARRAY /dev/md0 devices=/dev/sdb6,/dev/sdc6
ARRAY /dev/md1 devices=/dev/sdb7,/dev/sdc7
#ARRAY /dev/md2 devices=/dev/sdb8,/dev/sdc8

# definitions of existing MD arrays

# This file was auto-generated on Tue, 08 May 2012 18:26:22 -0700
# by mkconf $Id$
$

just quickly started gparted to check if all my dirves show up and fair enough, my partitions 6 & 7 show up just fine on drives sdb and sdc. Also, seems like mdadm is running just fine:

$ ps ax | grep mdadm
 1945 ?        Ss     0:00 /sbin/mdadm --monitor --pid-file /run/mdadm/monitor.pid --daemonise --scan --syslog
 7869 pts/8    S+     0:00 grep --color=auto mdadm
$

what could be my problem here, I don't quite follow.

$ dmesg | tail
[ 1437.716819] EXT4-fs (md0): bad geometry: block count 61034935 exceeds size of device (61034912 blocks)
[ 1437.717197] EXT4-fs (md1): mounting ext3 file system using the ext4 subsystem
[ 1437.717422] EXT4-fs (md1): bad geometry: block count 59585077 exceeds size of device (59585056 blocks)
[ 3030.508560] init: nvidia-persistenced main process (21110) terminated with status 1
[ 3757.003367] current rate 16719823 is different from the runtime rate 48000
[ 3757.006356] current rate 10357448 is different from the runtime rate 48000
[ 3850.605129] EXT4-fs (md0): mounting ext3 file system using the ext4 subsystem
[ 3850.605299] EXT4-fs (md0): bad geometry: block count 61034935 exceeds size of device (61034912 blocks)
[ 3850.605585] EXT4-fs (md1): mounting ext3 file system using the ext4 subsystem
[ 3850.605782] EXT4-fs (md1): bad geometry: block count 59585077 exceeds size of device (59585056 blocks)
reg@regDesktopHome:~/semios$

and syslog says nothing but

$ sudo mdadm --detail --scan
mdadm: cannot open /dev/md/127_0: No such file or directory
ARRAY /dev/md1 metadata=0.90 UUID=0abe503f:401d8d09:cd707cc0:bc3f8165
ARRAY /dev/md0 metadata=0.90 UUID=91e560f1:4e51d8eb:cd707cc0:bc3f8165
reg@regDesktopHome:~/semios$

edit 1

$ cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sdb6[1] sdc6[0]
      244139648 blocks [2/2] [UU]

md1 : active raid1 sdb7[1] sdc7[0]
      238340224 blocks [2/2] [UU]

md127 : active raid1 sdb3[1] sdc3[0]
      390628416 blocks [2/2] [UU]

unused devices: <none>
$ 

$ sudo mdadm --detail /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : Tue Sep  1 19:15:33 2009
     Raid Level : raid1
     Array Size : 244139648 (232.83 GiB 250.00 GB)
  Used Dev Size : 244139648 (232.83 GiB 250.00 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sun May  4 21:32:24 2014
          State : clean 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 91e560f1:4e51d8eb:cd707cc0:bc3f8165
         Events : 0.75007

    Number   Major   Minor   RaidDevice State
       0       8       38        0      active sync   /dev/sdc6
       1       8       22        1      active sync   /dev/sdb6
reg@regDesktopHome:~/semios$ sudo mdadm --detail /dev/md1
/dev/md1:
        Version : 0.90
  Creation Time : Tue Sep  8 21:19:15 2009
     Raid Level : raid1
     Array Size : 238340224 (227.30 GiB 244.06 GB)
  Used Dev Size : 238340224 (227.30 GiB 244.06 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Sun May  4 21:32:24 2014
          State : clean 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 0abe503f:401d8d09:cd707cc0:bc3f8165
         Events : 0.3973383

    Number   Major   Minor   RaidDevice State
       0       8       39        0      active sync   /dev/sdc7
       1       8       23        1      active sync   /dev/sdb7
$ 

$ sudo mdadm --examine /dev/sdb6
/dev/sdb6:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 91e560f1:4e51d8eb:cd707cc0:bc3f8165
  Creation Time : Tue Sep  1 19:15:33 2009
     Raid Level : raid1
  Used Dev Size : 244139648 (232.83 GiB 250.00 GB)
     Array Size : 244139648 (232.83 GiB 250.00 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Sun May  4 21:32:24 2014
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : bfa701ff - correct
         Events : 75007


      Number   Major   Minor   RaidDevice State
this     1       8       22        1      active sync   /dev/sdb6

   0     0       8       38        0      active sync   /dev/sdc6
   1     1       8       22        1      active sync   /dev/sdb6
reg@regDesktopHome:~/semios$ sudo mdadm --examine /dev/sdb7
/dev/sdb7:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 0abe503f:401d8d09:cd707cc0:bc3f8165
  Creation Time : Tue Sep  8 21:19:15 2009
     Raid Level : raid1
  Used Dev Size : 238340224 (227.30 GiB 244.06 GB)
     Array Size : 238340224 (227.30 GiB 244.06 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1

    Update Time : Sun May  4 21:32:24 2014
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 2a7376fd - correct
         Events : 3973383


      Number   Major   Minor   RaidDevice State
this     1       8       23        1      active sync   /dev/sdb7

   0     0       8       39        0      active sync   /dev/sdc7
   1     1       8       23        1      active sync   /dev/sdb7
reg@regDesktopHome:~/semios$ sudo mdadm --examine /dev/sdc6
/dev/sdc6:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 91e560f1:4e51d8eb:cd707cc0:bc3f8165
  Creation Time : Tue Sep  1 19:15:33 2009
     Raid Level : raid1
  Used Dev Size : 244139648 (232.83 GiB 250.00 GB)
     Array Size : 244139648 (232.83 GiB 250.00 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Sun May  4 21:32:24 2014
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : bfa7020d - correct
         Events : 75007


      Number   Major   Minor   RaidDevice State
this     0       8       38        0      active sync   /dev/sdc6

   0     0       8       38        0      active sync   /dev/sdc6
   1     1       8       22        1      active sync   /dev/sdb6
reg@regDesktopHome:~/semios$ sudo mdadm --examine /dev/sdc7
/dev/sdc7:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 0abe503f:401d8d09:cd707cc0:bc3f8165
  Creation Time : Tue Sep  8 21:19:15 2009
     Raid Level : raid1
  Used Dev Size : 238340224 (227.30 GiB 244.06 GB)
     Array Size : 238340224 (227.30 GiB 244.06 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1

    Update Time : Sun May  4 21:32:24 2014
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 2a73770b - correct
         Events : 3973383


      Number   Major   Minor   RaidDevice State
this     0       8       39        0      active sync   /dev/sdc7

   0     0       8       39        0      active sync   /dev/sdc7
   1     1       8       23        1      active sync   /dev/sdb7
$ 

$ sudo parted /dev/sdb unit mib print free
Model: ATA ST31000528AS (scsi)
Disk /dev/sdb: 953870MiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start      End        Size       Type      File system     Flags
 1      0.03MiB    95370MiB   95370MiB   primary   ext4            boot
 3      95370MiB   476843MiB  381473MiB  primary   ext4
 2      476843MiB  953859MiB  477016MiB  extended
 6      476843MiB  715261MiB  238418MiB  logical   ext3
 7      715261MiB  948015MiB  232754MiB  logical   ext3
 5      948015MiB  953859MiB  5844MiB    logical   linux-swap(v1)
        953859MiB  953870MiB  10.3MiB              Free Space

reg@regDesktopHome:~/semios$ sudo parted /dev/sdc unit mib print free
Model: ATA ST31000528AS (scsi)
Disk /dev/sdc: 953870MiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start      End        Size       Type      File system     Flags
 1      0.03MiB    95370MiB   95370MiB   primary   ext4            boot
 3      95370MiB   476843MiB  381473MiB  primary   ext4
 2      476843MiB  953859MiB  477016MiB  extended
 6      476843MiB  715261MiB  238418MiB  logical   ext3
 7      715261MiB  948015MiB  232754MiB  logical   ext3
 5      948015MiB  953859MiB  5844MiB    logical   linux-swap(v1)
        953859MiB  953870MiB  10.3MiB              Free Space

$

edit 2

I tried the suggested but my file system wouldn't mount:

# sudo losetup --find --show --read-only /dev/sdc7 /dev/loop0
# mount -o ro /dev/loop0 /mnt/loop0 mount: unknown filesystem type 'linux_raid_member'
#

edit 3 file -s /dev/loop0 returns me the following:

$ sudo file -s /dev/loop0
[sudo] password for reg: 
/dev/loop0: Linux rev 1.0 ext3 filesystem data, UUID=b2d66029-eeb9-4e4a-952c-0a3bd0696159 (large files)
  • 1
    need a lot more info. /proc/mdstat, mdadm --detail /dev/md?, mdadm --examine /dev/sdx?, parted /dev/sdx unit mib print free, ... – frostschutz May 5 '14 at 7:26
  • @frostschutz Thanks for your reply! I added some more info above under edit 1 – cerr May 5 '14 at 13:18
3

bad geometry: block count 61034935 exceeds size of device (61034912 blocks)

According to these error messages, it seems your RAID shrank by a few blocks; which is an unusual problem considering that 0.90 metadata is at the end, and ext3 metadata is at the beginning of a partition, so the partition size itself cannot have changed, as such a change on either end would lead to missing metadata.

I can't imagine how this happened; in order to break things this way intentionally, you'd have to do something like running resize2fs on the member disks instead of the RAID itself; or start out without RAID and add it only as an afterthought.

Since it's RAID1 and the raid metadata is at the end of the disk, you should be able to mount either disk directly. A simple way to test this would be with a read-only loop device. Provided the filesystem does not have a defective/dirty superblock as well, this should do:

# losetup --find --show --read-only /dev/sdc7
/dev/loop4
# mount -o ro /dev/loop4 /mnt/test

If that works and the files are intact, it would be a good time to backup.

If the filesystem requires fsck as well, the safest way to do so is to make a copy of the entire partition with dd and then fsck the image.

# dd if=/dev/sdc7 of=/mnt/ext/sdc7.img bs=1M
# losetup --find --show /mnt/ext/sdc7.img
/dev/loop5
# fsck /dev/loop5
# mount /dev/loop5 /mnt/test

If that works, you might be able to fix your RAID in place using these steps:

  1. remove the other drive:

    # mdadm /dev/md0 --fail /dev/sdb7
    # mdadm /dev/md0 --remove /dev/sdb7
    
  2. fix and resize the filesystem on the remaining drive. block size taken from the error message you provided

    # fsck /dev/sdc7
    # resize2fs /dev/sdc7 61034912
    # sync
    
  3. test if it worked

    # echo 3 > /proc/sys/vm/drop_caches
    # mount /dev/md0 /mnt/somewhere
    
  4. readd and resync the other drive

    # mdadm /dev/md0 --add /dev/sdb7
    

However, you might consider re-creating the RAID from scratch with current metadata (1.2) and remaking the filesystem and copying the files back onto it instead.

  • Hi, Thanks for the suggestion but unfortunately my partition wouldn't mount, please see edit 2 above. Thanks! – cerr May 8 '14 at 4:59
  • @cerr, mount might refuse to mount raid members because normally, you should be mounting the raid itself instead. if file -s /dev/loop0 says it's a filesystem, try adding -t fstype. – frostschutz May 8 '14 at 18:17
  • please see edit3 above. Thanks! – cerr May 24 '14 at 23:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.