0

I've created a virtual machine running on KVM only for testing and learning purposes. During the installation RAID 1 array has been configured with 3 disks for root and 3 for boot. After some playing and testing I've decide to write zeroes to one of the drives and check what will happens:

dd if=/dev/zero of=/dev/vdc2 

After while the system gone to read-only, but there weren't any errors in mdamd.

dmesg:

[ 2177.091939] RAID1 conf printout:
[ 2177.091947]  --- wd:2 rd:3
[ 2177.091954]  disk 0, wo:0, o:1, dev:vda2
[ 2177.091956]  disk 1, wo:0, o:1, dev:vdb2
[ 2177.091958]  disk 2, wo:1, o:1, dev:vdc2
[ 2177.095315] md: recovery of RAID array md1
[ 2177.095321] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 2177.095323] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[ 2177.095330] md: using 128k window, over a total of 9792512k.
[ 2217.132610] RAID1 conf printout:
[ 2217.132616]  --- wd:2 rd:3
[ 2217.132622]  disk 0, wo:0, o:1, dev:vda1
[ 2217.132625]  disk 1, wo:0, o:1, dev:vdb1
[ 2217.132626]  disk 2, wo:1, o:1, dev:vdc1
[ 2217.135129] md: delaying recovery of md0 until md1 has finished (they share one or more physical units)
[ 2225.567664] md: md1: recovery done.
[ 2225.572072] md: recovery of RAID array md0
[ 2225.572081] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 2225.572083] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[ 2225.572087] md: using 128k window, over a total of 682432k.
[ 2225.574833] RAID1 conf printout:
[ 2225.574836]  --- wd:3 rd:3
[ 2225.574904]  disk 0, wo:0, o:1, dev:vda2
[ 2225.574906]  disk 1, wo:0, o:1, dev:vdb2
[ 2225.574908]  disk 2, wo:0, o:1, dev:vdc2
[ 2229.036805] md: md0: recovery done.
[ 2229.042732] RAID1 conf printout:
[ 2229.042736]  --- wd:3 rd:3
[ 2229.042740]  disk 0, wo:0, o:1, dev:vda1
[ 2229.042742]  disk 1, wo:0, o:1, dev:vdb1
[ 2229.042744]  disk 2, wo:0, o:1, dev:vdc1
[ 5241.129626] md/raid1:md1: Disk failure on vdc2, disabling device.
               md/raid1:md1: Operation continuing on 2 devices.
[ 5241.131639] RAID1 conf printout:
[ 5241.131642]  --- wd:2 rd:3
[ 5241.131645]  disk 0, wo:0, o:1, dev:vda2
[ 5241.131647]  disk 1, wo:0, o:1, dev:vdb2
[ 5241.131648]  disk 2, wo:1, o:0, dev:vdc2
[ 5241.131655] RAID1 conf printout:
[ 5241.131656]  --- wd:2 rd:3
[ 5241.131658]  disk 0, wo:0, o:1, dev:vda2
[ 5241.131684]  disk 1, wo:0, o:1, dev:vdb2
[ 5326.850032] md: unbind<vdc2>
[ 5326.850050] md: export_rdev(vdc2)
[ 5395.301755] md: export_rdev(vdc2)
[ 5395.312985] md: bind<vdc2>
[ 5395.315022] RAID1 conf printout:
[ 5395.315024]  --- wd:2 rd:3
[ 5395.315027]  disk 0, wo:0, o:1, dev:vda2
[ 5395.315029]  disk 1, wo:0, o:1, dev:vdb2
[ 5395.315031]  disk 2, wo:1, o:1, dev:vdc2
[ 5395.318161] md: recovery of RAID array md1
[ 5395.318168] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 5395.318170] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[ 5395.318174] md: using 128k window, over a total of 9792512k.
[ 5443.707445] md: md1: recovery done.
[ 5443.712678] RAID1 conf printout:
[ 5443.712682]  --- wd:3 rd:3
[ 5443.712686]  disk 0, wo:0, o:1, dev:vda2
[ 5443.712688]  disk 1, wo:0, o:1, dev:vdb2
[ 5443.712689]  disk 2, wo:0, o:1, dev:vdc2
[ 8017.777012] EXT4-fs error (device md1): ext4_lookup:1584: inode #36: comm systemd-sysv-ge: deleted inode referenced: 135
[ 8017.782244] Aborting journal on device md1-8.
[ 8017.785487] EXT4-fs (md1): Remounting filesystem read-only
[ 8017.876415] EXT4-fs error (device md1): ext4_lookup:1584: inode #36: comm systemd: deleted inode referenced: 137

cat /proc/mdstat:

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid1 vdb2[1] vda2[0]
      9792512 blocks super 1.2 [3/2] [UU_]

md0 : active raid1 vdc1[2] vdb1[1] vda1[0]
      682432 blocks super 1.2 [3/3] [UUU]

unused devices: <none>

I've tried to mount back the root as read-write without success:

mount -o remount /

Segmentation fault (core dumped)

then:

fsck -Af

fsck from util-linux 2.27.1
Segmentation fault (core dumped)

I was hopping that without to remove the vdc2 drive I will successfully re-balancing it, but I was wrong. The broken drive has been removed:

mdadm --manage /dev/md1 --fail /dev/vdc2
mdadm --manage /dev/md1 --remove /dev/vdc2

And try to delete and create the drive again using fdisk or cfdisk but I've got the same error: Segmentation fault (core dumped)

I am pasting the status of md1 and the drive using mdadm:

mdadm -D /dev/md1

/dev/md1:
        Version : 1.2
  Creation Time : Mon Nov  7 21:22:29 2016
     Raid Level : raid1
     Array Size : 9792512 (9.34 GiB 10.03 GB)
  Used Dev Size : 9792512 (9.34 GiB 10.03 GB)
   Raid Devices : 3
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Tue Nov  8 02:38:26 2016
          State : clean, degraded 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : ubuntu-raid:1  (local to host ubuntu-raid)
           UUID : c846618f:d77238fe:95edac3d:dd19e295
         Events : 108

    Number   Major   Minor   RaidDevice State
       0     253        2        0      active sync   /dev/vda2
       1     253       18        1      active sync   /dev/vdb2
       4       0        0        4      removed

mdadm -E /dev/vdc2

/dev/vdc2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : c846618f:d77238fe:95edac3d:dd19e295
           Name : ubuntu-raid:1  (local to host ubuntu-raid)
  Creation Time : Mon Nov  7 21:22:29 2016
     Raid Level : raid1
   Raid Devices : 3

 Avail Dev Size : 19585024 (9.34 GiB 10.03 GB)
     Array Size : 9792512 (9.34 GiB 10.03 GB)
    Data Offset : 16384 sectors
   Super Offset : 8 sectors
   Unused Space : before=16296 sectors, after=0 sectors
          State : clean
    Device UUID : 25a823f7:a301598a:91f9c66b:cc27d311

    Update Time : Tue Nov  8 02:20:34 2016
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : d6d7fc77 - correct
         Events : 101


   Device Role : Active device 2
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)

OS: Ubuntu 16.04.1 Kernel: 4.4.0-31-generic mdadm version: v3.3

So I have a two questions: why this happen, what is the main reason to be not able to mount the array read-write and the second question is how to prevent this in future. Of course this is a testing environment but I am looking for a way to fix it without rebooting or something similar

  • filling one active device of a raid array with zeroes is not a real life failure simulation. – Ipor Sircer Nov 8 '16 at 0:57
  • I've just search a way to broke the system and this attempt was successful on 100%. The system can go to read-only with a lot of scenarios. – ValeriRangelov Nov 8 '16 at 1:01
1

The Linux md system depends on the component drives of your RAID array to either provide good data, or provide no data. In a real-world failure situation, this is a reasonable assumption: disks have error-correction information on them, and it is extremely unlikely that a bad sector will corrupt itself in a way that can't be detected.

By writing zeroes to a disk, you bypass this protection. The md system thinks the data is still good, and it passes the corrupt data on up to the filesystem layer, which reacts badly. Since you're using RAID 1, md will balance reads across all drives to increase performance; the crashes you're getting are because pieces of mount and fsck are getting read from the bad drive.

To recover, remove the failed disk from the system entirely (since you're using a VM, do this using the VM's management tools; if this was a physical system, you'd unplug the drive). This will force the md system to realize the drive has failed and stop reading from it; you can then perform any filesystem-level recovery needed.

If you want to play this sort of game with your disks, format them with ZFS or BTRFS: these filesystems don't make the assumption of "good data or no data", and use checksums to spot bad data being read from the disk.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.