5

I'm stumped. I had a perfectly functioning RAID1 setup on 16.10. After upgrading to 17.10, it auto-magically detected the array and re-created md0. All my files are fine, but when I mount md0, it says that the array is read-only:

cat /proc/mdstat 
Personalities : [raid1] 
md0 : active (read-only) raid1 dm-0[0] dm-1[1]
      5860390464 blocks super 1.2 [2/2] [UU]
      bitmap: 0/44 pages [0KB], 65536KB chunk

unused devices: <none>

sudo mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Sat Jul  9 23:54:40 2016
     Raid Level : raid1
     Array Size : 5860390464 (5588.90 GiB 6001.04 GB)
  Used Dev Size : 5860390464 (5588.90 GiB 6001.04 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Sat Nov  4 23:16:18 2017
          State : clean 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : x6:0  (local to host x6)
           UUID : baaccfeb:860781dd:eda253ba:6a08916f
         Events : 11596

    Number   Major   Minor   RaidDevice State
       0     253        0        0      active sync   /dev/dm-0
       1     253        1        1      active sync   /dev/dm-1

There are no errors in /var/log/kern.log nor dmesg.

I can stop it and re-assemble it, to no effect:

sudo mdadm --stop /dev/md0
sudo mdadm --assemble --scan

I do not understand why it worked perfectly before, but now the array is read-only for no reason I can detect. And this is the same array that auto-magically re-assembled when I upgraded from 16.04 to 16.10.

Researching the problem, I found a post about the problem being /sys mounted read-only, which mine indeed is:

ls -ld /sys
dr-xr-xr-x 13 root root 0 Nov  5 22:28 /sys

But neither works to fix it, as /sys stays read-only:

sudo mount -o remount,rw /sys
sudo mount -o remount,rw -t sysfs sysfs /sys
ls -ld /sys
dr-xr-xr-x 13 root root 0 Nov  5 22:29 /sys

Can anyone provide some insight that I am missing?

Edited to include /etc/mdadm/mdadm.conf:

# mdadm.conf
#
# !NB! Run update-initramfs -u after updating this file.
# !NB! This will ensure that initramfs has an uptodate copy.
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md/0  metadata=1.2 UUID=baaccfeb:860781dd:eda253ba:6a08916f name=x6:0

# This configuration was auto-generated on Sun, 05 Nov 2017 15:37:16 -0800 by mkconf

The device mapper files, which appear to be writable:

ls -l /dev/dm-*
brw-rw---- 1 root disk 253, 0 Nov  5 16:28 /dev/dm-0
brw-rw---- 1 root disk 253, 1 Nov  5 16:28 /dev/dm-1

And something else that Ubuntu or Debian has changed; I have no idea what these osprober files are doing here. I thought they were only used at installation time:

ls -l /dev/mapper/
total 0
crw------- 1 root root 10, 236 Nov  5 15:34 control
lrwxrwxrwx 1 root root       7 Nov  5 16:28 osprober-linux-sdb1 -> ../dm-0
lrwxrwxrwx 1 root root       7 Nov  5 16:28 osprober-linux-sdc1 -> ../dm-1

parted info:

sudo parted -l
Model: ATA SanDisk Ultra II (scsi)
Disk /dev/sda: 960GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name  Flags
 1      1049kB  81.9GB  81.9GB  ext4
 2      81.9GB  131GB   49.2GB  linux-swap(v1)
 3      131GB   131GB   99.6MB  fat32                 boot, esp
 4      131GB   960GB   829GB   ext4


Model: ATA WDC WD60EZRZ-00R (scsi)
Disk /dev/sdb: 6001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  6001GB  6001GB                     raid


Model: ATA WDC WD60EZRZ-00R (scsi)
Disk /dev/sdc: 6001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  6001GB  6001GB                     raid


Error: /dev/mapper/osprober-linux-sdc1: unrecognised disk label
Model: Linux device-mapper (linear) (dm)                                  
Disk /dev/mapper/osprober-linux-sdc1: 6001GB
Sector size (logical/physical): 512B/4096B
Partition Table: unknown
Disk Flags: 

Error: /dev/mapper/osprober-linux-sdb1: unrecognised disk label
Model: Linux device-mapper (linear) (dm)                                  
Disk /dev/mapper/osprober-linux-sdb1: 6001GB
Sector size (logical/physical): 512B/4096B
Partition Table: unknown
Disk Flags: 

Model: Linux Software RAID Array (md)
Disk /dev/md0: 6001GB
Sector size (logical/physical): 512B/4096B
Partition Table: loop
Disk Flags: 

Number  Start  End     Size    File system  Flags
 1      0.00B  6001GB  6001GB  ext4

Device mapper info:

$ sudo dmsetup table
osprober-linux-sdc1: 0 11721043087 linear 8:33 0
osprober-linux-sdb1: 0 11721043087 linear 8:17 0

$ sudo dmsetup info
Name:              osprober-linux-sdc1
State:             ACTIVE (READ-ONLY)
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 1
Number of targets: 1

Name:              osprober-linux-sdb1
State:             ACTIVE (READ-ONLY)
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 0
Number of targets: 1

strace output for attempt to set array to rw (with some context):

openat(AT_FDCWD, "/dev/md0", O_RDONLY)  = 3
fstat(3, {st_mode=S_IFBLK|0660, st_rdev=makedev(9, 0), ...}) = 0
ioctl(3, RAID_VERSION, 0x7fffb3813574)  = 0
fstat(3, {st_mode=S_IFBLK|0660, st_rdev=makedev(9, 0), ...}) = 0
ioctl(3, RAID_VERSION, 0x7fffb38134c4)  = 0
ioctl(3, RAID_VERSION, 0x7fffb38114bc)  = 0
fstat(3, {st_mode=S_IFBLK|0660, st_rdev=makedev(9, 0), ...}) = 0
readlink("/sys/dev/block/9:0", "../../devices/virtual/block/md0", 199) = 31
openat(AT_FDCWD, "/sys/block/md0/md/metadata_version", O_RDONLY) = 4
read(4, "1.2\n", 4096)                  = 4
close(4)                                = 0
openat(AT_FDCWD, "/sys/block/md0/md/level", O_RDONLY) = 4
read(4, "raid1\n", 4096)                = 6
close(4)                                = 0
ioctl(3, GET_ARRAY_INFO, 0x7fffb3813580) = 0
ioctl(3, RESTART_ARRAY_RW, 0)           = -1 EROFS (Read-only file system)
write(2, "mdadm: failed to set writable fo"..., 66mdadm: failed to set writable for /dev/md0: Read-only file system
) = 66
  • What does mount | grep /sys tell you about read-only or read-write for the /sys filesystem? (Ignore the permissions on the /sys directory itself.) – roaima Nov 6 '17 at 21:31
6

This won’t explain why your array ended up in read-only mode, but

mdadm --readwrite /dev/md0

should return it to normal. In your case it doesn’t, and the reason isn’t entirely obvious: if constituent devices are themselves read-only, the RAID array is read-only (which is what matches the behaviour you’re seeing, and the code-paths used when you try to re-enable read-write).

The dmsetup table information gives a strong hint as to what’s going on: osprober (I imagine, given the names of the devices) is finding the real RAID components, and for some reason it’s creating device mapper devices on top of them, and these are being picked up and used for the RAID device. Since the only device mapper devices are those two osprober devices, the simplest solution is to stop the RAID device, stop the DM devices, and re-scan the RAID array so that the underlying components devices are used. To stop the DM devices, run

dmsetup remove_all

as root.

  • Could you please elaborate a bit on the solution? I think I'm in a similar position after upgrade to 18.04, with two osprober-linux-... devices in dmsetup table (plus some more which seem to be OK) and a raid1 on these two devices, which won't mount as readwrite. I would like to understand the whole problem a bit more before I start making any changes. – Ondrej Nov 25 '18 at 20:48
  • No matter, I've tried it and it worked - thank you very much! – Ondrej Nov 29 '18 at 20:34
  • 1
    Oh, sorry, I was trying to remember the details of the various steps. I’m glad you figured things out! – Stephen Kitt Nov 29 '18 at 21:57
  • Yes I did what you wrote - stopped the RAID, removed the two devices with dmsetup remove <device name> since there were also devices for a logical volume (RAID0), rescanned - everything fine. However, after reboot the RAID1 is working fine but the RAID0 lv is missing, the dm table is empty. No idea why, it worked just before reboot. I have the table backed up, thinking of just entering the missing devices back via dmsetup. Is it a good idea? – Ondrej Dec 1 '18 at 11:31
  • Sorry, it's not RAID0, it's just a linear device created with lvm2... – Ondrej Dec 1 '18 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.