My Ubuntu RAID configuration changes during a reboot and I don't understand why.
- Problem 1: /dev/md/host:name device is not created at boot. I don't understand how, when, or why the descriptive names are created and it would be better if it were more predictable.
This array always gets a device name:
jak # mdadm --detail /dev/md127 [...] Name : jak:neat (local to host jak) UUID : 593fc406:87eefd53:0a076a84:f1405112
This array almost never gets a device name:
jak # mdadm --detail /dev/md130 [...] Name : jak:sour (local to host jak) UUID : 809a185b:a2613844:3975b412:759ec297
My understanding is that the purpose of these human-readable names is for use in /etc/fstab:
jak # tail -2 /etc/fstab /dev/md130 /jak/data/sour ext4 defaults 0 0 /dev/md/jak:neat /jak/data/neat ext4 defaults 0 0
You can see that for jak:sour I am mounting /dev/md130 explicitly which is problematic since that device name changes occasionally. I also don't understand why the device number changes. What is the reliable /dev entry that I should use in fstab?
- Problem 2: My spare drives fall out of the arrays each reboot.
Both jak:neat and jak:sour have spare drives (mdadm output is below). After every reboot, the spare drives vanish from the arrays. The disk devices do appear in /dev/sd* and I can re-attach them to the arrays easily enough (mdadm --add /dev/md127 /dev/sdn1) but obviously I'd rather they not fall out of the array.
Groveling: Google and Stack Exchange searches for specific linux software raid array problems is kind of a miserable exercise in signal to noise. I hope these are good questions to ask.
Config details:
# uname -a Linux jak 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
jak # mdadm --detail /dev/md127 /dev/md127: Version : 1.2 Creation Time : Fri Jun 1 07:33:21 2018 Raid Level : raid10 Array Size : 23441682432 (22355.73 GiB 24004.28 GB) Used Dev Size : 7813894144 (7451.91 GiB 8001.43 GB) Raid Devices : 6 Total Devices : 7 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Sun Oct 6 07:02:47 2019 State : clean, checking Active Devices : 6 Working Devices : 7 Failed Devices : 0 Spare Devices : 1 Layout : near=2 Chunk Size : 512K Consistency Policy : bitmap Check Status : 18% complete Name : jak:neat (local to host jak) UUID : 593fc406:87eefd53:0a076a84:f1405112 Events : 201984 Number Major Minor RaidDevice State 0 8 17 0 active sync set-A /dev/sdb1 1 65 1 1 active sync set-B /dev/sdq1 2 8 81 2 active sync set-A /dev/sdf1 3 65 33 3 active sync set-B /dev/sds1 4 8 49 4 active sync set-A /dev/sdd1 6 65 17 5 active sync set-B /dev/sdr1 7 8 177 - spare /dev/sdl1 jak #
jak # mdadm --detail /dev/md130 /dev/md130: Version : 1.2 Creation Time : Sat May 26 10:51:23 2018 Raid Level : raid6 Array Size : 39065217024 (37255.49 GiB 40002.78 GB) Used Dev Size : 9766304256 (9313.87 GiB 10000.70 GB) Raid Devices : 6 Total Devices : 7 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Sun Oct 6 07:03:43 2019 State : clean, checking Active Devices : 6 Working Devices : 7 Failed Devices : 0 Spare Devices : 1 Layout : left-symmetric Chunk Size : 512K Consistency Policy : bitmap Check Status : 41% complete Name : jak:sour (local to host jak) UUID : 809a185b:a2613844:3975b412:759ec297 Events : 139649 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 33 1 active sync /dev/sdc1 2 8 129 2 active sync /dev/sdi1 3 8 145 3 active sync /dev/sdj1 4 8 161 4 active sync /dev/sdk1 6 8 241 5 active sync /dev/sdp1 7 8 209 - spare /dev/sdn1 jak #
I don't suspect /dev/sdn is suspicious but here are the details anyway.
jak # smartctl -a /dev/sdn smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-65-generic] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: WDC WD100EFAX-68LHPN0 Serial Number: JEKG3TLZ LU WWN Device Id: 5 000cca 267f04f25 Firmware Version: 83.H0A83 User Capacity: 10,000,831,348,736 bytes [10.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Oct 6 07:04:58 2019 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 93) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: (1116) minutes. SCT capabilities: (0x003d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0 2 Throughput_Performance 0x0004 131 131 054 Old_age Offline - 104 3 Spin_Up_Time 0x0007 100 100 024 Pre-fail Always - 0 4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 3 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0 7 Seek_Error_Rate 0x000a 100 100 067 Old_age Always - 0 8 Seek_Time_Performance 0x0004 128 128 020 Old_age Offline - 18 9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 1148 10 Spin_Retry_Count 0x0012 100 100 060 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 3 22 Unknown_Attribute 0x0023 100 100 025 Pre-fail Always - 100 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 51 193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 51 194 Temperature_Celsius 0x0002 144 144 000 Old_age Always - 45 (Min/Max 20/54) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. jak #
# mdadm.conf # # !NB! Run update-initramfs -u after updating this file. # !NB! This will ensure that initramfs has an uptodate copy. # # Please refer to mdadm.conf(5) for information about this file. # # by default (built-in), scan all partitions (/proc/partitions) and all # containers for MD superblocks. alternatively, specify devices to scan, using # wildcards if desired. #DEVICE partitions containers # automatically tag new arrays as belonging to the local system HOMEHOST # instruct the monitoring daemon where to send mail alerts MAILADDR root # definitions of existing MD arrays ARRAY /dev/md/0 metadata=1.2 UUID=7636d5d0:8c9d0823:7252c563:30724789 name=jak:0 ARRAY /dev/md130 metadata=1.2 UUID=809a185b:a2613844:3975b412:759ec297 name=jak:sour # This configuration was auto-generated on Sat, 28 Apr 2018 16:34:38 -0700 by mkconf
Remaining question now
/etc/md/jak:neat
and/etc/md/jak:sour
do not appear. What process creates those?running
blkid /dev/md127
returns no output. It does return expected output against/dev/md0
. Therefore, myfstab
now lists the file system devices as/dev/md127
and/dev/md130
which feels temporary.
Comments appreciated in advance!