5

I use SSD and NVME RAID1 arrays to store mostly virtual machine disks. More than 75% of the data are zeros (preallocated images, free space).

If a disk fails and gets replaced, the rebuild copies and writes all the data to the replacement disk, which causes thermal throttling in NVME, and I assume more wear on the SSD/NVME. Is there a way to configure the rebuild to compare data from both disks first and write to the new disk only if needed?

Or are SSD/NVME chips supposed to check whether the data are just zeros and if the target blocks are not written yet (giving zeros on read), it would just discard the data without wasting write cycles? Or if there is any target data should it just trim the block to produce zeros?

I found an old thread at https://www.spinics.net/lists/raid/msg57529.html but it did not provide an answer.

I tried a workaround but I think it is ugly plus the RAID must be offline.

mdadm --fail /dev/md0 /dev/sde
mdadm -r /dev/md0 /dev/sde

(replace /dev/sde)

mdadm -S /dev/md0
ddpt if=/dev/sdd of=/dev/sde verbose=1 oflag=sparing
mdadm -C -v /dev/md0 --assume-clean -l 1 -n 2 /dev/sdd /dev/sde

Any ideas for compare-write RAID1 rebuild? Thanks.

Improve this question
2
  • I participated in that thread, if there was any change I'm not aware of it. There is no SSD/NVME friendly MD RAID rebuild. You'd have to hack it yourself which can be done but probably not worth the bother. If you experience thermal throttling, the practical solution here might be to provide additional cooling. There is no magic in RAID rebuilds so if there's throttling during rebuild, there will be throttling doing other I/O.
    – frostschutz
    Commented Oct 9, 2020 at 11:18
  • I experimented with repair, seems not possible to control the repair direction - uses device 0 even if flagged writemostly, thus actively changing the data previously returned by the array - even if you used it, it'd still be an offline operation. All in all, the hack you already have seems more reliable...
    – frostschutz
    Commented Oct 14, 2020 at 19:52

2 Answers 2

Reset to default
2
+25

The short answer is: No.

The md-driver is tuned for performance. And it has a simply dirty-map to keep raid 1 members in sync.

So if a member fails, the whole map is dirty. Since md is just block-based it does not care for the contents of the blocks it just copies over blocks and clears the dirty-bits for these block.

KISS principle_ keep it simple stupid. Anything else would be on a higher level.

If you want that you could use drbd with two local members instead of md raid1. DRBD provides the means to verify before syncing.

Improve this answer
2
  • The read speed of my devices is 2x faster than the write speed, so if the amount of nonzero data is not large, the faster way would be compare-write than simple copy. And I might care more for wear of the disk than just speed. I think that compare-write should be a non-default option. If the inconsistent array is build by --assume-clean, then I think 'echo repair > /sys/block/md0/md/sync_action' will do the compare-write. So the code seems to be there, I wonder if anybody else would consider adding a hot-add option in compare-write rebuild mode to be a good idea.
    – John Lee McMahon
    Commented Oct 13, 2020 at 20:43
  • @JohnLeeMcMahon the repair idea is interesting, it might be possible to exploit that, but it's also very dangerous
    – frostschutz
    Commented Oct 14, 2020 at 19:28
2

So the only way to do compare-write rebuild in MDRAID1 (apart from syncing offline by e.g. ddpt) seems to be the repair (the array has to be stopped and reconfigured when replacing the disk, the good disk must go first in the new array to become the ID 0 and the data source)

mdadm -C -v /dev/md0 -l 1 -n 2 /dev/sdd /dev/sde
mdadm /dev/md0 -f /dev/sdd -r /dev/sdd
mdadm -S /dev/md0
mdadm -C -v /dev/md0 -l 1 -n 2 --assume-clean /dev/sde /dev/sdd
echo repair > /sys/block/md0/md/sync_action

It seems to work for multiple mirrors too:

mdadm -C -v /dev/md0 -l 1 -n 3 /dev/sdd /dev/sde /dev/sdf
mdadm /dev/md0 -f /dev/sde -f /dev/sdf -r /dev/sde -r /dev/sdf
mdadm -S /dev/md0
mdadm -C -v /dev/md0 -l 1 -n 3 --assume-clean /dev/sdd /dev/sde /dev/sdf
echo repair > /sys/block/md0/md/sync_action

I also experimented with ideas of reshaping the array without stopping (RAID0 -> RAID1_or_RAID10 -> RAID0), but I did not see compare-write in RAID10 (apparently used for RAID10 per the original linux-raid thread) and things do not work as I expected in RAID0 -> RAID1 conversions.

Full copy, but works:

mdadm -C -v /dev/md0 -l 0 -n 2 /dev/sdd /dev/sde
mdadm -G /dev/md0 -n 4 -l 10 -a /dev/sdf -a /dev/sdg
mdadm /dev/md0 -f /dev/sdf -f /dev/sdg -r /dev/sdf -r /dev/sdg
mdadm -G /dev/md0 -n 2 -l 0

Full copy, also works:

mdadm -C -v /dev/md0 -l 0 -n 3 /dev/sdd /dev/sde /dev/sdf
mdadm -G /dev/md0 -n 6 -l 10 -a /dev/sdg -a /dev/sdh -a /dev/sdi
mdadm /dev/md0 -f /dev/sdg -f /dev/sdh -f /dev/sdi -r /dev/sdg -r /dev/sdh -r /dev/sdi
mdadm -G /dev/md0 -n 3 -l 0

But problems here:

mdadm -C -v /dev/md0 -l 1 -n 2 /dev/sdd /dev/sde
mdadm /dev/md0 -f /dev/sde -r /dev/sde
mdadm -G /dev/md0 -n 1 -l 0

Cannot change number of disks in RAID1->RAID0 conversion

mdadm -G /dev/md0 -l 0

but without -n 1 it works

mdadm -G /dev/md0 -l 1 -n 2 --force -a /dev/sde

Impossible level change requested

mdadm -C -v --force /dev/md0 -l 0 -n 1 /dev/sdd
mdadm -G /dev/md0 -l 1 -n 2 --force -a /dev/sde

Impossible level change requested

Improve this answer
2
  • Note that even if you let mdadm "repair" this way, it's still an offline operation. As soon as you --create --assume-clean, the array might read (false) data from the additional drives. You can reduce this risk by flagging those drives --write-mostly but the risk is not 0. Plus there are corner cases such as what happens when the main drive encounters read error during repair etc. All in all this approach is interesting but not in any way reliable.
    – frostschutz
    Commented Oct 16, 2020 at 10:36
  • Also consider being more specific in the re-create command. Data offsets and other defaults have changed in the past and might change again in the future, the offset also regularly changes with some --grow operations, so you might encounter failure with this method at some point. Also consider using partitions instead of full disk for the array itself.
    – frostschutz
    Commented Oct 16, 2020 at 10:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .