2

About 10 days ago, my RAID 1 managed with mdadm had a failure. The array consists of 2 external 2 TB hard drives which are connected via USB and I suspect there was a loose connection which lead to a failure.

According to my bash history, I entered these commands:

sudo mdadm --manage /dev/md1 --fail /dev/sda1
sudo mdadm --manage /dev/md1 --remove /dev/sda1
cat /proc/mdstat
sudo mdadm --manage /dev/md1 --add /dev/sda1
cat /proc/mdstat
mailx
cat /proc/mdstat

I found instructions for this online but iirc, the first and/or second command failed because /dev/sda1 was already not part of the array anymore or something like that.

Data definitely was written to the file system of the array between the failure and the point where I entered the commands above.

I assumed mdadm would copy all the data of /dev/sdb1 over to /dev/sda1 after I issued the 4th command which would've taken quite some time. However, when I checked back in the last line after a few minutes, it was already done.

Not knowing how it managed to apparently only copy the changed data over, I feared for the integrity of the data stored on my RAID and sha256summed /dev/sda1 and /dev/sdb1 except for the first 128 MiB because I know the first 128 MiB are different. Turns out the rest of the disks store exactly the same data respectively, so mdadm did indeed restore the integrity of the array.

How did it know what parts changed? Would it still have worked had I rebooted between the point in time where the failure occurred and the point in time I issued the commands above?

1

cat /proc/mdstat should show you that the respective RAID has a bitmap. That is similar to the journal of a filesystem: When the system crashes (or the RAID breaks) the sync function has a place where it can check which data can be damaged. Everything else can just be ignored. Thus the rebuild is very fast.

meaning of the bitmap line

from https://raid.wiki.kernel.org/index.php/Mdstat

bitmap: 0/10 pages [0KB], 16384KB chunk

What would it mean when it's, eg: 23/234

This refers to the in-memory bitmap (basically a cache of what's in the on-disk bitmap − it allows bitmap operations to be more efficient).

If it's 23/234 that means there are 23 of 234 pages allocated in the in-memory bitmap. The pages are allocated on demand, and get freed when they're empty (all zeroes). The in-memory bitmap uses 16 bits for each bitmap chunk to count all ongoing writes to the chunk, so it's actually up to 16 times larger than the on-disk bitmap.

  • It prints "bitmap: 0/15 pages [0KB], 65536KB chunk". I'm not sure how to interpret raid.wiki.kernel.org/index.php/Mdstat#bitmap_line. Does this mean that there is 1 bit for every 64 MiB of storage space? Does it mean there are 16 bit for every 64 MiB? Do you know how I can find out what size the bitmap is? I think it's one created automatically because my array is bigger than 100 GiB and the man page says it then creates one automatically if not otherwise specified but it doesn't say anything about the properties it chooses. – UTF-8 Sep 17 '17 at 17:43
  • @UTF-8 See my edit – Hauke Laging Sep 17 '17 at 19:23
  • I read that. I even linked to the exact section you quoted that from in my comment. – UTF-8 Sep 17 '17 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.