5

I currently have 1 TB of data on a RAID-1 mdadm array. This array consists of one 1 TB disk and a 1 TB partition on a 2 TB disk. I have now bought a second 1 TB disk, and I would like to use this to create a 2 TB RAID-1 array.

The data on this array will be encrypted with dm-crypt, so I want to overwrite each disk with data from /dev/urandom before putting data on it.

The question is if I can create this 2 TB RAID-1 array, and securely overwrite each disk with random data while preserving the 1 TB of data I have on my current 1 TB RAID-1 array.

So to recap: I have 2 x 1 TB disks and 1 x 2 TB disk. I want to create a 2 TB RAID-1 array where one side of the mirror consists of two 1 TB disks pooled together, and the other side of the mirror is the whole 2 TB disk. Also, I want to preserve 1 TB of data using only these disks, while still being able to overwrite each disk with random data.

6
  • My hunch is, this can be accomplished with LVM. Maybe someone more experienced can say for sure one way or the other. Setting up RAID over LVM sounds brittle to me, though... – Joseph R. Aug 19 '13 at 19:01
  • It looks like it would be possible to remove the 2 TB disk from the existing RAID-1 array and setup the other 2 1TB disks. Then add the 2 TB disk, sync and resize the RAID array and the filesystems. However I would rather make a back-up of the data. – gypaetus Aug 19 '13 at 19:23
  • Sorry, it's unclear to me. Is the data already encrypted, or is this something new you want to add now? – frostschutz Aug 19 '13 at 19:38
  • @frostschutz The data is not encrypted currently. The thing with a backup though is that it makes the whole operation redundant. If I had a drive that I could backup the 1 TB of data to, I wouldn't need to go through all the hoops to preserve the data using only these three disks. – runeks Aug 21 '13 at 11:37
  • 1
    If you don't have a backup, your data wasn't important. – frostschutz Aug 21 '13 at 11:42
5

As for overwriting each disk with random data, it's redundant. Since you're going to build a new encrypted RAID, the resync will overwrite everything anyway.

As for the method of overwriting, /dev/urandom is horribly slow, people who try to use it for wiping terabytes usually cancel halfway through because it just takes way too long. Encrypting the device with a random key and then wiping that with /dev/zero is faster, and shred -n 1 is much faster still. So if you must have random data on your disk, I recommend you use those methods instead.

Now to your RAID, I'd do the following: (in this example loop2 is the 2TB disk)

  • Add the new 1TB disk partition to the RAID-1. Wait until the sync is finished. This way, your 1TB of data spans three disks.

    $ mdadm /dev/md99 --grow --raid-devices=3 --add /dev/loop1p1
    mdadm: added /dev/loop1p1
    raid_disks for /dev/md99 set to 3
    $ cat /proc/mdstat
    md99 : active raid1 loop1p1[2] loop2p1[1] loop0p1[0]
          100224 blocks super 1.2 [3/3] [UUU]
    
  • remove the 2TB disk from the RAID-1 array. Your 1TB is still preserved redundantly on the two 1TB disks.

    $ mdadm /dev/md99 --fail /dev/loop2p1
    mdadm: set /dev/loop2p1 faulty in /dev/md99
    $ mdadm /dev/md99 --remove /dev/loop2p1
    mdadm: hot removed /dev/loop2p1 from /dev/md99
    $ mdadm /dev/md99 --grow --raid-devices=2
    raid_disks for /dev/md99 set to 2
    $ cat /proc/mdstat
    md99 : active raid1 loop1p1[2] loop0p1[0]
          100224 blocks super 1.2 [2/2] [UU]
    
  • wipe the 2TB disk

    $ shred -n 1 /dev/loop2
    
  • repartition the disk to 2TB. Note that you will need a boot partition if you have no other boot device, as you can not boot from encrypted devices.

    $ parted /dev/loop2
    
  • create a new RAID-1 array using that partition, and missing for the 2nd device. The 2nd device will be added later on. Also, the size will be limited for now, to be grown later, as we're not sure of the size of the 2x1TB disks at this point. (If you are sure, feel free to use a different size here, but you won't be able to add the 2x1TB later if you make it too large).

    $ mdadm /dev/md42 --create --level=1 --raid-devices=2 --size=1000G /dev/loop2p1 missing
    mdadm: largest drive (/dev/loop2p1) exceeds size (102400K) by more than 1%
    Continue creating array? yes
    mdadm: Defaulting to version 1.2 metadata
    mdadm: array /dev/md42 started.
    $ cat /proc/mdstat
    md42 : active raid1 loop2p1[0]
          102400 blocks super 1.2 [2/1] [U_]
    
    md99 : active raid1 loop1p1[2] loop0p1[0]
          100224 blocks super 1.2 [2/2] [UU]
    
  • encrypt it, mkfs, copy data (use your preferred cipher and settings, LVM, filesystem, copy methods, ...)

    $ cryptsetup luksFormat /dev/md42
    $ cryptsetup luksOpen /dev/md42 luksmd42
    $ mkfs.ext4 /dev/mapper/luksmd42
    $ mount /dev/md99 /mnt/old
    $ mount /dev/md42 /mnt/new
    $ rsync -aAHSX /mnt/old/. /mnt/neW/.
    $ umount /mnt/old /mnt/new
    
  • Now your data is redundant on the old RAID 1 (unencrypted), and the new non-redundant RAID 1 (encrypted). But at this point you have to break redundancy in order to add the 2x1TB disks to the new RAID 1.

    $ mdadm --stop /dev/md99
    mdadm: stopped /dev/md99
    
  • At this point you could also wipe the 2x1TB disks if you like to waste your time. It's not necessary as the RAID-1 will sync the random data from the 2TB disk over to the 1TB disks.

    $ shred -n 1 /dev/loop0 &
    $ shred -n 1 /dev/loop1 &
    $ wait # for shred
    $ parted /dev/loop0
    $ parted /dev/loop1
    
  • Combine 2x1TB using mdadm, either linear or 0, depending on your preference.

    $ mdadm /dev/md43 --create --level=0 --raid-devices=2 /dev/loop0p1 /dev/loop1p1
    $ cat /proc/mdstat
    md43 : active raid0 loop1p1[1] loop0p1[0]
          199680 blocks super 1.2 512k chunks
    
    md42 : active raid1 loop2p1[0]
          102400 blocks super 1.2 [2/1] [U_]
    
  • Add it to the RAID 1 and grow the RAID 1 at the same time.

     $ mdadm /dev/md42 --add /dev/md43
     mdadm: added /dev/md43
     $ mdadm /dev/md42 --grow --size=max
     mdadm: component size of /dev/md42 has been set to 199616K
     $ cat /proc/mdstat
     md43 : active raid0 loop1p1[1] loop0p1[0]
           199680 blocks super 1.2 512k chunks
    
     md42 : active raid1 md43[2] loop2p1[0]
           199616 blocks super 1.2 [2/2] [UU]
    
  • Grow the cryptsetup and filesystem also.

     $ cryptsetup resize luksmd42
     $ resize2fs /dev/mapper/luksmd42
     resize2fs 1.42.7 (21-Jan-2013)
     Resizing the filesystem on /dev/mapper/luksmd42 to 197568 (1k) blocks.
     The filesystem on /dev/mapper/luksmd42 is now 197568 blocks long.
    

Aaand you're done.

Edit:

Here's an example /etc/mdadm.conf to go with this setup (use mdadm --detail --scan to get a starting point):

ARRAY /dev/md43 metadata=1.2 UUID=b9f590d7:9984dad4:cb75131b:63bca165
ARRAY /dev/md42 metadata=1.2 UUID=3a70188d:9ecacda7:ac715e16:9402fc55

In particular:

  • no DEVICE lines (if you must have them, make sure it includes md* devices)
  • order of ARRAY lines is important - the RAID0 array must be built first, which happens if it's listed first in the file.
6
  • Just finished overwriting the 2 TB drive with data from /dev/urandom (yeah I went that route) 2000398934016 bytes (2.0 TB) copied, 208934 s, 9.6 MB/s. I figure I only need to do it once, and I might as well do it properly. Copying files now. – runeks Aug 25 '13 at 13:15
  • After adding the RAID0 array to the RAID1 array and growing the array, I shut down my computer. Now when I boot up I get a message on boot saying I have a faulty array, and /proc/mdstat looks like this for the RAID1 array: md127 : active (auto-read-only) raid1 sdb1[0] 1953382488 blocks super 1.2 [2/1] [U_]. I can't figure out what's wrong here. – runeks Aug 26 '13 at 10:56
  • @runeks: It's probably your mdadm.conf. I'll append it to the answer. – frostschutz Aug 26 '13 at 12:54
  • I still get the "WARNING: degraded RAID devices detected" message when booting. I've tried doing what's described in this answer: askubuntu.com/a/209750/9567 but it doesn't make a difference. Here's how everything looks when it's running fine: pastebin.com/VS1AHbd0 and here's how it looks after a reboot: pastebin.com/VxyK8kD1 Here's the relevant part of dmesg: pastebin.com/5t6AbR2N. So it looks like md0 is assembled first, as it should be, but md2 is assembled with "active with 1 out of 2 mirrors". – runeks Aug 27 '13 at 12:09
  • I'm a bit stumped, particularly about the DeviceDisappeared message. Did you try with plain md0/md2 instead of naming them stripe/mirror? There is a bug report on Ubuntu, maybe something going wrong there? bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1171945 – frostschutz Aug 27 '13 at 15:16
0

Take a backup first.

I'll pretend for my answer that sda is the 2TB disk, and sdb and sdc are 1TB disks. sdc is the new, blank, 1TB disk. Currently, you have a mirror of sda1 and sdb1. sda2 is unused. (Change number as needed).

I think you can get where you want by:

  1. Do a full array consistency check/repair by echo repair > /sys/block/md0/md/sync_action (or whichever md device your mirror is). Check cat /proc/mdstat; wait a while (many hours) for this to complete. This is to ensure you don't have, e.g., bad blocks hiding.
  2. Wipe the brand new, empty disk (let this run at the same time as the array check).
  3. Drop sda1 out of the mirror. Confirm that you dropped the right disk. Confirm that you actually took a backup before even reading this post. Go ahead and wipe /dev/sda (fill with random). Re-partition /dev/sda to only have one partition (well, maybe two, if you have /boot on there). Also, when you re-partition it, you may want to leave a little space free, you need to make sure this is no bigger than a RAID10 of sdb and sdc will be.
  4. Create a new mirror out of sda1 and missing. I'll assume this is md1. Go ahead and set up LUKS (encryption) on it, and make a filesystem on md1_crypt.
  5. Copy the files from the old mirror to your new mirror.
  6. Unmount the old filesystem, and stop md0 (the old array).
  7. Wipe /dev/sdb.
  8. Make sure /dev/sdc is done wiping.
  9. Create a RAID0 array on /dev/sdb1 and /dev/sdb2. I'll call this md2
  10. Now add md2 (the RAID0) to md1 (the new mirror).

The basic idea is to transition to having:

                     sda
sdb                   |
    \                \|/
     ---> RAID10 -> RAID1 ---> LUKS/dmcrypt ---> filesystem
    /
sdc

You'll probably need an additional array (3-way mirror) for /boot. You will probably want to leave room for swap, either on its own array(s) by putting LVM between dmcrypt and the filesystem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.