0

High Level Description

So I currently have a convoluted ZFS setup and want to restructure it, reusing some of the existing hardware.

I know that the recommended way of doing something like this is to backup all data, destroy the old pools, create the new ones and restore the data, the question is on how to best do this

The Setup and Details

My current setup consists of 3×1TB and 3×4TB drives set up in the following way Two of the 4TB drives are each formatted in one 1TB and one 3TB Partition.

  • media_pool is a raidz1 pool consisting of 5×1TB disks/ partitions (-> 4TB Avaiable, 3.26TB used)
  • three_t_pool is a mirror consisting of the 2 3TB partitions (-> 3TB Avaiable, 1.93TB used)
  • non_redundant is a pool just consisting of one 4TB drive (1.07TB used)

Each of those pools has exactly one encrypted dataset spanning the entire pool. media_pool/media_encrypted, three_t_pool/backups, and non_redundant/nr_encrypted.

My future setup will retire the 1TB drives and add a 12TB drive as follows:

  • media_and_backups a raidz1 pool consisting of the 3 4TB drives and a 4TB partition of the 12TB drive ((4-1)×4TB = 12TB avaiable)
  • non_redundant_foo the remaining 8TB of the big drive.

And encrypted datasets media_and_backups/media_encrypted, media_and_backups/backups, and non_redundant_foo/nr_encrypted.

(Although keeping the media and backup datasets separate is not a must if this would complicate things)

Now my migration process would probably look like this (With all 7 drives connected to the same machine):

  • Format the 12TB drive in the desired 4 and 8TB partitions
  • backup the data of the exiting pools/datasets to the 8TB partition
  • destroy the old three pools, create the media_and_backups pool and corresponding datasets (media_and_backups/media_encrypted, media_and_backups/backups), restore them from backup
  • [Optionally even move the nr_encrypted data to media_and_backups, destroy the filesystem on the 8TB partition create a new pool there and restore the nr_encrypted dataset to there]

But like I mentioned above the part where I'm unsure about is the backup and restore progress

  • One naïve method I can think of is to just use rsync, create a folder for each dataset, sync-in all the data, create the pools and datasets I desire and rsync back to those.
    • One obvious disadvantage would be that the data would be decrypted on backup and reencrypted on restore
    • It just feels wrong how would I for example recognize in-transfer data corruption
  • zfs send and zfs receive seems to be the way to go. But... how would I actually do that?
    • I read the answer to ZFS send/recv full snapshot but there are some key differences between my setup and theirs:
      • They directly create the new pool on the new server I want to first store the backup and then later restore it on the same server
      • I have three datasets from three different pools i want to store on the same pool
    • Also there's the thing about the available storage: the 8TB (= 7.28TiB) should be enough to hold the 3.26+1.93+1.07=6.26TiB of used data, but of course a 8TB pool would not be able to host 4+3+4TB worth of datasets
  • So I think I should zfs send the current datasets into one big file each (or somehow split those into blocks) on the 8TB pool, then when that's done destroy the old pools and create the new ones and zfs recv from the files to the pools

Is this the best way of doing this, or is there a better/ recommened way? Or any best practicies for using zfs send/ zfs receive. If I do it this way are there any caveats to watch out for?

Improve this question

1 Answer 1

Reset to default
0

So I found out that error correction happens on receive, so creating an intermediate file isn't a good idea.

I also found out that my datasets don't have a set volsize (checked with zfs get volsize) so I only have to watch out for the total used size, as opposed to the pool size.

I also cam across how to one-way mirror an entire zfs pool to another zfs pool from which I gathered that I can use the -d (discard) option on zfs receive so that the filesystem path matches the new pool.

I also needed the --raw flag on zfs send because since my datasets are encrypted and I wanted to avoid decrypting and reencrypting them.

Finally I since I was sending multiple datasets originating from different pools all to the same destination pool, I had to watch out to only send the dataset snapshot, not the pool snapshot, otherwise I would get errors about conflicting snapshots on the pool.

In total, my transfer command looked like this:

zfs send -R --raw origin_pool/dataset@transfer_some_timestamp | pv -Wbraft | zfs receive -Fduv destination_pool

  • zfs send
    • -R Generate replication stream package to preserve all properties, snapshots, ...
    • --raw For encrypted datasets, send data exactly as it exists on disk. Avoid reencryption
  • pv -Wbraft monitor status
  • zfs receive
    • -F` expand target pool
    • -d discard origin_pool part of filesystem name and use destination_pool instead
    • -u Don't mount
    • -v Print verbose information about the stream and the time required to perform the receive operation. (With hindsight I would have proably not needed both pv and the -v option but oh well)

So the above was the general takeaway but for completeness below the steps I took

  • Prerequisite: Set open up a tmux session so is could easily disconnect without stopping running processes
  • Format the 12T drive in 4T and 8T, taking care that the 4T partition is exactly the size the other 4T drives will also have (By looking at one of the existing partiotion tables using fdisk -l /dev/sdX)
  • Create the pool non_redundant_two on the 8T partition
  • Creating the recursive snapshot on the first source pool zfs snap -r drei_t_pool@transfer_$(date '+%Y%m%d%H%M%S')
  • Transferring to the non_redundant pool: zfs send -R --raw drei_t_pool@transfer_20231210194637| pv -Wbraft | zfs receive -Fduv non_redundant_two (note that this was not ideal, see below)
  • Create a snapshot of the next source pool zfs snap -r media_pool@transfer_$(date '+%Y%m%d%H%M%S')
  • Try to send it and get error, inspect snapshots (zfs list -t snapshot) see that there is a very small snapshot for the pool and a huge snapshot for the dataset. Delete the pool snapshot: zfs destroy non_redundant_two@transfer_20231210194637
  • I also considered and tried to create a "holder" dataset and sending the entire pool-snapshot to it
    • zfs create non_redundant_two/media_holder
    • zfs send -R --raw media_pool@transfer_20231210232518 | pv -Wbraft | zfs receive -Fduv non_redundant_two/media _holder
    • This also seemed to work but I canceled+destroyed the attempt it because I had a better Idea:
  • Send just the dataset: zfs send -R --raw media_pool/media_encrypted@transfer_20231210232518 | pv -Wbraft | zfs receive -Fduv non_redundant_two
  • Do the same with the last pool/dataset zfs snap -r non_redundant@transfer_$(date '+%Y%m%d%H%M%S') and zfs send -R --raw non_redundant/nr_encrypted@transfer_20231211121954 | pv -Wbraft | zfs receive -Fduv non_redundant_two
  • Test if I can open the backed up datasets and read files (did this for all pools):
    • zfs load-key non_redundant_two/backups
    • Inspect...
    • Unmount and unload key zfs unmount -a, zfs unload-key non_redundant/nr_encrypted
  • Backup properties of pools before I destroy them, e.g. pool get all drei_t_pool > zfs.get.drei_t_pool
  • Destroy origin snapshot, then dataset, then pool (I know this could be done with a recursive flag, but I wanted to be super sure of what I was doing) (For all source pools)
    • zfs destroy drei_t_pool@transfer_20231210194637, zfs destroy drei_t_pool/backups@transfer_20231210194637
    • zfs destroy drei_t_pool/backups
    • zpool destroy drei_t_pool
  • Create new pool I wanted to migrate to media_and_backups
  • Transfer datasets to there (no need for creating a snapshot again, since the data didn't change)
    • zfs send -R --raw non_redundant_two/backups@transfer_20231210194637 | pv -Wbraft | zfs receive -Fduv media_and_backups
    • zfs send -R --raw non_redundant_two/media_encrypted@transfer_20231210232518 | pv -Wbraft | zfs receive -Fduv media_and_backups
  • Check decrypting and mounting works as expected and destroy datasets from non_redundant_two
  • Destroy transfer snapshots

One additional curiosity: NFS wasn't working so I checked zfs get sharenfs pool_name/dataset_name where it showed my the preferences I had set, but the SOURCE tab showed remote. So I re-set that property, after which the SOURCE tab showed local and NFS was working again

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .