What exactly does zpool export do?

Question

After having read the section about zpool export in man zpool, I am worried somehow:

Exports the given pools from the system. All devices are marked as exported, but are still considered in use by other subsystems. The devices can be moved between systems (even those of different endianness) and imported as long as a sufficient number of devices are present.

What does "but are still considered in use by other subsystems" mean? And what does "as long as a sufficient number of devices are present" mean?

Background:

I have a fairly complex backup script which replicates VM storage on a production server to a standby server, based on ZFS snapshots (to be precise, it is a system of various scripts on the host and in the VMs which work together to freeze the file systems in the VMs, take the ZFS snapshots, thaw the filesystems in the VMs, and replicate the snapshots to the standby server). This part is working like a charm.

But actually, I need a sort of backup in addition to the replication. That is, I would like to backup the VM storage onto a disk which I can afterwards remove from the box where it is attached to, and store it safely at another location.

I have thought thoroughly about the best method and have come to the following idea:

• In the standby server, mount a SATA / SAS HDD tray so that I can plug and unplug HDDs there (or use one of its existing trays which are connected to its backplane).
• At the standby server, plugin a new HDD and make a new ZPOOL, consisting of only one VDEV which consists only of that HDD.
• Let the production server replicate its VM storage to that new ZPOOL in addition to the ZPOOL to which it replicates already.
• After the replication, export the new ZPOOL, unplug the new HDD, and store it at a safe place. **
• At the standby server, plug in a second new HDD which also represents a ZPOOL, and repeat the previous two steps. ****
• And so on ... (for example, daily replace the HDD currently attached to the standby server by the other one, or use 7 HDDs and rotate them on a weekly basis, etc.).

** This is the step which I am not comfortable with (after having read the section cited above). On one hand, it should be no problem to remove the HDD after the ZPOOL (whose only device it is) has been exported, because file systems get dismounted (and hence flushed) before the export. On the other hand, the manual says that this HDD will still be "considered in use by other subsystems" even after exporting the pool, which makes me believe that it is a bad idea to simply remove it in this situation.

Therefore, I would like to know what this statement exactly means, and how I can make the HDD being considered not in use any more by other subsystems.

**** I am aware that I'll have to put some effort into this. The plan is to create a script along with an appropriate udev rule to have the standby server recognize when the HDD is plugged in and to import the existing ZPOOL (the HDD will be connected directly via SATA-3 or SAS 12G, not via USB). But this is not part of this question.

To summarize:

What exactly does zpool export do, or, in other words, what steps do I need to perform before I can safely physically remove a HDD which is the only device in a ZPOOL after that ZPOOL has been exported?

Answer 1

I do something similar to what you are describing and I think you may be a little mixed up in the use of zpool export.

zpool export removes the entire pool from the system... snapshots and all. This isn't what you want.

To do what you are asking, the first step is to understand what kind of zpool we're dealing with. In particular, you probably want a simple mirrored pool. At least 3 drives.

In my case, I have a zpool with 3 mirrors. When I want to take the data offsite for safe keeping, I "offline" the drive I want to take offsite and then remove it from the array. This looks like:

      pool: pool_09
 state: ONLINE
  scan: resilvered 1000G in 2h15m with 0 errors on Sun Nov 22 19:49:53 2020

config:

        NAME                         STATE     READ WRITE CKSUM
        pool_09                      ONLINE       0     0     0
          mirror-0                   ONLINE       0     0     0
            c0t5000C500B0BCFB13d0    ONLINE       0     0     0
            c0t5000C500B0889E5Bd0    ONLINE       0     0     0
            c0t5000C500B09F0C54d0    ONLINE       0     0     0
        logs
          mirror-1                   ONLINE       0     0     0
            c0t5F8DB4C095690612d0s3  ONLINE       0     0     0
            c0t5F8DB4C095691282d0s3  ONLINE       0     0     0


      # zpool offline  pool_09 c0t5000C500B0BCFB13d0

This will leave the pool with two mirrored drives so I can still take a local failure. For me, I don't import the drive into another system, but I don't see why you couldn't. I just take mine to another building.

When I'm ready to re-sync the mirrors, I put the removed drive back into the array and "online" it:

zpool online pool_09 c0t5000C500B0BCFB13d0

ZFS will automatically resilver the drives once you online it and then you can repeat the process.

I don't think you'd want to do this with a bunch of rotated drives. I'm not even sure that would work. I guess it would if you had 10x mirrored set or something. Just keep in mind, the longer the removed drive(s) are out of the array, the longer it will take to resilver.

Another option would be to create another pool and zfs send all the data to it and then export that pool and carry it offsite. I've not done that, but it should work... probably a lot slower though.

.