After having read the section about zpool export in man zpool, I am worried somehow:

Exports the given pools from the system. All devices are marked as exported, but are still considered in use by other subsystems. The devices can be moved between systems (even those of different endianness) and imported as long as a sufficient number of devices are present.

What does "but are still considered in use by other subsystems" mean? And what does "as long as a sufficient number of devices are present" mean?


I have a fairly complex backup script which replicates VM storage on a production server to a standby server, based on ZFS snapshots (to be precise, it is a system of various scripts on the host and in the VMs which work together to freeze the file systems in the VMs, take the ZFS snapshots, thaw the filesystems in the VMs, and replicate the snapshots to the standby server). This part is working like a charm.

But actually, I need a sort of backup in addition to the replication. That is, I would like to backup the VM storage onto a disk which I can afterwards remove from the box where it is attached to, and store it safely at another location.

I have thought thoroughly about the best method and have come to the following idea:

  • In the standby server, mount a SATA / SAS HDD tray so that I can plug and unplug HDDs there (or use one of its existing trays which are connected to its backplane).
  • At the standby server, plugin a new HDD and make a new ZPOOL, consisting of only one VDEV which consists only of that HDD.
  • Let the production server replicate its VM storage to that new ZPOOL in addition to the ZPOOL to which it replicates already.
  • After the replication, export the new ZPOOL, unplug the new HDD, and store it at a safe place. **
  • At the standby server, plug in a second new HDD which also represents a ZPOOL, and repeat the previous two steps. ****
  • And so on ... (for example, daily replace the HDD currently attached to the standby server by the other one, or use 7 HDDs and rotate them on a weekly basis, etc.).

** This is the step which I am not comfortable with (after having read the section cited above). On one hand, it should be no problem to remove the HDD after the ZPOOL (whose only device it is) has been exported, because file systems get dismounted (and hence flushed) before the export. On the other hand, the manual says that this HDD will still be "considered in use by other subsystems" even after exporting the pool, which makes me believe that it is a bad idea to simply remove it in this situation.

Therefore, I would like to know what this statement exactly means, and how I can make the HDD being considered not in use any more by other subsystems.

**** I am aware that I'll have to put some effort into this. The plan is to create a script along with an appropriate udev rule to have the standby server recognize when the HDD is plugged in and to import the existing ZPOOL (the HDD will be connected directly via SATA-3 or SAS 12G, not via USB). But this is not part of this question.

To summarize:

What exactly does zpool export do, or, in other words, what steps do I need to perform before I can safely physically remove a HDD which is the only device in a ZPOOL after that ZPOOL has been exported?


I do something similar to what you are describing and I think you may be a little mixed up in the use of zpool export.

zpool export removes the entire pool from the system... snapshots and all. This isn't what you want.

To do what you are asking, the first step is to understand what kind of zpool we're dealing with. In particular, you probably want a simple mirrored pool. At least 3 drives.

In my case, I have a zpool with 3 mirrors. When I want to take the data offsite for safe keeping, I "offline" the drive I want to take offsite and then remove it from the array. This looks like:

      pool: pool_09
 state: ONLINE
  scan: resilvered 1000G in 2h15m with 0 errors on Sun Nov 22 19:49:53 2020


        NAME                         STATE     READ WRITE CKSUM
        pool_09                      ONLINE       0     0     0
          mirror-0                   ONLINE       0     0     0
            c0t5000C500B0BCFB13d0    ONLINE       0     0     0
            c0t5000C500B0889E5Bd0    ONLINE       0     0     0
            c0t5000C500B09F0C54d0    ONLINE       0     0     0
          mirror-1                   ONLINE       0     0     0
            c0t5F8DB4C095690612d0s3  ONLINE       0     0     0
            c0t5F8DB4C095691282d0s3  ONLINE       0     0     0

      # zpool offline  pool_09 c0t5000C500B0BCFB13d0

This will leave the pool with two mirrored drives so I can still take a local failure. For me, I don't import the drive into another system, but I don't see why you couldn't. I just take mine to another building.

When I'm ready to re-sync the mirrors, I put the removed drive back into the array and "online" it:

zpool online pool_09 c0t5000C500B0BCFB13d0

ZFS will automatically resilver the drives once you online it and then you can repeat the process.

I don't think you'd want to do this with a bunch of rotated drives. I'm not even sure that would work. I guess it would if you had 10x mirrored set or something. Just keep in mind, the longer the removed drive(s) are out of the array, the longer it will take to resilver.

Another option would be to create another pool and zfs send all the data to it and then export that pool and carry it offsite. I've not done that, but it should work... probably a lot slower though.


  • Thank you very much for the answer - much appreciated, and +1. However, you eventually have misread my question: I was talking about about creating a NEW zpool for the backup, as you describe in the last paragraph of your answer. My question actually is whether I really can remove the HDD from the system after having exported that "backup" ZPOOL, or whether I have to perform additional steps before actually pulling that HDD. – Binarus Dec 16 '20 at 8:25
  • 1
    Oh, sorry. Yes, once you have exported the pool, you can remove the drives. "In use by other subsystems" means that other low-level processes may still be "using" the drives. For example, even though you have exported the pool, when you remove the drive, the OS may warn/complain that a device has been removed. That hardware monitoring process is a subsystem that might still be using the device. Once the pool is exported, from a zfs perspecive, the data is safe. – mikem Dec 16 '20 at 8:37
  • 1
    On a resilver, the metadata gets read to determine what needs updating, but only changed/missing data gets written. To "prove" this, remove a drive from a mirror and create a small file on the pool... ie 50MB. Put the removed drive back in and it will probably resilver before you can even run a zpool status on it! – mikem Dec 16 '20 at 8:46
  • 1
    Missed your other question -- there are no other steps necessary before removing the drive from the array. And that goes for exported pools or offline'd drives. – mikem Dec 16 '20 at 8:48
  • 1
    Actually not all metadata has to be read either. Using the "prove" test I gave above, I just dropped a 6TB drive, created a 200MB tarball, and then added the drive back into the pool. The result was "scan: resilvered 219M in 31s with 0 errors " and all drives reported ONLINE. ZFS handles this kind of data delta very, very well. One caveat... I am using Solaris 11.4 vs. Linux, but surely there isn't that drastic of a performance gap between the two. – mikem Dec 17 '20 at 16:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.