I have one zfs pool containing several zvols and datasets of which some are also nested. All datasets and zvols are periodically snapshotted by zfs-auto-snapshot. All datasets and zvols also have some manually created snapshots.

I have setup a remote pool on which due to lack of time, initial copying over local high speed network via zfs send -R did not complete (some datasets are missing, some datasets have outdated or missing snapshots).

Now the pool is physically remote over a slow speed connection and I need to periodically sync the remote pool with local pool, meaning data present in local pool must be copied to remote pool, data gone from local pool must be deleted from remote pool, and data present in remote pool but not in local pool must be deleted from remote pool, by data meaning 'zvols', 'datasets' or 'snapshots'.

If I was doing this between two regular filesystems using rsync, it would be "-axPHAX --delete" (that's what I actually do to backup some systems).

How do I setup a synchronizing task so the remote pool zvols & datasets (including their snapshots) can be in sync with local zvols,datasets&snapshots?

I would like to avoid transferring over ssh, because of low throughput performance of ssh; I'd prefer mbuffer or iscsi instead.

  • 3
    How did you do your initial zfs send -R ...? If you piped the output via ssh, did you disable escape characters with zfs send -R ... | ssh -e none ...? Commented Feb 16, 2016 at 20:32
  • Also - you need to make sure your slow connection has enough bandwidth to keep the remote copy up-to-date. If you're getting more changes to the local system than you can send to the remote system, you will never be able to keep the remote copy up-to-date. Take an incremental zfs replication stream and save it to a file. If the the file is larger than the amount of data you can send to the remote site in the amount of time between the snapshots, you'll never keep up. zfs send -R -i pool@snap1 pool@snap2 | gzip --fast > /output/file.gz Commented Feb 16, 2016 at 20:46
  • You also could try to use this script to do it automaticly: github.com/psy0rz/zfs_autobackup/blob/master/README.md Commented Jan 10, 2017 at 21:16
  • @AndrewHenle disabling the escape character is required only for interactive sessions, or non-interactive sessions where a pty was explicitly requested (-t).
    – rtx13
    Commented Feb 1, 2021 at 15:45
  • @rtx13 And that feature was added to OpenSSH when? Do all versions that are currently being used support that feature? Commented Feb 1, 2021 at 16:43

6 Answers 6


Disclaimer: As I've never used zvols, I cannot say if they are any different in replication than normal filesystems or snapshots. I assume they are, but do not take my word for it.

Your question is actually multiple questions, I try to answer them separately:

How to replicate/mirror complete pool to remote location

You need to split the task into two parts: first, the initial replication has to be complete, afterwards incremental replication is possible, as long as you do not mess with your replication snapshots. To enable incremental replication, you need to preserve the last replication snapshots, everything before that can be deleted. If you delete the previous snapshot, zfs recv will complain and abort the replication. In this case you have to start all over again, so try not to do this.

If you just need the correct options, they are:

  • zfs send:
    • -R: send everything under the given pool or dataset (recursive replication, needed all the time, includes -p). Also, when receiving, all deleted source snapshots are deleted on the destination.
    • -I: include all intermediate snapshots between the last replication snapshot and the current replication snapshot (needed only with incremental sends)
  • zfs recv:
    • -F: expand target pool, including deletion of existing datasets that are deleted on the source
    • -d: discard the name of the source pool and replace it with the destination pool name (the rest of the filesystem paths will be preserved, and if needed also created)
    • -u: do not mount filesystem on destination

If you prefer a complete example, here is a small script:


# Setup/variables:

# Each snapshot name must be unique, timestamp is a good choice.
# You can also use Solaris date, but I don't know the correct syntax.
timestamp=$(/usr/gnu/bin/date '+%Y%m%d%H%M%S')

# Initial send:

# Create first recursive snapshot of the whole pool.
zfs snapshot -r "$new_snap"
# Initial replication via SSH.
zfs send -R "$new_snap" | ssh "$destination_host" zfs recv -Fdu "$destination_pool"

# Incremental sends:

# Get old snapshot name.
old_snap=$(zfs list -H -o name -t snapshot -r "$source_pool" | grep "$source_pool"@"$snapshot_string" | tail --lines=1)
# Create new recursive snapshot of the whole pool.
zfs snapshot -r "$new_snap"
# Incremental replication via SSH.
zfs send -R -I "$old_snap" "$new_snap" | ssh "$destination_host" zfs recv -Fdu "$destination_pool"
# Delete older snaps on the local source (grep -v inverts the selection)
delete_from=$(zfs list -H -o name -t snapshot -r "$source_pool" | grep "$snapshot_string" | grep -v "$timestamp")
for snap in $delete_from; do
    zfs destroy "$snap"

Use something faster than SSH

If you have a sufficiently secured connection, for example IPSec or OpenVPN tunnel and a separate VLAN that only exists between sender and receiver, you may switch from SSH to unencrypted alternatives like mbuffer as described here, or you could use SSH with weak/no encryption and disabled compression, which is detailed here. There also was a website about recomiling SSH to be much faster, but unfortunately I don't remember the URL - I'll edit it later if I find it.

For very large datasets and slow connections, it may also be useful to to the first transmission via hard disk (use encrypted disk to store zpool and transmit it in sealed package via courier, mail or in person). As the method of transmission does not matter for send/recv, you can pipe everything to the disk, export the pool, send the disk to its destination, import the pool and then transmit all incremental sends via SSH.

The problem with messed up snapshots

As stated earlier, if you delete/modify your replication snapshots, you will receive the error message

cannot send 'pool/fs@name': not an earlier snapshot from the same fs

which means either your command was wrong or you are in an inconsistent state where you must remove the snapshots and start all over.

This has several negative implications:

  1. You cannot delete a replication snapshot until the new replication snapshot was successfully transferred. As these replication snapshots include the state of all other (older) snapshots, empty space of deleted files and snapshots will only be reclaimed if the replication finishes. This may lead to temporary or permanent space problems on your pool which you can only fix by restarting or finishing the complete replication procedure.
  2. You will have many additional snapshots, which slows down the list command (except on Oracle Solaris 11, where this was fixed).
  3. You may need to protect the snapshots against (accidental) removal, except by the script itself.

There exists a possible solution to those problems, but I have not tried it myself. You could use zfs bookmark, a new feature in OpenSolaris/illumos created specifically for this task. This would free you of snapshot management. The only downside is that at present, it only works for single datasets, not recursively. You would have to save a list of all your old and new datasets and then loop over them, bookmarking, sending and receiving them, and then updating the list (or small database, if you prefer).

If you try the bookmark route, I would be interested to hear how it worked out for you!

  • thank you very much for this detailed answer. i am just send..receive-ing a zpool.
    – jitter
    Commented Nov 29, 2018 at 21:52
  • 2
    nice script. I'd add -d 1 to both of the zfs list commands to limit the search depth (there's no need to search below the pool name). This avoids long delays on pools with lots of snapshots (e.g. my "backup" pool has 320000 snapshots, and zfs list -r -t snapshot backup takes 13 minutes to run. It only takes 0.06 seconds with -d 1). The zfs destroy command in the for loop then needs the -r option to recursively delete all snapshots with the same snapname.
    – cas
    Commented May 1, 2019 at 3:22
  • > “There also was a website about recomiling SSH to be much faster, but unfortunately I don't remember the URL - I'll edit it later if I find it.” This one, maybe: psc.edu/hpn-ssh Commented Aug 14, 2020 at 16:29

Personally, I would make myself a list of zvols, datasets etc on the remote server that don't have up-to-date snapshots, and then bring those snapshots up to date with zfs send, even if this is time-consuming and uses a lot of bandwidth.

Then I could just continue using zfs send from then on and not have to reinvent the wheel by writing my own synchronisation code. rsync is nice for older filesystems but zfs send is much better for zfs - it knows exactly which blocks have changed in the snapshot and sends only them, while rsync has to compare individual files and/or timestamps between local and remote servers. same applies to btrfs send for btrfs pools.

If you have only a small number of snapshots that need to be brought up-to-date, this could be done manually. Otherwise to do it automatically, you need a list of the latest local snapshots vs remote snapshots, and a script to compare versions and then zfs send local snapshots that are out of date on the rmeote server.

That will be enough if you only care about the latest snapshot for each dataset. If you care about all previous snapshots, obviously your script is going to have to handle them too....and that becomes a LOT more complicated. In some cases, you may have to rollback on the remote server so you can re-send the intermediate/missing snapshots.

If you want a secure connection to the remote server, you've really got little choice but to use ssh - or perhaps set up a tunnel with openvpn or something and use netcat.

  • What about to use Zrep? bolthole.com/solaris/zrep
    – Xdg
    Commented Jan 11, 2017 at 21:13
  • dunno, never used it. looks like it would make a good answer, though if someone were to do a little research and testing and write it up (that's a hint).
    – cas
    Commented Jan 14, 2017 at 7:44
  • I've tested it on Ubuntu (ZFS on linux) and it wasn't working on deeper datasets (tank/something/someother). I was using this port to shell - link . Recursive flag export ZREP_R=-R wasn't working at all. :(
    – Xdg
    Commented Jan 14, 2017 at 9:05

Have a look at `zrepl', on FreeBSD, which could make your life, and anyone's for that matter, a lot easier. It was presented a few days ago during BSDCan2018 in Ottawa. It looks promising and may be a solution to your problems


The easiest way for me is to use this great utility:

syncoid -r pool1 pool2

-r will transfer also beneath datasets of the pool.

Note that it will transfer the pool and datasets contents ONLY if the pool, datasets does not exist yet on destination. After initial sync, you can run the command again to keep the pool2 in sync.

  • 1
    yes, that's what I eventually ended up doing as well. not perfect, but acceptable Commented Mar 10, 2021 at 16:56
  • 1
    per the github.com/jimsalterjrs/sanoid/issues/… the -r switch is needed to sync also the datasets of the pool and the command will only work/transfer the data if the destination pool, dataset doe snot yet exist on destination. Same command can be used repeatedly and it should keep the pool2 datasets in sync with pool1. Also it supported resume in my case. So i could do Ctrl+C and then run it again to resume sync.
    – 16851556
    Commented Apr 24, 2021 at 12:16
  • 2
    Another switched worth considering: --skip-parent --no-stream It may not sync parent(pool, but only datasets, which may get rid of a syncoid warning). --no-stream is important for keeping only most recent syncoid snapshot, saving the disk space.
    – 16851556
    Commented Nov 3, 2022 at 17:50

zrep is a nice all-in-one solution, AND has documentation+hooks on how to get faster transfers than just plain SSH transfers


it's also crossplatform: supported on linux, freebsd, and solaris/illumos

  • 1
  • 1
    The question in the Question is: "How do I setup a synchronizing task so the remote pool zvols & datasets (including their snapshots) can be in sync with local zvols,datasets&snapshots?"
    – Jeff Schaller
    Commented Sep 7, 2018 at 17:34
  • Jeff, are you suggesting that the best "answer", would be to cut-n-paste bits from the zrep documentation, rather than just giving a reference to zrep? Commented Sep 8, 2018 at 20:19
  • 1
    I don’t know what the best answer would be, but a link to software is not a solution. It’s already been mentioned, in fact. The question asks: “How do I setup a synchronizing task so the remote pool zvols & datasets (including their snapshots) can be in sync with local zvols,datasets&snapshots?”
    – Jeff Schaller
    Commented Sep 8, 2018 at 20:42
  • yes that is the question. However, to accomplish the task WELL, requires a whole lot more than a little writeup on a webpage here. That is why zrep is a 2000 line shellscript. Even if one were to remove all the parts that the original problem never needed, there would still be a couple hundred lines of script required to do it WELL. Commented Sep 9, 2018 at 21:24

A speed-up note ref. answer by user 121391 in "Initial replication" line as speed was a consideration. I am new to zfs.

Note if you have compression on the source and target volumes enabled, you should use the -c switch on the send command as this will negate the need to decompress the data in the stream and recompressing on the target.

Secondly, I have tested that when I run a command like yours, I was getting 2MB/s transfer rate on a local machine for my test with a USB hard drive target. I got a 10x increase by piping in an mbuffer command. zpool iostat -v showed 21MB/s read from source pool and 20 MB/s write to target pool. This was marginally faster than a dd test writing from /dev/null so I was able to saturate the write speed of the target. Using the default of mbuffer at 128k threw up a warning about too many blocks and asked for a higher block size. My command is below. -P gives a parsable verbose output and estimate of total size and reports transfer rate and progress. The transfer appears fast until the buffer fills. 2G is probably more than needed.

zfs send -c -P -R SourceZpool/SourceDataset@SnapshotName | mbuffer -s 1M -m 2G | zfs recv -F TargetZpool/TargetDataset

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .