1

we're copying a large zfs snapshot (zfs send) from site-1 to site-2. site-1 and site-2 are remote, and are connected via IPsec (read: slow)

on site-2, we need the data on two different zfs servers. currently we're using two simultaneous ssh "zfs send" | zfs receive commands. this is of course a waste of bandwidth.

we've experimented with the following (1 remote ssh, tee, 1 local ssh):

[root@site-2-1 ~]# \
    time \
    ssh root@site-1-0 \
        "cat /tmp/100000000.rnd | pigz -1" \
    | pv -B 10m -s 100000000 \
    | tee \
        >(ssh root@site-2-2 "pigz -d > /tmp/test.rnd.1") \
    | pigz -d \
    > /tmp/test.rnd.1

95.4MiB 0:00:52 [1.83MiB/s] [===<snipped>===>] 100%

real    0m52.062s
user    0m0.979s
sys 0m1.100s

which turned out to be slower than a simulation of the original solution (2 remote ssh's simultaneously):

[root@site-2-1 ~]# \
    time \
    ssh root@site-1-0 \
        "cat /tmp/100000000.rnd | pigz -1" \
    | pv -B 10m -s 100000000 \
    | pigz -d \
    > /tmp/test.rnd.2
95.4MiB 0:00:30 [3.09MiB/s] [===<snipped>===>] 100%

real    0m30.837s
user    0m0.827s
sys 0m1.003s


[root@site-2-2 ~]# \
    time \
    ssh root@site-1-0 \
        "cat /tmp/100000000.rnd | pigz -1" \
    | pv -B 10m -s 100000000 \
    | pigz -d \
    > /tmp/test.rnd.2
95.4MiB 0:00:38 [ 2.5MiB/s] [===<snipped>===>] 100%

real    0m38.187s
user    0m0.840s
sys 0m0.846s

we've looked into tee's code, our prime suspect, and it seems it operates out of a single thread.

can anyone recomment a faster alternative to tee, or a totally different solution maybe?

thanks in advance, guven

1

suggestion:

zfs send > file
scp file server-at-site2:
ssh server-at-site2 zfs receive < file
ssh server-at-site2 cat file | ssh second-server-at-site2 zfs receive

That requires two transfers, but presumably the second will be faster (because local network, etc)

  • our file is huge as well (> 1TB) so we'd need another storage for that – guven Jun 22 '16 at 13:30
  • I was afraid you'd say that :-) How about receiving it on the first server, and then sending it again from that server to the second one? That might take a little longer depending on the speed of your ZFS, but at least you don't need the extra storage and waste less bandwidth. And since a ZFS receive requires that the start snapshot is present and creates (at least) the end snapshot, this doesn't require extra snapshots of something. – Wouter Verhelst Jun 22 '16 at 14:54
  • that is one of our last resort options, if all else fails :) – guven Jun 24 '16 at 9:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.