@pdo's answer is good, but one can increase the speed with a buffer and good compression and add a progress bar.
Often the network is the bottleneck and the speed varies over time. Therefore, it helps to buffer the data before sending them over the network. This can be done with with pv
.
Additionally, one can usually increase the speed with a proper compression algorithm. Gzip (like used above) is a fast compression algorithm, but in general zstandard (zstd
) (and for high compression ratios LZMA/LZMA2 (xz
) will compress better and be faster the same time. New xz and zstd have multi core support already built in. To use gzip with multiple cores pigz can be used.
Here is an example to send data with a progress bar, buffering and zstandard compression over a network:
tar cf - . | pv -perabs $(du -sk . | cut -f 1)K | zstd -14 --long=31 -T0 | pv -qCB 512M | ssh user@host "cd /wherever && pv -qCB 512M | zstd -cd -T0 --long=31 | tar xf -"
The first pv
is to show the progress (p), estimated time (e), transfer rate (r), average rate (a), total transferred bytes (b). The total size is estimated with du
and added to the size option (s). The progress is measured before compression and buffering, therefore it's not very accurate, but still helpful.
zstd
is used with the compression setting 14. This number can be reduced or increased depending on the network and CPU speed so zstd is a little bit faster than the the network speed. With four cores on a Haswell 3.2 GHz CPU 14 gives a speed of around 120 MB/s.
In the example, the long mode 31 (uses a 2 GB window, needs a lot of RAM, but very good e.g. to compress database dumps) is used. The T0 options sets the amount of threads to the number of cores. One should be aware that together with the long mode these settings use a lot of memory.
A problem with zstd is that most operating systems don't ship with version >= 1.3.4. This version is necessary for proper multi core and long support. If not available, it can be compiled and installed from https://github.com/facebook/zstd with just make -j4 && sudo make install
. Instead of zstd, one can also use xz or pigz. xz is slow but compresses very well (good over slow connections), pigz/gzip is fast but compresses not so well.
pv
is then used again, but for buffering (q
for quiet, C
for the no splice mode [always needed for buffering] and B
to set the buffer size).
In the example a buffer is also used on the receiver side. This is often unnecessary (because decompression and hard disk writing speed is most time higher than the network speed), but does usually not harm, either.