I'm asking about a scenario of copying a big file to a remote server.

A simplest case is:

tar c myfile | ssh myserver tar x

If network connectivity is fast then all is fine.

On a slower network I do

tar c myfile | bzip2 -1 | ssh myserver tar xj

-- making my transfer faster at the cost of CPU time.

Of course I can play with compression ratio, typically trying to guess the right one so my CPU is not too busy and the network is saturated.

Is there a compression utility or a compression flag that would tell bzip2/xz/... to compress as much as possible while the output buffer is busy?

zstd --adapt

The zstd compression utility has an option that turns on adaptive compression (the option was added in zstd v1.3.6). This would adjust the compression to "the current perceived I/O conditions".

See the zstd manual for more information.

A complete pipeline may look something like this:

tar -c -f - source_directory |
zstd --adapt |
ssh user@server 'cd /someplace && { zstd -d | tar -x -f -;}'


tar -c -f - source_directory |
zstd --adapt |
ssh user@server 'zstd -d | tar -x -C /someplace -f -'

If you add -v to the first zstd in the pipeline, you will get a progress indicator line saying something like

(L7) Buffered :  32 MB - Consumed : 192 MB - Compressed :  72 MB => 37.50%

where the (L7) indicates the compression level. For any moderately large amount of data, you would expect it to fluctuate over time, showing that zstd is indeed adapting to the I/O conditions (and presumably also to the data itself).

  • 1
    Sadly, in ubuntu bionic repos there's currently zstd 1.3.3, which doesn't support the flag, but 1.4.0 I've built from sources does. Thank you. – bohdan_trotsenko Jun 7 at 10:18
  • @bohdan_trotsenko Thanks for the heads-up about that. I'll figure out when that option was added to the utility, and then I'll mention that in the answer too. – Kusalananda Jun 7 at 10:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.