Archive folder onto server, compress on the host

Question

I try to do the following:

1. Upload a folder (say Pictures) onto a server
2. Let the server compress the input stream and save it to a file

Importantly, step 1) should be uncompressed as the connection is very fast and I assume compression at the client-side would be slower. Further, the server does not feature enough space to store the directory without compression. Ideally, the compression on the server happens concurrently.

I don't care too much about the actual compression algorithm as long as it is multithreaded.

Answer 1

Preliminary note: Assuming your pictures are already compressed (JPEG, PNG, WEBP,…), no matter what compression you use, there's not much that can be done, space-wise. There's simply no entropy to be reduced anymore. So, your whole scheme might fall flat here.

A usual method for this is using the Tape ARchive format (.tar). You can serialize a directory into a single TAR stream, pipe that through the internet (e.g. using SSH) and compress it on the target.

You also want to switch off SSH's built-in transport compression, otherwise it makes no sense at all to not compress at the sending host to begin with.

Something like

tar c Pictures/ | ssh -o "Compression=no" user@server "zstd -15 -T0 > Pictures.tar.zstd"

replace zstd -T0 (use as many CPU cores as you can get, for zstandard compression; the -15 is the compression level; 15 is pretty brutal, -3 is better than gzip but usually faster) with xz -T0 (for parallel XZip compression) or pigz (for parallel gzip compression) as you see fit. I do recommend zstd, it has a very good compression ratio / speed tradeoff; xz is pretty slow, and gzip yields only very limited compression and isn't faster than zstd in comparably low compression settings.

Note, however, that encryption of your data probably needs more CPU than compression, so the question really is why you don't just

zstd -r --stdout Pictures/ | ssh -o "Compression=no" user@server "cat > Pictures.zst"

Answer 2

You can do it with a compressed file system.

Look for example at Btrfs or ZFS - just define a volume as compressed on the host and it satisfy your requirements.