I have an application that has many thousands of files totaling over 10TB.

I'd need to backup this data somewhere (probably to AWS S3).

I'd like to:

  1. compress data being backed up
  2. save the backup as a single file

For example as a gzipped tarfile.

Because of the size, I cannot create the gzipped tarfile locally because it'd be too large.

How can I:

  1. Stream all of these folders and files onto AWS S3 as a single compressed file?
  2. Stream the compressed file from S3 back onto my disk to the original filesystem layout?

1 Answer 1


This is a basic piping and ssh use case.

$ tar zcf - -C /path/to/your/files . | ssh S3_hostname 'cat > yourfile.tar.gz'

To decompress:

$ ssh S3_hostname 'cat yourfile.tar.gz' | tar zxf - -C /path/to/extract/to

The key here is telling tar that it should write to or read from stdout/stdin instead of a file on the local filesystem. In the case of tar creating the archive, the stdout from tar is fielded by ssh which pipes it to the remote invocation of cat running on the S3 host, where the output gets written to file yourfile.tar.gz. In the decompression scenario, ssh is again used to invoke cat on the remote host to read the file, and that stream becomes the stdin stream for a local invocation of tar which extracts the archive to the path specified in the -C argument.

  • This is great, thank you! However how do you open an SSH connection to your S3 bucket? I wasn't aware you could do this and can't seem to find documentation.
    – nick314
    Dec 1, 2022 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.