How to backup many large files to single compressed file on S3

Question

I have an application that has many thousands of files totaling over 10TB.

I'd need to backup this data somewhere (probably to AWS S3).

I'd like to:

1. compress data being backed up
2. save the backup as a single file

For example as a gzipped tarfile.

Because of the size, I cannot create the gzipped tarfile locally because it'd be too large.

How can I:

1. Stream all of these folders and files onto AWS S3 as a single compressed file?
2. Stream the compressed file from S3 back onto my disk to the original filesystem layout?

Answer 1

This is a basic piping and ssh use case.

$ tar zcf - -C /path/to/your/files . | ssh S3_hostname 'cat > yourfile.tar.gz'

To decompress:

$ ssh S3_hostname 'cat yourfile.tar.gz' | tar zxf - -C /path/to/extract/to

The key here is telling tar that it should write to or read from stdout/stdin instead of a file on the local filesystem. In the case of tar creating the archive, the stdout from tar is fielded by ssh which pipes it to the remote invocation of cat running on the S3 host, where the output gets written to file yourfile.tar.gz. In the decompression scenario, ssh is again used to invoke cat on the remote host to read the file, and that stream becomes the stdin stream for a local invocation of tar which extracts the archive to the path specified in the -C argument.