I have an application that has many thousands of files totaling over 10TB.
I'd need to backup this data somewhere (probably to AWS S3).
I'd like to:
- compress data being backed up
- save the backup as a single file
For example as a gzipped tarfile.
Because of the size, I cannot create the gzipped tarfile locally because it'd be too large.
How can I:
- Stream all of these folders and files onto AWS S3 as a single compressed file?
- Stream the compressed file from S3 back onto my disk to the original filesystem layout?