I need to write data (multiple files and directories) from a Google cloud storage bucket onto a Tar archive on my local disk. The directories are multiple TB, so I do not want to create the Tar archive on disk and then copy it.
My current strategy is to use gsutil (Google's python utility for copying from cloud storage) and pipe that to tar. I'm testing it on a single text file using the following command:
gsutil cp gs://data/myfile.txt - | tar -czvf myfile.tar.gz -T -
The issue I'm having is that the shell seems to be expanding the actual contents of the text file and trying to pass the text file as arguments to tar. I get the following error message:
'ar: unrecognized option '---- BASELINE
Since my text file contains a line that reads "---- BASELINE" I take it that the shell is trying to expand the input from the text file. I have tried putting double quotes in a couple places but haven't been able to figure it out.