1

I need to write data (multiple files and directories) from a Google cloud storage bucket onto a Tar archive on my local disk. The directories are multiple TB, so I do not want to create the Tar archive on disk and then copy it.

My current strategy is to use gsutil (Google's python utility for copying from cloud storage) and pipe that to tar. I'm testing it on a single text file using the following command:

gsutil cp gs://data/myfile.txt - | tar -czvf myfile.tar.gz -T -

The issue I'm having is that the shell seems to be expanding the actual contents of the text file and trying to pass the text file as arguments to tar. I get the following error message:

'ar: unrecognized option '---- BASELINE

Since my text file contains a line that reads "---- BASELINE" I take it that the shell is trying to expand the input from the text file. I have tried putting double quotes in a couple places but haven't been able to figure it out.

Thanks!

  • The -T option for GNU tar takes a file from which filenames are read to add to the archive. You give it a text file so it naturally uses the contents of the file as the filenames to add, and fails. tar can't add files whose data comes over standard input to an archive. – Kusalananda Oct 23 '19 at 18:13
  • Ah, I see, so perhaps what I'm trying to do is not possible with Tar. @Kusalananda do you know a way around this? – Benjamin Antin Oct 23 '19 at 18:35
  • You must feed tar with path of files, not file content. You're here feeding the file's ontent. It's possible to feed content of file if the file has path of files. – Nizam Mohamed Oct 23 '19 at 19:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.