I have a large tar.gz file with 122000 files that I'm downloading from the web to a compute resource that is limited to 14GB. The tar.gz is 3.3GB and the fully extracted archive is 29GB. I know I can extract each file by name, but this is painfully slow.

So we're all talking about the same thing...

# make a tar
for i in {0..9}
echo "file $i" > file$i.txt

tar -czvf files.tar.gz file*.txt 
rm *.txt

# extract each file one at a time
for files in `tar tf files.tar.gz`
tar Oxvzf files.tar.gz $files | gzip > $files.gz

Is there a way to extract each file sequentially without tar having to find the desired file each loop through the listing of files?

  • The tar Oxvzf provides the STDIN stream to gzip. Removing the O extracts all of the files before compressing them
    – PD Schloss
    Commented Sep 22, 2022 at 0:15

1 Answer 1


How about this - since you have > 100k files, can we therefore assume that 10k files use <10% of the unpacked space? A single pipe buffers a finite amount before blocking, so you can have the tar wait after extracting a few thousand files:

tar xvfz files.tar.gz |while read filepath; do 
    [ -f "$filepath" ] && gzip "$filepath"
  • It's not reasonable to assume anything. It could very well be that 100 of the files take up 30, 50, or even 75 percent of the space. Commented Sep 22, 2022 at 0:36
  • You've seen the data? Of course anything is possible, but with over 100k files it seems more likely that sizes would be averaged out to some degree - so it seems reasonable to ask if this is a valid assumption - and if not then sure, this solution will not be the answer. Commented Sep 22, 2022 at 0:45
  • No, I haven't, and neither have you. That's my point. No one knows unless they are Charles Xavier or Doctor Strange. It would have been reasonable to ask about this in a comment rather than posting it as an answer when nothing indicates one thing or another. Commented Sep 22, 2022 at 1:17
  • as a solution the OP can monitor the filesystem, and kill -STOP/kill -CONT the tarjob.. If one file can possible fill up the entire filesystem alone, then the OP can do all this based on streaming io inside python pretty handily. Other than that I would use xz for the compression. much better than gz.
    – toppk
    Commented Sep 22, 2022 at 2:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .