Sequentially extract files from a tar.gz file

Question

I have a large tar.gz file with 122000 files that I'm downloading from the web to a compute resource that is limited to 14GB. The tar.gz is 3.3GB and the fully extracted archive is 29GB. I know I can extract each file by name, but this is painfully slow.

So we're all talking about the same thing...

# make a tar
for i in {0..9}
do
echo "file $i" > file$i.txt
done

tar -czvf files.tar.gz file*.txt 
rm *.txt


# extract each file one at a time
for files in `tar tf files.tar.gz`
do
tar Oxvzf files.tar.gz $files | gzip > $files.gz
done

Is there a way to extract each file sequentially without tar having to find the desired file each loop through the listing of files?

Answer 1

How about this - since you have > 100k files, can we therefore assume that 10k files use <10% of the unpacked space? A single pipe buffers a finite amount before blocking, so you can have the tar wait after extracting a few thousand files:

tar xvfz files.tar.gz |while read filepath; do 
    [ -f "$filepath" ] && gzip "$filepath"
done