How to turn gzipped file into one tar.gz archive in-place?

Question

I have a lot (~5 millions) of small files, each one compressed individually with gzip. I would like to turn them into one big tar.gz archive to save some space (metadata), but there are limitations:

• I don't have enough space to decompress files and compress again into tar.gz, so I have to do it in place. Reduction ratio is quite nice, about 15:1.
• I can't use much CPU time, so recompression may take a very long time.
• Files can't be moved to somewhere else or removed, because important security reasons

So. What can I do?

Answer 1

You don't have to compress the archive, just add the already compressed files to it.

tar -cf archive.tar directory_with_compressed_files

Answer 2

If only have enough space to do a small number of files, you can do the archiving in chunks, and it can even be automated if you want. Assuming your files are stored something like this:

.
+-dir0000
| +-file0000.gz
| +-file0001.gz
| [...]
+-dir0001
| +-file1000.gz
| +-file1001.gz
[...]

For each directory, run (from the shared root of the files):

tar rf /path/to/archive_name.tar dir_name
rm -r dir_name

If you're OK with automating this (I tried it and it worked, but you get error messages from find which can be ignored) try the following:

find * -maxdepth 1 -type d -exec sh -c "tar rf /path/to/archive_name '{}' && rm -r '{}'" \;

(As usual, try this on a test directory structure first so you can verify it's doing what you want!)

I quoted the {} in case your directories have spaces. If you have a flat file structure, then you can do something similar, only without -type d. Make sure not to use + at the end of -exec, or else find will try to do a whole bunch of files/directories at a time, which defeats the point of breaking the archiving into pieces in the first place!

Eventually all the files will be moved into the archive. It'll be slow, but it can be done.