1

I have a lot (~5 millions) of small files, each one compressed individually with gzip. I would like to turn them into one big tar.gz archive to save some space (metadata), but there are limitations:

  • I don't have enough space to decompress files and compress again into tar.gz, so I have to do it in place. Reduction ratio is quite nice, about 15:1.
  • I can't use much CPU time, so recompression may take a very long time.
  • Files can't be moved to somewhere else or removed, because important security reasons

So. What can I do?

  • Buy an additional hard drive. – Cyrus Mar 24 '18 at 11:32
  • So just to be clear: You don't have space for uncompressing the files and you also don't have space for an archive of the compressed files? – Kusalananda Mar 24 '18 at 12:33
0

You don't have to compress the archive, just add the already compressed files to it.

tar -cf archive.tar directory_with_compressed_files
  • I can't do that, because I have no space for new archive :) And my tar version don't have --remove-files option. – maln0ir Mar 24 '18 at 11:26
  • @maln0ir Ah, I thought you said you did not have space for the decompressed files. – Kusalananda Mar 24 '18 at 11:27
0

If only have enough space to do a small number of files, you can do the archiving in chunks, and it can even be automated if you want. Assuming your files are stored something like this:

.
+-dir0000
| +-file0000.gz
| +-file0001.gz
| [...]
+-dir0001
| +-file1000.gz
| +-file1001.gz
[...]

For each directory, run (from the shared root of the files):

tar rf /path/to/archive_name.tar dir_name
rm -r dir_name

If you're OK with automating this (I tried it and it worked, but you get error messages from find which can be ignored) try the following:

find * -maxdepth 1 -type d -exec sh -c "tar rf /path/to/archive_name '{}' && rm -r '{}'" \;

(As usual, try this on a test directory structure first so you can verify it's doing what you want!)

I quoted the {} in case your directories have spaces. If you have a flat file structure, then you can do something similar, only without -type d. Make sure not to use + at the end of -exec, or else find will try to do a whole bunch of files/directories at a time, which defeats the point of breaking the archiving into pieces in the first place!

Eventually all the files will be moved into the archive. It'll be slow, but it can be done.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.