7

I have about 2 million (60GiB) of gzipped small files and I would like to create a compressed archive containing all of them in an uncompressed version. Unfortunately, I cannot just uncompress them all and then create the compressed archive as I only have about 70GiB of free disk space. In other words, how can I do an equivalent of tar --file-filter="zcat" zcf file.tar.gz directory if the command-line switch like --file-filter doesn't exist in GNU tar?

  • Do you have a multiprocessor machine? – Anthon Nov 6 '14 at 14:23
  • 1
    @Anthon: not on this machine, but for the future readers we might assume that yes. – d33tah Nov 6 '14 at 14:32
  • As you have to recompress, there is something gain there. Any particular reason why to use gzip? Combining and compressing saves space, but you would gain a lot more if you would compress to an xz-ed tar file. Is that an option? – Anthon Nov 6 '14 at 14:34
  • Any compression program would do. If I can create a tar file of the decompressed-but-not-stored files, I can pipe it to any other program. – d33tah Nov 6 '14 at 14:44
6

An option could be to use avfs (here assuming a GNU system):

mkdir ~/AVFS &&
avfsd ~/AVFS &&
cd ~/AVFS/where/your/gz/files/are/ &&
find . -name '*.gz' -type f -printf '%p#\0' |
  tar --null -T - --transform='s/.gz#$//' -cf - | pigz > /dest/file.tar.gz
3

Take note, that this is fragile when it comes to nasty file names.

dir_with_small_files=/home/john/files
tmpdir=/tmp/ul/dst
tarfile=/tmp/ul.tar
mkfifo "${tarfile}"

gzip <"${tarfile}" >"${tarfile}.gz" &

find "$dir_with_small_files" -type f | \
while read src; do
    dstdir="${tmpdir}/$(dirname $src)"
    dst="$(basename $src .gz)"
    mkdir -p "$dstdir"
    gunzip <"$src" >"${dstdir}/${dst}"
    # rm "$src" # uncomment to remove the original files
    echo "${dstdir}/${dst}"
done | \
cpio --create --format=ustar -v --quiet 2>&1 >"${tarfile}" | \
while read x; do
    rm "$x"
done

# clean-up
rm "$tarfile"
rm -r "$tmpdir"

The files are uncompressed temporarily under $tmpdir, passed to cpio then as soon as they're added to the archive, removed.

  • 1
    Also, if you have multi-threads, I would suggest using pigz as an alternative to gzip :) – Christopher Stanley Nov 6 '14 at 19:04
2

Here's what I tried so far - it seems to work, but is terribly slow, even with PyPy:

#!/usr/bin/python

import tarfile
import os
import gzip
import sys
import cStringIO

tar = tarfile.open("/dev/stdout", "w|")
for name in sys.stdin:
    name = name[:-1]  # remove the trailing newline
    try:
        f = gzip.open(name)
        b = f.read()
        f.close()
    except IOError:
        f = open(name)
        b = f.read()
        f.close()
    # the [2:] there is to remove ./ from "find" output
    ti = tarfile.TarInfo(name[2:])
    ti.size = len(b)
    io = cStringIO.StringIO(b)
    tar.addfile(ti, io)
tar.close()

Usage: find . | script.py | gzip > file.tar.gz

  • Uncompressing and especially recompressing on a disk which is almost full, is going to be slow, no matter what. – Cristian Ciupitu Nov 6 '14 at 14:08
  • @CristianCiupitu: I measured without |gzip and the uncompressed file didn't basically touch the HDD, so IMHO it shouldn't be THAT slow. – d33tah Nov 6 '14 at 14:21
  • 1
    De and Recompressing is done in optimised C code in CPython. There might be buffering involved that cause the disc not to be touched. – Anthon Nov 6 '14 at 14:24
  • 1
    find . -exec cat \{\} \; > /dev/null should provide a lower bound on the amount of time this operation could possibly take. I would imagine that part of your problem is the creation of a bunch of large python objects containing your files in both compressed and uncompressed forms and then letting the garbage collector clean up after yourself. see here: stackoverflow.com/questions/6115066/… – BitShifter Nov 6 '14 at 17:30
  • You could probably save some memory by figuring out the uncompressed size and passing to tar the gzip file like object. – Cristian Ciupitu Nov 6 '14 at 17:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.