What's the best way to estimate the compression level that can be achieved for a directory without storing the compressed files?

I have a large amount of directories and sub-directories with files with different nature (binaries, text files, compressed files, and so on). I need to compress some of these directories, but I don't have enough space to compress all of them (or even some of them) and them sort them by compression rate.

For this reason I need to list their compression rate (how much they can be compressed) to only then decide the ones that will be compressed.

I have a list of directories size for uncompressed directories:

du -f --maxdepth=3 > /tmp/list_of_directories

Tar, bz2 or gzip could be used to compress each directory using the fast compression level to a null pipe, but no compression rate statics are generated if the file is not preserved.

Since the entire contents must be somehow compressed to a zip pipe, this solution would be very processing intensive, but very light in terms of storage since the compressed results would be discarded as they are generated.

in short the question can be resumed to "Is there any tool that generates compression rate stats for a given stream, file or directory without storing the final compressed file?"

  • not really. I'm basically asking how to do it without writing an script. Is there any way to figure out the compression level without actually generating the compressed files? (thus not using the space to store it) – Rafael Borja Jun 27 at 19:15
  • question rephrased for clarity – Rafael Borja Jun 27 at 19:25
  • 1
    Re: no compression rate statics are generated if the file is not preserved. gzip, for one, will generate compression rate statistics (-v) regardless of whether the file is preserved. – Mark Plotnick Jun 27 at 20:28
  • yes, I've actually considered gzip -v, but percentage stats per file are troublesome to summarize. I've found a better option using bzip --verbose -zc >> /dev/null, but stats are still troublesome to parse and summarize , but it solves the lack of space to store intermediate files. Basically the only missing point now, is how to bzip2 directories without creating the intermediate tar file (always due to lack of space) – Rafael Borja Jun 27 at 20:43

It can be done by piping the contents of a directory to a compressing tool and redirecting the compressed stream to a null pipe (to avoid using space). This compressing tool must: 1 - Compress TO piped stream 2 - Be able to compress to /dev/null 2 - Provide compression statistics over a discarded file

Among most popular tools, bzip2 is the one that best fits, specially due the output status with number bytes in and out.

The final solution looks like:

cat $(find DIRECTORY_PATH) | bzip2 -c --verbose > /dev/null

Where the first part will concatenate all the contents of any file within the chosen directory.

cat $(find DIRECTORY_PATH)

While second part will compresses the concatenated input from the first part generating a single report (--verbose) at the end while redirecting the output to a null pipe

bzip2 -c --verbose > /dev/null

Output should look like the following, where a "(\d+) in" regex should capture the original size for all files in the directories of bytes and "(\d+) in" the compressed size in bytes. The output should look like:

1.581:1, 5.059 bits/byte, 36.77% saved, 1039062 in, 657029 out.

This solution has a limitation on the number of files on the directories, but can be used as a good starting point to identify directories with good potential to be compressed to save space.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.