Estimate compressibility of file

Question

Is there a quick and dirty way of estimating gzip-compressibility of a file without having to fully compress it with gzip?

I could, in bash, do

bc <<<"scale=2;$(gzip -c file | wc -c)/$(wc -c <file)"

This gives me the compression factor without having to write the gz file to disk; this way I can avoid replacing a file on disk with its gz version if the resultant disk space savings do not justify the trouble. But with this approach the file is indeed fully put through gzip; it's just that the output is piped to wc rather than written to disk.

Is there a way to get a rough compressibility estimate for a file without having gzip work on all its contents?

Answer 1

Here's a (hopefully equivalent) Python version of Stephane Chazelas's solution

python -c "
import zlib
from itertools import islice
from functools import partial
import sys
with open(sys.argv[1], "rb") as f:
  compressor = zlib.compressobj()
  t, z = 0, 0.0
  for chunk in islice(iter(partial(f.read, 4096), b''), 0, None, 10):
    t += len(chunk)
    z += len(compressor.compress(chunk))
  z += len(compressor.flush())
  print(z/t)
" file

Answer 2

You could try compressing one every 10 blocks for instance to get an idea:

perl -MIPC::Open2 -nE 'BEGIN{$/=\4096;open2(\*I,\*O,"gzip|wc -c")}
                       if ($. % 10 == 1) {print O $_; $l+=length}
                       END{close O; $c = <I>; say $c/$l}'

(here with 4K blocks).

Answer 3

I had a multi-gigabyte file and I wasn't sure if it was compressed, so I test-compressed the first 10M bytes:

head -c 10000000 large_file.bin | gzip | wc -c

It's not perfect but it worked well for me.

Answer 4

I wrote a quick script to recursively check each file in a directory and then only compress the most compressible files. It grabs a couple of megabytes in the interior of the file and tests them with gzip --fast, then it uses xz to compress the file if needed.

You can run it with: ./compress.if.compressible dir_name

cores=$(grep '^core id' /proc/cpuinfo | sort -u | wc -l)

IFS=$'\n'
for file in $(find "$1" -type f); do

#Skip small files. The savings is minimal and sometimes xz just makes these larger.
size_b=`du --apparent-size -b "$file" | sed 's/\t.*//'`
if [[ $size_b -le 1024 ]]; then
    echo -e "\nSkipping small file $file"
    continue
fi

size=`du --apparent-size --block-size=1M "$file" | sed 's/\t.*//'`
reduction=$(dd if="$file" bs=1M count=2 skip=$(expr $size / 3) 2>/dev/null | gzip --fast -v 2>&1 > /dev/null)

echo -e "\n$reduction $file"

if [[ $(echo $reduction | sed 's/\..*//') -ge 20 ]]; then
    echo "$size_b * .9 > 90" | bc
    xz -v -T $cores "$file"
    new_size=`du --apparent-size -b "$file.xz" | sed 's/\t.*//'`
    if [[ `echo "$new_size * 100 / $size_b" | bc` -ge 90 ]]; then
        echo "Insufficent Compression. Reverting..."
        unxz "$file.xz"
    fi
fi

done

Answer 5

Here is an improved Python version based on iruvar's awesome solution. The main improvement is that the script reads only the data blocks from disk that it actually compresses:

import zlib
def Predict_file_compression_ratio(MyFilePath):
 blocksize = (4096 * 1) # Increase if you want to read more bytes per block at once.
 blocksize_seek = 0

 # r = read, b = binary
 with open(MyFilePath, "rb") as f:
  # Make a zlib compressor object, and set compression level.
  # 1 is fastest, 9 is slowest
  compressor = zlib.compressobj(1)
  t, z, counter = 0, 0, 0

  while True:
    # Use this modulo calculation to check every "number" of blocks.
    if counter % 10 == 0:
      # Seek to the correct byte position of the file.
      f.seek(blocksize_seek)
      # The block above will be read, increase the seek distance by one block for the next iteration.
      blocksize_seek += blocksize
      # Read data chunk of file into this variable.
      data = f.read(blocksize)
      
      # Stop if there are no more data.
      if not data:
        # For zlib: Flush any remaining compressed data. Not doing this can lead to a tiny inaccuracy.
        z += len(compressor.flush())
        break

      # Uncompressed data size, add size to variable to get a total value.
      t += len(data)
      # Compressed data size
      z += len(compressor.compress(data))

    # When we skip, we want to increase the seek distance. This is vital for correct skipping.
    else:
      blocksize_seek += blocksize
    # Increase the block / iteration counter.
    counter += 1

 # Print the results. But avoid division by 0 >_>
 if not t == 0:
  print('Compression ratio: ' + str(z/t))
 else:
  print('Compression ratio: none, file has no content.')
 print('Compressed: ' + str(z))
 print('Uncompressed: ' + str(t))

If high data rates are crucial and accurate compression ratios not so much, you can use lz4 instead. It's great if you only want to find out what files can be compressed the most, with low CPU usage. This module needs to be installed with pip from here. In the Python code itself, you pretty much only need this:

import lz4.block
z += len(lz4.block.compress(data))

Please note, I observed that using this script does thrash the standby memory (does for sure on Windows), which degrades file performance - especially on computers with classic hard drives, and if you use this function for lots of files at once. This memory trashing can be avoided by setting a low memory page priority on the script's Python process. I opted to do this with AutoHotkey on Windows. Helpful source here.

Answer 6

For binary files, or a single large file that doesn't vary a lot in the type of content (inside the file) I use this. This takes the first 1MB of data, runs gzip verbosely, and reports it's compression expectation:

cat content.bson | ( dd bs=1024 count=1024 | gzip -v - > /dev/null ) 2>&1 | tail -1
 85.8%
# or 
( dd if=content.bson bs=1024 count=1024 | gzip -v - > /dev/null ) 2>&1 | tail -1
 85.8%

In this case, I'm compressing a single file Mongo DB data collection which holds many string fields.

If you're compressing a whole directory tree, with many types of files it's hard to know that the different files will have consistent compression rates. You can use the above technique but you may want to test with more of the data (meaning try count=10240 for 10MB or higher):

$ time ( tar cf - . | dd bs=1024 count=1024 | gzip -v - > /dev/null ) 2>&1 | tail -1
 85.8%

real    0m0.028s
user    0m0.027s
sys     0m0.008s

$ time ( tar cf - . | dd bs=1024 count=102400 | gzip -v - > /dev/null ) 2>&1 | tail -1
 80.4%

real    0m3.599s
user    0m3.643s
sys     0m0.263s

Above I tar the current directory (.) for 1MB which takes 0.028 secs for %85.8 compression, then I do a more accurate 100MB (102400) which takes 3.599 secs and report approx 80.4% compression.