# Estimate compressibility of file

Is there a quick and dirty way of estimating `gzip`-compressibility of a file without having to fully compress it with `gzip`?

I could, in `bash`, do

``````bc <<<"scale=2;\$(gzip -c file | wc -c)/\$(wc -c <file)"
``````

This gives me the compression factor without having to write the `gz` file to disk; this way I can avoid replacing a file on disk with its `gz` version if the resultant disk space savings do not justify the trouble. But with this approach the file is indeed fully put through `gzip`; it's just that the output is piped to `wc` rather than written to disk.

Is there a way to get a rough compressibility estimate for a file without having `gzip` work on all its contents?

Here's a (hopefully equivalent) Python version of Stephane Chazelas's solution

``````python -c "
import zlib
from itertools import islice
from functools import partial
import sys
with open(sys.argv[1], "rb") as f:
compressor = zlib.compressobj()
t, z = 0, 0.0
for chunk in islice(iter(partial(f.read, 4096), b''), 0, None, 10):
t += len(chunk)
z += len(compressor.compress(chunk))
z += len(compressor.flush())
print(z/t)
" file
``````
• I don't know if that's equivalent (as python is not by cup of coffee ;-b)), but that gives slightly different results (even for very large files where the gzip header size overhead can be ignored), possibly because zlib.compressobj uses different settings than gzip (I find that it's closer to `gzip -3`) or maybe because zlib.compressobj compresses each chunk in isolation (as opposed to a stream as a whole). In any case both approaches should be good enough. I find that perl is slightly faster. Commented Sep 17, 2014 at 11:55
• @StéphaneChazelas, judging by the documentation `compressobj` should be acting on the stream as a whole. I find that both Python and Perl solutions produce approximately the same result on my data files (they differ in the second decimal place, that's good enough for me). Thanks for the great idea Commented Sep 17, 2014 at 12:48
• Thanks, this is super handy. Is there any way to use this in Python without thrashing the standby memory or to reduce the data this reads? Because it appears more data is being read into memory than compressed (since it does it every 10 blocks, but the whole file is read anyway). Commented Jul 14, 2020 at 18:48
• @Testerhood, a naive way might involve seeking forward 10 blocks, read 1 block, rinse and repeat. Having said that I am not sure if doing that would prevent file-caching effects (in other words prevent the i0 block ignore-worthy chunks from being read into memory). This is worthy of a new question Commented Jul 14, 2020 at 22:45
• @iruvar Alright, thanks for the suggestion :) I will share my results if I can manage to make improvements. Commented Jul 15, 2020 at 23:53

You could try compressing one every 10 blocks for instance to get an idea:

``````perl -MIPC::Open2 -nE 'BEGIN{\$/=\4096;open2(\*I,\*O,"gzip|wc -c")}
if (\$. % 10 == 1) {print O \$_; \$l+=length}
END{close O; \$c = <I>; say \$c/\$l}'
``````

(here with 4K blocks).

I had a multi-gigabyte file and I wasn't sure if it was compressed, so I test-compressed the first 10M bytes:

``````head -c 10000000 large_file.bin | gzip | wc -c
``````

It's not perfect but it worked well for me.

• aidan, a compressibility estimation based on a contiguous subset of the file could yield misleading results if the compressibility varies from one part of the file to the other - Stephane's sampling approach is superior in this regard Commented Jan 9, 2019 at 17:41
• @iruvar Agreed. As I say, it's not perfect, but it'll definitely tell you the difference between a gigabyte of ascii text and a gigabyte of H.264. It also has the highest chance of running in very stripped down environments (Perl may be missing). Commented Jan 9, 2019 at 22:48

I wrote a quick script to recursively check each file in a directory and then only compress the most compressible files. It grabs a couple of megabytes in the interior of the file and tests them with `gzip --fast`, then it uses `xz` to compress the file if needed.

You can run it with: ./compress.if.compressible dir_name

``````cores=\$(grep '^core id' /proc/cpuinfo | sort -u | wc -l)

IFS=\$'\n'
for file in \$(find "\$1" -type f); do

#Skip small files. The savings is minimal and sometimes xz just makes these larger.
size_b=`du --apparent-size -b "\$file" | sed 's/\t.*//'`
if [[ \$size_b -le 1024 ]]; then
echo -e "\nSkipping small file \$file"
continue
fi

size=`du --apparent-size --block-size=1M "\$file" | sed 's/\t.*//'`
reduction=\$(dd if="\$file" bs=1M count=2 skip=\$(expr \$size / 3) 2>/dev/null | gzip --fast -v 2>&1 > /dev/null)

echo -e "\n\$reduction \$file"

if [[ \$(echo \$reduction | sed 's/\..*//') -ge 20 ]]; then
echo "\$size_b * .9 > 90" | bc
xz -v -T \$cores "\$file"
new_size=`du --apparent-size -b "\$file.xz" | sed 's/\t.*//'`
if [[ `echo "\$new_size * 100 / \$size_b" | bc` -ge 90 ]]; then
echo "Insufficent Compression. Reverting..."
unxz "\$file.xz"
fi
fi

done
``````

Here is an improved Python version based on iruvar's awesome solution. The main improvement is that the script reads only the data blocks from disk that it actually compresses:

``````import zlib
def Predict_file_compression_ratio(MyFilePath):
blocksize = (4096 * 1) # Increase if you want to read more bytes per block at once.
blocksize_seek = 0

# r = read, b = binary
with open(MyFilePath, "rb") as f:
# Make a zlib compressor object, and set compression level.
# 1 is fastest, 9 is slowest
compressor = zlib.compressobj(1)
t, z, counter = 0, 0, 0

while True:
# Use this modulo calculation to check every "number" of blocks.
if counter % 10 == 0:
# Seek to the correct byte position of the file.
f.seek(blocksize_seek)
# The block above will be read, increase the seek distance by one block for the next iteration.
blocksize_seek += blocksize
# Read data chunk of file into this variable.

# Stop if there are no more data.
if not data:
# For zlib: Flush any remaining compressed data. Not doing this can lead to a tiny inaccuracy.
z += len(compressor.flush())
break

# Uncompressed data size, add size to variable to get a total value.
t += len(data)
# Compressed data size
z += len(compressor.compress(data))

# When we skip, we want to increase the seek distance. This is vital for correct skipping.
else:
blocksize_seek += blocksize
# Increase the block / iteration counter.
counter += 1

# Print the results. But avoid division by 0 >_>
if not t == 0:
print('Compression ratio: ' + str(z/t))
else:
print('Compression ratio: none, file has no content.')
print('Compressed: ' + str(z))
print('Uncompressed: ' + str(t))
``````

If high data rates are crucial and accurate compression ratios not so much, you can use lz4 instead. It's great if you only want to find out what files can be compressed the most, with low CPU usage. This module needs to be installed with pip from here. In the Python code itself, you pretty much only need this:

``````import lz4.block
z += len(lz4.block.compress(data))
``````

Please note, I observed that using this script does thrash the standby memory (does for sure on Windows), which degrades file performance - especially on computers with classic hard drives, and if you use this function for lots of files at once. This memory trashing can be avoided by setting a low memory page priority on the script's Python process. I opted to do this with AutoHotkey on Windows. Helpful source here.

• Is it also possible to predict compression time with this information? Commented Sep 28, 2020 at 11:55

For binary files, or a single large file that doesn't vary a lot in the type of content (inside the file) I use this. This takes the first 1MB of data, runs gzip verbosely, and reports it's compression expectation:

``````cat content.bson | ( dd bs=1024 count=1024 | gzip -v - > /dev/null ) 2>&1 | tail -1
85.8%
# or
( dd if=content.bson bs=1024 count=1024 | gzip -v - > /dev/null ) 2>&1 | tail -1
85.8%
``````

In this case, I'm compressing a single file Mongo DB data collection which holds many string fields.

If you're compressing a whole directory tree, with many types of files it's hard to know that the different files will have consistent compression rates. You can use the above technique but you may want to test with more of the data (meaning try count=10240 for 10MB or higher):

``````\$ time ( tar cf - . | dd bs=1024 count=1024 | gzip -v - > /dev/null ) 2>&1 | tail -1
85.8%

real    0m0.028s
user    0m0.027s
sys     0m0.008s

\$ time ( tar cf - . | dd bs=1024 count=102400 | gzip -v - > /dev/null ) 2>&1 | tail -1
80.4%

real    0m3.599s
user    0m3.643s
sys     0m0.263s
``````

Above I tar the current directory (`.`) for 1MB which takes 0.028 secs for %85.8 compression, then I do a more accurate 100MB (102400) which takes 3.599 secs and report approx 80.4% compression.