I am looking for a compression tool with an arbitrarily large dictionary (and "block size"). Let me explain by way of examples.

First let us create 32MB random data and then concatenate it to itself to make a file of twice the length of length 64MB.

head -c32M /dev/urandom > test32.bin
cat test32.bin test32.bin > test64.bin

Of course test32.bin is not compressible because it is random but the first half of test64.bin is the same as the second half, so it should be compressible by roughly 50%.

First let's try some standard tools. test64.bin is of size exactly 67108864.

  • gzip -9. Compressed size 67119133.
  • bzip2 -9. Compressed size 67409123. (A really big overhead!)
  • xz -7. Compressed size 67112252.
  • xz -8. Compressed size 33561724.
  • zstd --ultra -22. Compressed size 33558039.

We learn from this that gzip and bzip2 can never compress this file. However with a big enough dictionary xz and zstd can compress the file and in that case zstd does the best job.

However, now try:

head -c150M /dev/urandom > test150.bin
cat test150.bin test150.bin > test300.bin

test300.bin is of size exactly 314572800. Let's try the best compression algorithms again at their highest settings.

  • xz -9. Compressed size 314588440
  • zstd --ultra -22. Compressed size 314580017

In this case neither tool can compress the file.

Is there a tool that has an arbitrarily large dictionary size so it can compress a file such as test300.bin?

Thanks to the comment and answer it turns out both zstd and xz can do it. You need zstd version 1.4.x however.

  • zstd --long=28. Compressed size 157306814
  • xz -9 --lzma2=dict=150MiB. Compressed size 157317764.
  • Maybe not unlimited, but apparently 7-zip allows dictionary size of up to 1536MB for LZMA compression. See sevenzip.osdn.jp/chm/cmdline/switches/method.htm#LZMA. – kyrill Jan 18 '20 at 22:17
  • 2
    zstd --long=28 compresses your larger file. But see the manual about decompressing files with --long=N where N is greater than 27. – Kusalananda Jan 18 '20 at 23:02
  • 1
    @Kusalananda Could you give the full command line please? I can't get zstd to work with --long=28. It just reports "Incorrect parameters". – Anush Jan 19 '20 at 17:54
  • 1
    @Anush That's more or less the full command, except for the filename at the end. I did not use any other option. I'm using zstd v1.4.4. – Kusalananda Jan 19 '20 at 17:56
  • 2
    The --long mode of zstd was introduced in v1.3.2 : github.com/facebook/zstd/releases/tag/v1.3.2 , in 2017. – Cyan Jan 20 '20 at 22:37

It's at least available with the xz command. The xz manpage has:

The following table summarises the features of the presets:

Preset    DictSize    CompCPU     CompMem     DecMem
    -0    256 KiB        0          3 MiB      1 MiB


    -9     64 MiB        6        674 MiB     65 MiB

Column descriptions:

DictSize is the LZMA2 dictionary size. It is waste of memory to use a dictionary bigger than the size of the uncompressed file. This is why it is good to avoid using the presets -7 ... -9 when there's no real need for them. [...]

As documented in the Custom compressor filter chains section, you can simply supply manually the dictionary size to xz with for example --lzma2=dict=150MiB (we have insight information telling 150MiB is enough, else in doubt the file size would have to be used).

xz -9 --lzma2=dict=150MiB test300.bin

While doing this the xz process on amd64 stayed most of the time at about 1.6g usage of resident memory.

$ ls -l test*
-rw-r--r--. 1 user user 157286400 Jan 19 16:03 test150.bin
-rw-r--r--. 1 user user 157317764 Jan 19 16:03 test300.bin.xz
  • Thanks very much. On my PC the RAM use goes up to 1.634 gigabytes according to top. But it does work. – Anush Jan 19 '20 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.