1

I'm trying to compress a large archive of similar JPG images. The best compression algorithm I've found so far is fp8. In this question I show my tests and ask for better options. But fp8 is the best I've found so far. It reduces my dataset to 80% of its size, way better than any other traditional compression utilities (zip, z7, rar, tar, bz2, etc.), even if it is still not ideal (please visit that question if you have suggestions of better options).

However, fp8 seem to be an abandoned utility I found only for windows (I've done the tests running it under wine).

Is there any utility with implementation of the fp8 algorithm for linux?

I've found zpaq in Ubuntu repositories, that also offer one implementation of PAQ compression. However, it perform much worse on similar JPG images. That's why I'm looking specifically for fp8 or one that have a similar o better performance with similar JPG images.

2 Answers 2

3

The FP8 format appears to have been retired by its author in favour of the ZPAQ format, which is why you can't find newer versions:

ZPAQ is intended to replace PAQ and its variants (PAQ8, PAQ9A, LPAQ, LPQ1, etc) with similar or better compression in a portable, standard format. Current versions of PAQ break archive compatibility with each compression improvement. ZPAQ is intended to fix that. I no longer maintain the older PAQ code.

The Ubuntu distribution of ZPAQ appears to be very out of date, so you probably want to upgrade using the version on the ZPAQ website instead: https://pkgs.org/download/zpaq. The newer versions support many more options regarding compression behaviour (from the ZPAQ documentation):

-mtype[Blocksize[.pre[.arg][comp[.arg]]…]]

-method type[Blocksize[.pre[.arg][comp[.arg]]…]]

With add, select a compression method. type may be 0, 1, 2, 3, 4, 5, x, or s. The optional Blocksize may be 0..11, written with no space after the type, like -m10 or -method 511. The remaining arguments, separated by periods or commas without spaces, are only allowed for types x or s, for example -mx4.3ci1.

If type is numeric, then higher numbers compress better but are slower. The default is -m1. It is recommended for backups. -m2 compresses slower but decompresses just as fast as 1. It is recommended for archives to be compressed once and decompressed many times, such as downloads. -m0 stores with deduplication but no further compression.

Blocksize says to pack fragments into blocks up to 2^Blocksize MiB. Using larger blocks can improve compression but require more memory and may be slower because each block is compressed or decompressed by a separate thread. The memory requirement is up to 8 times Blocksize per thread for levels up to 4 and 16 times block size per thread for level 5. The default Blocksize is 4 (16 MiB) for types 0 and 1, and 6 (64 MiB) otherwise.

[...]

I would first try using -m2 through -m5 to see if you can get similar compression (-m1 is the default and is designed for a balance between size and speed: it's equivalent to -1 in other programs.)

There are other tweaks that you can use (check the documentation), but unless you have to have a specific setting like -method x6.1.4.0.5.27.1 ("64 MiB blocks (6), variable length LZ77 without E8E9 (1), minimum match length 4, no secondary search (0), search depth 2^5 = 32 in each direction in the suffix array (27 = 6 + 21), and 1 byte lookahead" [from the documentation!]), don't worry about it; the predefined settings likely work fine.

If you absolutely must have the original FP8 program, it is available in source form on GitHub. You'll need git, nasm and the 32-bit GCC toolchain (the programs have 32-bit assembly files, which prevents them from being built as 64-bit programs.)

3
  • Thanks a lot for the answer. However, the link to newer versions of zpaq takes to the same one that is in Ubuntu repositories (v1.10). The latest version is v7.15 can be found here: mattmahoney.net/dc/zpaq715.zip I downloaded and compiled it in my system. But unfortunately I can't get anywhere close to the 81% compression of fp8. I've tried many options ( -m5, -m511, -m511.3 and others), and the best is ~96%. Any suggestions? Apr 9, 2018 at 16:55
  • Without testing all the options, I don't know. This is a really specific zpaq question and I'm not sure which tuning options would help here. Possibly checking with the encode.ru forum or SuperUser would get you further; if all else fails, try contacting the author (at the bottom of this page): I can't guarantee that he will respond, though.
    – ErikF
    Apr 9, 2018 at 17:16
  • Ubuntu 19.10 repositories have the latest version: 7.15
    – caduceus
    Jan 20, 2020 at 6:49
1

After some extra research I found out that the latest generation of the paq8 algorithm (of which fp8 was a version) is being developed in the paq8pxd software. The git repository is here, and the history of previous versions is here. And benchmark tests, binaries and more information is often posted in the Paq8pxd dict thread at encode.ru.

paq8 includes a JPG model, therefore its ability to compress JPG files loselessly down to 70-80% of its original size. However, zpaq, despite being a newer package, do not include such model, explaining its much lower performance on JPG images.

Regarding zpaq, the version in Ubuntu repositories is very outdated (v1.10). The latest version (v7.14) can be found here, and it includes a Makefile that makes very easy to compile in Ubuntu. However, it still doesn't include a JPG model, therefore, it doesn't do very good on jpg files, and is outperformed in that particular application by older packages based on paq8.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.