34

I have a few thousand files that are individually GZip compressed (passing of course the -n flag so the output is deterministic). They then go into a Git repository. I just discovered that for 3 of these files, Gzip doesn't produce the same output on macOS vs Linux. Here's an example:

macOS

$ cat Engine/Extras/ThirdPartyNotUE/NoRedist/EnsureIT/9.7.0/bin/finalizer | shasum -a 256
0ac378465b576991e1c7323008efcade253ce1ab08145899139f11733187e455  -

$ cat Engine/Extras/ThirdPartyNotUE/NoRedist/EnsureIT/9.7.0/bin/finalizer | gzip --fast -n | shasum -a 256
6e145c6239e64b7e28f61cbab49caacbe0dae846ce33d539bf5c7f2761053712  -

$ cat Engine/Extras/ThirdPartyNotUE/NoRedist/EnsureIT/9.7.0/bin/finalizer | gzip -n | shasum -a 256
3562fd9f1d18d52e500619b4a5d5dfa709f5da8601b9dd64088fb5da8de7b281  -

$ gzip --version
Apple gzip 272.250.1

Linux

$ cat Engine/Extras/ThirdPartyNotUE/NoRedist/EnsureIT/9.7.0/bin/finalizer | shasum -a 256
0ac378465b576991e1c7323008efcade253ce1ab08145899139f11733187e455  -

$ cat Engine/Extras/ThirdPartyNotUE/NoRedist/EnsureIT/9.7.0/bin/finalizer | gzip --fast -n | shasum -a 256
10ac8b80af8d734ad3688aa6c7d9b582ab62cf7eda6bc1a0f08d6159cad96ddc  -

$ cat Engine/Extras/ThirdPartyNotUE/NoRedist/EnsureIT/9.7.0/bin/finalizer | gzip -n | shasum -a 256
cbf249e3a35f62a4f3b13e2c91fe0161af5d96a58727d17cf7a62e0ac3806393  -

$ gzip --version
gzip 1.6
Copyright (C) 2007, 2010, 2011 Free Software Foundation, Inc.
Copyright (C) 1993 Jean-loup Gailly.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.

Written by Jean-loup Gailly.

How is this possible? I thought the GZip implementation was completely standard?

UPDATE: Just to confirm that macOS and Linux versions do produce the same output most of the time, both OSes output the same hash for:

$ echo "Vive la France" | gzip --fast -n | shasum -a 256
af842c0cb2dbf94ae19f31c55e05fa0e403b249c8faead413ac2fa5e9b854768  -
Improve this question
6
  • 4
    Can you post a hexdump or xxd of the these gzip files? That way we can analyze and know for sure whether it's a metadata/header difference or the data stream itself.
    – 640KB
    Commented Mar 1, 2020 at 17:30
  • 5
    It would be interesting if you posted those files (both compressed and the uncompressed) somewhere -- if they don't contain anything sensitive, of course.
    – user313992
    Commented Mar 1, 2020 at 18:25
  • 2
    For more gzip-related fun, offhand, I believe AdvanceComp compresses gzip as well as zip. That is, they take common files and compress them to be even smaller, while still being able to decompress right. So you can go to advancemame.it/comp-readme.html and download AdvanceComp and run advzip -z4pki 10000 file.gz to compress a typical small gzip file (using gzip -9) into a smaller gzip file.
    – TOOGAM
    Commented Mar 2, 2020 at 7:39
  • 6
    I see no guarantee in the docs that (-n (i.e. --noname)) produces deterministic behavior. This would then seem to be a faulty assumption on your part. Some of the answers already explain that the used compression has certain parameters or freedom to do different things. The options --fast and --best also already indicate that the process is not deterministic. So why expect the same behavior from these two different programs?
    – hkBst
    Commented Mar 2, 2020 at 16:40
  • 2
    @hkBst, I think the behavior is expected, because most of the time the results are identical. There must be a reason why it works most of the time, but not in some situations.
    – JPhi1618
    Commented Mar 2, 2020 at 20:11

4 Answers 4

Reset to default
63

Note that the compression algorithm (Deflate) in GZip is not strictly bijective. To elaborate: For some data, there's more than one possible compressed output depending on the algorithmic implementation and used parameters. So there's no guarantee at all that Apple GZip and gzip 1.6 will return the same compressed output. These outputs are all valid GZip streams, the standard just guarantees that every of these possible outputs will be decompressed to the same original data.

Improve this answer
6
  • 1
    That's what I thought too (different compression levels, etc), but according to the OP, it's only 3 files out of thousands which are different between the 2 OSes.
    – user313992
    Commented Mar 1, 2020 at 18:29
  • 16
    Doesn't change much. e.g. different deflate implementations can decide to split big data into different sized chunks or change the compression strategy depending on file content. So Apple Gzip and gzip 1.6 could be so similar that they make the same decisions in most cases, but with some exceptions.
    – schnaader
    Commented Mar 1, 2020 at 18:38
  • 5
    Well, for a detailed definite answer, testing using the files from OP would be needed (and/or code-diving in both implementations). It's true that without them, we are speculating about the source of the difference. But it still is a fact that there are more possible differences than one would expect. Another example is that there are two possible encodings of match length 258 (see stackoverflow.com/questions/27152629/… ) - which one is used depends on the implementation, and the difference would only occur on repetitive data.
    – schnaader
    Commented Mar 2, 2020 at 11:54
  • 7
    Even two different versions of gnu gzip could potentially produce different compressed results, depending on changes in the algorithms used between versions. And, at a glance, i'm not sure that this particular version of Apple gzip and this particular version of GNU gzip actually are derived from the same source. The same upstream project? Sure. But the upstream source for the apple implementation could be from different versions of that source than the gnu implementation. And that's assuming they aren't completely independent forks started at different times in gzip's development.
    – Cliff Armstrong
    Commented Mar 2, 2020 at 19:11
  • 4
    This is enough to answer the main part of the question: How is this possible
    – slebetman
    Commented Mar 3, 2020 at 11:57
20

The format should be very stable, but see its description. It contains a field for operating system ID.  Obviously that may differ for macOS and Linux and FreeBSD and ...

Improve this answer
9
  • 6
    I believe both OS's should produce files with the OS byte of 3 - Unix. This field is a legacy from the original PKZIP file format and any *nix-like OS should get a 3. Perhaps the difference has to do with OS-specific file attributes or differences in the POSIX ACLs. Or most likely I think @schnaader's comment below is probably the case. Perhaps the OP can post binary dumps of the created files so we can be certain.
    – 640KB
    Commented Mar 1, 2020 at 17:29
  • 3
    @640KB however, it's absolutely true that all of MacOS, FreeBSD, Linux use the same OS id (3 = Unix) -- you can see the whole list here.
    – user313992
    Commented Mar 1, 2020 at 18:21
  • 2
    If indeed the operating system ID is the same for Linux and macOS, then this answer (as written) is unfortunately incorrect.
    – Pol
    Commented Mar 1, 2020 at 19:36
  • 2
    The linked format description has Unix =3 and Macintosh =7.does the latter apply not to modern macOS?
    – planetmaker
    Commented Mar 2, 2020 at 10:58
  • 4
    echo hello | gzip | hexdump -C -s 9 -n 1 is a pretty easy verification of what the platform code is anywhere (it is 03 under both Apple gzip on macOS and GNU gzip on other Unix-like platforms).
    – Michael Homer
    Commented Mar 2, 2020 at 20:35
10

Gzip format is standard, the implementation - not necessarily. Wikipedia lists at least 5 free/oss independent implementations and there are also proprietary ones. Apple clearly outputs a different version string.

The format and the algorithm both allow for a lot of freedom and a lot of design choices that are either matter of taste and/or work better in different use cases.

See Zip Files: History, Explanation and Implementation

I generally would expect the results to be the same between different implementations only for a small percent of small-ish files.

Improve this answer
6
  • "The format and the algorithm both allow for a lot of freedom and a lot of design choices" are there any hard facts behind this claim? FWIW, apple's gzip is using the standard zlib library.
    – user313992
    Commented Mar 2, 2020 at 11:01
  • Apple uses the standard zlib library (zlib.h), but looking at the Linux gzip code ( though I only found it for 1.10 instead of 1.6) at fossies.org/linux/gzip/deflate.c their implementation seems to be a custom one. Also, even if both would be using zlib.h, the code that wraps the deflate calls (deflateInit, deflate, etc.) could be different.
    – schnaader
    Commented Mar 2, 2020 at 12:52
  • Follow-up: gzip 1.6 source code can be found at ftp.gnu.org/gnu/gzip and also doesn't use zlib.h
    – schnaader
    Commented Mar 2, 2020 at 13:02
  • @schnaader nope, zlib is derived ("abstracted") from the gzip program and both were written by the same people.
    – user313992
    Commented Mar 2, 2020 at 14:12
  • 2
    And I still wouldn't conclude that they always give the same compressed output from this. Looking at the gzip changelog at fossies.org/linux/gzip/ChangeLog , there have been many changes. Have these been adopted in Apple's code? Can't even find out which date Apple's gzip 272.250.1 is from. Also found this Debian bug report for gzip 1.4-1 ( bugs.debian.org/cgi-bin/bugreport.cgi?bug=647522 ) where compressed output wasn't the same always. Fixed in 1.4-5, but demonstrates the problem and gives another past example. The cause was non-zeroed memory polluting the dictionary.
    – schnaader
    Commented Mar 2, 2020 at 15:23
0

Are you sure the files before compression are identical? Some VCS chekout text files differently, using UTF8 or not, windows or linux newlines, ...

Run the SHA-command on the original files to see if you are doing the same thing.

Maybe try compression level 0 to see if that works properly.

Find some simple files you can post here that get encoded differently on both systems.

Do the files get unzipped correctly on both systems? Run the SHA-command again.

And always ask yourself: does it matter? :)

Improve this answer
1
  • 8
    Yes, the files are identical: see in my question how I compute their SHA256 on both platforms before gzip.
    – Pol
    Commented Mar 2, 2020 at 2:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .