I am trying to save space while doing a "dumb" backup by simply dumping data into a text file. My backup script is executed daily and looks like this:

  1. Create a directory named after the backup date.
  2. Dump some data into a text file "$name".
  3. If the file is valid, gzip it: gzip "$name". Otherwise, rm "$name".

Now I want to add an additional step to remove a file if the same data was also available in the day before (and create symlink or hardlink).

At first I thought of using md5sum "$name", but this does not work because I also store the filename and creation date.

Does gzip have an option to compare two gzipped files and tell me whether they are equal or not? If gzip does not have such an option, is there another way to achieve my goal?

  • 1
    Try this: linux.die.net/man/1/zdiff
    – mreithub
    Feb 8 '13 at 19:01
  • 2
    I was going to suggest diff <(zcat file1) <(zcat file2), but mrethub's suggestion of zdiff looks much better.
    – Kevin
    Feb 8 '13 at 19:06
  • backuppc does for you what you are trying to achieve manually
    – drone.ah
    Feb 8 '13 at 19:16
  • @drohne.ah backuppc might be kind of an overkill if it's just one file per day... (I guess it's sth like an SQL dump where it makes a lot of sense to gzip)
    – mreithub
    Feb 8 '13 at 20:34
  • 1
    @mdpc The algorithm problems in MD5 probably aren't relevant. It's possible to construct collisions, but likely the only concern is ones that happen by chance, not by an attacker. And that's still unlikely to happen until you have ~2^64 files. Even a preimage attack probably doesn't matter.
    – derobert
    Feb 8 '13 at 23:40

@deroberts answer is great, though I want to share some other information that I have found.

gzip -l -v

gzip-compressed files contain already a hash (not secure though, see this SO post):

$ echo something > foo
$ gzip foo
$ gzip -v -l foo.gz 
method  crc     date  time           compressed        uncompressed  ratio uncompressed_name
defla 18b1f736 Feb  8 22:34                  34                  10 -20.0% foo

One can combine the CRC and uncompressed size to get a quick fingerprint:

gzip -v -l foo.gz | awk '{print $2, $7}'


For checking whether two bytes are equal or not, use cmp file1 file2. Now, a gzipped file has some header with the data and footer (CRC plus original size) appended. The description of the gzip format shows that the header contains the time when the file was compressed and that the file name is a nul-terminated string that is appended after the 10-byte header.

So, assuming that the file name is constant and the same command (gzip "$name") is used, one can check whether two files are different by using cmp and skipping the first bytes including the time:

cmp -i 8 file1 file2

Note: the assumption that the same compression options is important, otherwise the command will always report the file as different. This happens because the compression options are stored in the header and may affect the compressed data. cmp just looks at raw bytes and do not interpret it as gzip.

If you have filenames of the same length, then you could try to calculate the bytes to be skipped after reading the filename. When the filenames are of different size, you could run cmp after skipping bytes, like cmp <(cut -b9- file1) <(cut -b10- file2).


This is definitely the best way to go, it first compresses data and starts comparing the bytes with cmp (really, this is what is done in the zcmp (zdiff) shellscript).

One note, do not be afraid of the following note in the manual page:

When both files must be uncompressed before comparison, the second is uncompressed to /tmp. In all other cases, zdiff and zcmp use only a pipe.

When you have a sufficiently new Bash, compression will not use a temporary file, just a pipe. Or, as the zdiff source says:

# Reject Solaris 8's buggy /bin/bash 2.03.
  • If byte 4 (FLG) is 0 then the file name is not in the header, so you don't need to worry about its length. Also, I found gzip -v -l will report the file time instead of MTIME if the four MTIME bytes in the header are zero. Also note if MTIME is there, it is typically a bit before the file time because it's when the compression started.
    – kitchin
    Feb 8 '16 at 5:25

You can use zcmp or zdiff as mreithub suggests in his comment (or Kevin's command, which is similar). These will be relatively inefficient, as they actually decompress both files and then pass them off to cmp or diff. If you just want to answer "are they the same", you want cmp, it'll be much faster.

Your approach with the md5sum is perfectly good, but you need to take the MD5 before running gzip. Then store it in a file alongside the resulting .gz file. You can then compare the file easily, before compressing it. If the name is the same, md5sum -c will do this for you.

$ mkdir "backup1"
$ cd backup1
$ echo "test" > backup-file
$ md5sum backup-file > backup-file.md5
$ gzip -9 backup-file

And the next backup:

$ mkdir "backup2"
$ cd backup2
$ echo "test" > backup-file
$ md5sum -c ../backup1/backup-file.md5 
backup-file: OK

So it hasn't changed. OTOH, had it changed:

$ echo "different" > backup-file
$ md5sum -c ../backup1/backup-file.md5 
backup-file: FAILED
md5sum: WARNING: 1 computed checksum did NOT match

If you pass --quiet to it, it'll just give you the exit code. 0 for matched, non-0 for differed.

MD5 is fairly quick, but not blazingly so. MD4 (openssl md4 is the best you get on the command line, I believe) is around twice as fast (neither it nor MD5 is secure, but both are about as collision resistant when no one is attempting to subvert them). SHA-1 (sha1sum) is more secure, but slower; SHA-256 (sha256sum) is secure, but even slower still. CRC32 should be many times faster, but is shorter and thus will have more random collisions. Its also entirely insecure.

  • zdiff seems a waste as I just want to know whether a file has changed, not what. zcmp looks interesting, I will try that.
    – Lekensteyn
    Feb 8 '13 at 21:24

To compare two gzip files, just the contents, one command, no diff, just comparing md5sum

$ diff -q <(zcat one.gz|md5sum|cut -f1 -d' ') \
          <(zcat two.gz|md5sum|cut -f1 -d' ') \
    && echo same || echo not_same

You can also "filter" for relevant differences,

$ diff -q <(zcat one.gz|grep -v '^-- Dump completed'|md5sum|cut -f1 -d' ') \
          <(zcat two.gz|grep -v '^-- Dump completed'|md5sum|cut -f1 -d' ') \
   && echo same || echo not_same

If scripting, I'd recommend a filter function (not tested, just an example),

do_filter_sum() {
  zcat $1 | grep -v '^-- Dump completed' | md5sum | cut -f1 -d' '

diff -q <(do_filter_sum one.gz) \
        <(do_filter_sum two.gz) \
        && echo same || echo not_same
  • The md5sum is a waste, you can use cmp. zcat and grep can be merged into zgrep.
    – Lekensteyn
    Mar 14 '13 at 16:22
  • true, md5sum isn't necessary to compare (unless you already have them generated); I just used it since derobert used it. zgrep is just a script that basically does gunzip and grep (or sed as the case may be), so there is little difference there. the script as posted is intentionally shown as a chain of pipes with pluggable parts; what's the fun in merging everything down into a single command?
    – michael
    Mar 14 '13 at 20:19
  • 1
    And zcat is just gunzip -c. Use the right tool for the right job, KISS is better than bloat. In this case I would spend my time on writing something that generates hard links as needed, that's more fun.
    – Lekensteyn
    Mar 14 '13 at 20:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.