Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am trying to save space while doing a "dumb" backup by simply dumping data into a text file. My backup script is executed daily and looks like this:

  1. Create a directory named after the backup date.
  2. Dump some data into a text file "$name".
  3. If the file is valid, gzip it: gzip "$name". Otherwise, rm "$name".

Now I want to add an additional step to remove a file if the same data was also available in the day before (and create symlink or hardlink).

At first I thought of using md5sum "$name", but this does not work because I also store the filename and creation date.

Does gzip have an option to compare two gzipped files and tell me whether they are equal or not? If gzip does not have such an option, is there another way to achieve my goal?

share|improve this question
1  
Try this: linux.die.net/man/1/zdiff –  mreithub Feb 8 '13 at 19:01
    
I was going to suggest diff <(zcat file1) <(zcat file2), but mrethub's suggestion of zdiff looks much better. –  Kevin Feb 8 '13 at 19:06
    
backuppc does for you what you are trying to achieve manually –  drone.ah Feb 8 '13 at 19:16
    
@drohne.ah backuppc might be kind of an overkill if it's just one file per day... (I guess it's sth like an SQL dump where it makes a lot of sense to gzip) –  mreithub Feb 8 '13 at 20:34
1  
@mdpc The algorithm problems in MD5 probably aren't relevant. It's possible to construct collisions, but likely the only concern is ones that happen by chance, not by an attacker. And that's still unlikely to happen until you have ~2^64 files. Even a preimage attack probably doesn't matter. –  derobert Feb 8 '13 at 23:40
show 6 more comments

3 Answers 3

You can use zcmp or zdiff as mreithub suggests in his comment (or Kevin's command, which is similar). These will be relatively inefficient, as they actually decompress both files and then pass them off to cmp or diff. If you just want to answer "are they the same", you want cmp, it'll be much faster.

Your approach with the md5sum is perfectly good, but you need to take the MD5 before running gzip. Then store it in a file alongside the resulting .gz file. You can then compare the file easily, before compressing it. If the name is the same, md5sum -c will do this for you.

$ mkdir "backup1"
$ cd backup1
$ echo "test" > backup-file
$ md5sum backup-file > backup-file.md5
$ gzip -9 backup-file

And the next backup:

$ mkdir "backup2"
$ cd backup2
$ echo "test" > backup-file
$ md5sum -c ../backup1/backup-file.md5 
backup-file: OK

So it hasn't changed. OTOH, had it changed:

$ echo "different" > backup-file
$ md5sum -c ../backup1/backup-file.md5 
backup-file: FAILED
md5sum: WARNING: 1 computed checksum did NOT match

If you pass --quiet to it, it'll just give you the exit code. 0 for matched, non-0 for differed.

MD5 is fairly quick, but not blazingly so. MD4 (openssl md4 is the best you get on the command line, I believe) is around twice as fast (neither it nor MD5 is secure, but both are about as collision resistant when no one is attempting to subvert them). SHA-1 (sha1sum) is more secure, but slower; SHA-256 (sha256sum) is secure, but even slower still. CRC32 should be many times faster, but is shorter and thus will have more random collisions. Its also entirely insecure.

share|improve this answer
    
zdiff seems a waste as I just want to know whether a file has changed, not what. zcmp looks interesting, I will try that. –  Lekensteyn Feb 8 '13 at 21:24
add comment
up vote 2 down vote accepted

@deroberts answer is great, though I want to share some other information that I have found.

gzip -l -v

gzip-compressed files contain already a hash (not secure though, see this SO post):

$ echo something > foo
$ gzip foo
$ gzip -v -l foo.gz 
method  crc     date  time           compressed        uncompressed  ratio uncompressed_name
defla 18b1f736 Feb  8 22:34                  34                  10 -20.0% foo

One can combine the CRC and uncompressed size to get a quick fingerprint:

gzip -v -l foo.gz | awk '{print $2, $7}'

cmp

For checking whether two bytes are equal or not, use cmp file1 file2. Now, a gzipped file has some header with the data and footer (CRC plus original size) appended. The description of the gzip format shows that the header contains the time when the file was compressed and that the file name is a nul-terminated string that is appended after the 10-byte header.

So, assuming that the file name is constant and the same command (gzip "$name") is used, one can check whether two files are different by using cmp and skipping the first bytes including the time:

cmp -i 8 file1 file2

Note: the assumption that the same compression options is important, otherwise the command will always report the file as different. This happens because the compression options are stored in the header and may affect the compressed data. cmp just looks at raw bytes and do not interpret it as gzip.

If you have filenames of the same length, then you could try to calculate the bytes to be skipped after reading the filename. When the filenames are of different size, you could run cmp after skipping bytes, like cmp <(cut -b9- file1) <(cut -b10- file2).

zcmp

This is definitely the best way to go, it first compresses data and starts comparing the bytes with cmp (really, this is what is done in the zcmp (zdiff) shellscript).

One note, do not be afraid of the following note in the manual page:

When both files must be uncompressed before comparison, the second is uncompressed to /tmp. In all other cases, zdiff and zcmp use only a pipe.

When you have a sufficiently new Bash, compression will not use a temporary file, just a pipe. Or, as the zdiff source says:

# Reject Solaris 8's buggy /bin/bash 2.03.
share|improve this answer
add comment

To compare two gzip files, just the contents, one command, no diff, just comparing md5sum

$ diff -q <(zcat one.gz|md5sum|cut -f1 -d' ') \
          <(zcat two.gz|md5sum|cut -f1 -d' ') \
    && echo same || echo not_same

You can also "filter" for relevant differences,

$ diff -q <(zcat one.gz|grep -v '^-- Dump completed'|md5sum|cut -f1 -d' ') \
          <(zcat two.gz|grep -v '^-- Dump completed'|md5sum|cut -f1 -d' ') \
   && echo same || echo not_same

If scripting, I'd recommend a filter function (not tested, just an example),

do_filter_sum() {
  zcat $1 | grep -v '^-- Dump completed' | md5sum | cut -f1 -d' '
}

diff -q <(do_filter_sum one.gz) \
        <(do_filter_sum two.gz) \
        && echo same || echo not_same
share|improve this answer
    
The md5sum is a waste, you can use cmp. zcat and grep can be merged into zgrep. –  Lekensteyn Mar 14 '13 at 16:22
    
true, md5sum isn't necessary to compare (unless you already have them generated); I just used it since derobert used it. zgrep is just a script that basically does gunzip and grep (or sed as the case may be), so there is little difference there. the script as posted is intentionally shown as a chain of pipes with pluggable parts; what's the fun in merging everything down into a single command? –  michael_n Mar 14 '13 at 20:19
    
And zcat is just gunzip -c. Use the right tool for the right job, KISS is better than bloat. In this case I would spend my time on writing something that generates hard links as needed, that's more fun. –  Lekensteyn Mar 14 '13 at 20:51
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.