Short question: I have two dynamically generated tar archives (so they have different timestamps), how can I compare them, ignoring any different in time?


I am doing some backup, in which I use a script to generate things that needs to be backed up, put them in to a directory, then tar the directory and keep several old versions. The backup script needs to run every 30 minutes to make sure we don't lose hours of work.

Now I realize that there are periods of time that the data doesn't change, so it doesn't make sense to store duplicates of the same thing over and over again. I would like to compare the archives before saving. My attempt was to run cmp newdata.tar.gz olddata.tar.gz and only store newdata.tar.gz if it contains new data. Apparently that didn't work, because there are different timestamps.

  • I'm suspecting timestamps because everything else is the same. Even diff reports nothing else but time changes.
    – phunehehe
    Commented Apr 1, 2011 at 13:47

4 Answers 4


Here are three approaches. The first has my preference.

Change the file generation process

Instead of regenerating the files indiscriminately (dump_table foo >foo.dump), keep the old file if it's identical to the new one.

dump_table foo >foo.dump.new
if cmp foo.dump foo.dump.new; then
  rm foo.dump.new
  mv foo.dump.new foo.dump

Mount the archives as directories

Use AVFS to create a view of your entire directory hierarchy where all archives have an associated directory (same name with # tacked on at the end) that appears to hold the archive content.

d=$(date -d %Y%m%d%H%M%S); mkdir $d; tar czf /path/to/$d.tgz $d; rmdir $d
cd /path/to/back/up
rsync -ac --compare-dest=~/.avfs/path/to/yesterday.tgz . ~/.avfs/path/to/$d.tgz

Use a backup tool with the requisite feature

Backup programs need to be extremely reliable. It's hard for a home-grown solution to cover all cases. So consider a full-blown backup program. I think duplicity would do what you require: it performs incremental backups, and it uses the rsync algorithm, so it shouldn't waste any space on data that's already there.


One option is to use a backup tool like backup2l, that can be configured to any level of differential backups and any number of full backups. backup2l is ran as a cronjob in any frequency you like and it is configured by setting some values to its conf file. It is actually a wrapper for tar or afio, it keeps file listings with hashes for finding changes and provides an easy way to get status or restore a file by date.

A second option is to use a version control system such as cvs, svn, git etc. Setup a cronjob that will make automatic commits (and maybe daily tagging). Based on the vcs choice, you may need some scripting to add new files or remove old ones.

For a frequency of every half hour, I'd recommend the vcs option. But you can combine the two by using backup2l (or any other backup tool) to also backup the vcs repository (redundancy of backups is always good).


Do the files inside the archives change, or are just new ones added?

If you only add files, and don't change any, try something like this:

tar ztf olddata.tar.gz | sort > tmpfile1
tar ztf newdata.tar.gz | sort > tmpfile2
diff tmpfile{1,2} > /dev/null
if test $? -eq 0
  echo ignore newdata.tar.gz
  echo keep olddata.tar.gz
rm tmpfile{1,2}

Note the absence of v from the tar table of contents commands.

If you just want to skip archives with identical contents, add a v to the mix, as in:

tar ztvf olddata.tar.gz | sort > tmpfile1

and so on.

  • Actually, the files inside the archives were generated through several ways, including querying the database...
    – phunehehe
    Commented Apr 1, 2011 at 10:03
  • @phunehehe: see my last edit, and give it a try. Just compare the (sorted) table of contents from the archives you want to compare. Omit the > /dev/null to see what has changed between revisions.
    – MattBianco
    Commented Apr 1, 2011 at 10:07
  • No I don't want to skip files just because of the name, I want to check the contents as well.
    – phunehehe
    Commented Apr 1, 2011 at 10:14
  • @phunehehe: the mtime of files change when they are updated, the diff of the sorted output of tar ztvf will get them. Try it
    – MattBianco
    Commented Apr 1, 2011 at 11:39
  • 1
    @phunehehe: then tar is not the proper tool to do incremental backups. You could probably keep a "staging area" for the previous backups contents, and diff each file before overwriting, and then archive the staging area if at least 1 file has changed since the last time.
    – MattBianco
    Commented Apr 1, 2011 at 13:36

Try this, then:

OLDSUM=`tar zxOf olddata.tar.gz | md5sum | awk '{print $1}'`
NEWSUM=`tar zxOf newdata.tar.gz | md5sum | awk '{print $1}'`
if test "${NEWSUM}" != "${OLDSUM}"
  echo save newdata.tar.gz
  echo nothing changed

The capital O extracts the archive contents to stdout.

It will not work, however, unless the files are added to the archive in the exact same order (which they probably are, though).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .