Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Short question: I have two dynamically generated tar archives (so they have different timestamps), how can I compare them, ignoring any different in time?

Backgrounds...

I am doing some backup, in which I use a script to generate things that needs to be backed up, put them in to a directory, then tar the directory and keep several old versions. The backup script needs to run every 30 minutes to make sure we don't lose hours of work.

Now I realize that there are periods of time that the data doesn't change, so it doesn't make sense to store duplicates of the same thing over and over again. I would like to compare the archives before saving. My attempt was to run cmp newdata.tar.gz olddata.tar.gz and only store newdata.tar.gz if it contains new data. Apparently that didn't work, because there are different timestamps.

share|improve this question
    
I'm suspecting timestamps because everything else is the same. Even diff reports nothing else but time changes. –  phunehehe Apr 1 '11 at 13:47
add comment

4 Answers

Here are three approaches. The first has my preference.

Change the file generation process

Instead of regenerating the files indiscriminately (dump_table foo >foo.dump), keep the old file if it's identical to the new one.

dump_table foo >foo.dump.new
if cmp foo.dump foo.dump.new; then
  rm foo.dump.new
else
  mv foo.dump.new foo.dump
fi

Mount the archives as directories

Use AVFS to create a view of your entire directory hierarchy where all archives have an associated directory (same name with # tacked on at the end) that appears to hold the archive content.

mountavfs
d=$(date -d %Y%m%d%H%M%S); mkdir $d; tar czf /path/to/$d.tgz $d; rmdir $d
cd /path/to/back/up
rsync -ac --compare-dest=~/.avfs/path/to/yesterday.tgz . ~/.avfs/path/to/$d.tgz

Use a backup tool with the requisite feature

Backup programs need to be extremely reliable. It's hard for a home-grown solution to cover all cases. So consider a full-blown backup program. I think duplicity would do what you require: it performs incremental backups, and it uses the rsync algorithm, so it shouldn't waste any space on data that's already there.

share|improve this answer
add comment

One option is to use a backup tool like backup2l, that can be configured to any level of differential backups and any number of full backups. backup2l is ran as a cronjob in any frequency you like and it is configured by setting some values to its conf file. It is actually a wrapper for tar or afio, it keeps file listings with hashes for finding changes and provides an easy way to get status or restore a file by date.

A second option is to use a version control system such as cvs, svn, git etc. Setup a cronjob that will make automatic commits (and maybe daily tagging). Based on the vcs choice, you may need some scripting to add new files or remove old ones.

For a frequency of every half hour, I'd recommend the vcs option. But you can combine the two by using backup2l (or any other backup tool) to also backup the vcs repository (redundancy of backups is always good).

share|improve this answer
    
+1. Another good tool is rdiff-backup. –  rvs Apr 2 '11 at 19:43
add comment

Do the files inside the archives change, or are just new ones added?

If you only add files, and don't change any, try something like this:

tar ztf olddata.tar.gz | sort > tmpfile1
tar ztf newdata.tar.gz | sort > tmpfile2
diff tmpfile{1,2} > /dev/null
if test $? -eq 0
then
  echo ignore newdata.tar.gz
else
  echo keep olddata.tar.gz
fi
rm tmpfile{1,2}

Note the absence of v from the tar table of contents commands.

If you just want to skip archives with identical contents, add a v to the mix, as in:

tar ztvf olddata.tar.gz | sort > tmpfile1

and so on.

share|improve this answer
    
Actually, the files inside the archives were generated through several ways, including querying the database... –  phunehehe Apr 1 '11 at 10:03
    
@phunehehe: see my last edit, and give it a try. Just compare the (sorted) table of contents from the archives you want to compare. Omit the > /dev/null to see what has changed between revisions. –  MattBianco Apr 1 '11 at 10:07
    
No I don't want to skip files just because of the name, I want to check the contents as well. –  phunehehe Apr 1 '11 at 10:14
    
@phunehehe: the mtime of files change when they are updated, the diff of the sorted output of tar ztvf will get them. Try it –  MattBianco Apr 1 '11 at 11:39
1  
@phunehehe: then tar is not the proper tool to do incremental backups. You could probably keep a "staging area" for the previous backups contents, and diff each file before overwriting, and then archive the staging area if at least 1 file has changed since the last time. –  MattBianco Apr 1 '11 at 13:36
show 1 more comment

Try this, then:

OLDSUM=`tar zxOf olddata.tar.gz | md5sum | awk '{print $1}'`
NEWSUM=`tar zxOf newdata.tar.gz | md5sum | awk '{print $1}'`
if test "${NEWSUM}" != "${OLDSUM}"
then
  echo save newdata.tar.gz
else
  echo nothing changed
fi

The capital O extracts the archive contents to stdout.

It will not work, however, unless the files are added to the archive in the exact same order (which they probably are, though).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.