6

My versioned project uses the contents of an upstream tarball. I would like to extract the contents in such a way as to be able to recreate the original tarball exactly. The reason is that I don't want to include big binary blobs in my versioned project. The retarring is unreliable, often producing slightly different tarballs. I suspect compression algorithm heuristics produce different results under different conditions.

EDIT: A further problem making the solution difficult is that some versioning systems (git) do not track empty dirs, thus excluding solutions than do not involve the creation of an auxiliary metadata file. See my comment to Hauke. (Thanks Hauke for this edit).

4
  • Which version of tar are you using? Which Unix/Linux? Which compressor is being used? Do you need to extract all of the files or only a few? Will you be modifying the files or extracting them for read only?
    – bsd
    Feb 18, 2014 at 20:42
  • @bdowning: Questions 1,2,3: I have no control of the upstream tarball format therefore the solution should be portable, though I'm willing to restrict portability to Gnu tar. 4,5: Having to reconstruct the original tarball you have to extract all files and not modify them.
    – Mayavimmer
    Feb 20, 2014 at 12:02
  • tar xf tarfile.tar will extract the contents, leaving the initial tarfile in place. Since you are not modifying any of the files contained, you can use the contents and leave the tarfile as is. Why do you need to "reconstruct" the tarfile?
    – bsd
    Feb 20, 2014 at 12:39
  • @bdowning: I need to track the upstream tarball in my versioned project. Imagine a 100MB tarball. Then one byte changes upstream. I commit a one byte change. I can still recreate the new modified tarball exactly. No need to add another 100MB to my project.
    – Mayavimmer
    Feb 20, 2014 at 22:06

2 Answers 2

5

Re-tarring the files can result in different archives because the order in which tar appends the files to the archive isn't necessarily stable. To ensure that the files are re-tarred in the same order as they were originally, list the files in the archive and save that list to a file:

tar -tf your-archive.tar > file-list.txt

Then re-tar them in the exact same order by reading the files to be added from the list you created earlier (since that list can also contain directories and we don't want to descend into them at that point, we need --no-recursion):

tar -cf your-new-archive.tar --no-recursion -T file-list.txt

Assuming that none of the files (or their attributes) were changed, your-new-archive.tar will be exactly identical to your-archive.tar.

You will however run into a problem when creating compressed archives: by default, gzip adds a timestamp to each compressed file it creates. To avoid that, you'll have to pipe tar's output to gzip (instead of using tar's -z switch) and disable the timestamps with -n:

tar -c --no-recursion -T file-list.txt | gzip -n > your-new-archive.tar.gz
1
  • +useful, thanks. Did not think the order could be arbitrary. Required, but not sufficient for full reconstruction. (Oops I'll upvote when I get 15 reputation)
    – Mayavimmer
    Feb 20, 2014 at 12:06
3

What you want is probably not possible at all, see this question:

UNIX ctime : how to keep this precious information in backups (tar)?

Thus if your original tar file contains ctime then you run into serious problems which cannot be solved with a reasonable amount of effort.

Another problem is a header string in the Gnu tar format which by default contains the PID of the process creating the archive. You can turn that off but that would not be enough as you have to copy it from the original.

Your chances are best if the original is in ustar format (you create that with --format=ustar).

And in order to keep the file order in the archive you should extract to an empty directory (or use the -T approach from the answer by n.st).

2
  • +useful (when I get 15 reputation). You bring up an interesting can of worms which makes this problem difficult. It looks like it cannot be solved with the tools tar gives you. On the other hand the requirement is only to recreate the original tarball, not to mirror file metadata exactly in the working directory. Therefore it seems possible to dig the file metadata out with a separate C program (reusing some of tar's parsing code), storing it in an auxiliary file, and using it to recreate the tarball using a second C program. 'Course it would be better if tar itself did it.
    – Mayavimmer
    Feb 20, 2014 at 12:16
  • @Mayavimmer :-) Feb 20, 2014 at 18:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .