8

I'm looking for a way to update thousands of .tbz archive files, so I'll be doing this with a shell script. I need to add one file to each.

My question is, is there a faster way to do this without extracting each tbz's contents, then re-compressing with the new file included in the contained tar? What would the commands look like?

Thanks

  • An obvious alternative is to put compressed files into an uncompressed tarball. But that changes the data format so it might not be viable for you, and it can be inefficient for large numbers of small files with redundancy between them. – Reinstate Monica Feb 15 '18 at 20:33
  • Discussion continued on Meta – Murphy Feb 16 '18 at 10:15
12

While tar can add files to an already existing archive, it cannot be compressed. You will have to bunzip2 the compressed archive, leaving a standard tarball. You can then use tar's ability to add files to an existing archive, and then recompress with bzip2.

From the manual:

 -r      Like -c, but new entries are appended to the archive.  Note that this only
         works on uncompressed archives stored in regular files.  The -f option is
         required.
| improve this answer | |
  • it's indeed true – Kiwy Feb 15 '18 at 16:20
  • This is one method, but it isn't the only method. It is possible to modify a bzip2 stream without completely uncompressing it. I don't know if it's possible to to it in a way that allows appending cleanly to a tar archive, but dhag shows a partial method. – Gilles 'SO- stop being evil' Feb 15 '18 at 18:48
10

The other answer is correct: you cannot properly update a compressed tar archive without uncompressing it. The GNU tar documentation hints at it, and attempting to update fails with an explicit error message:

$ tar --concatenate --file=cat.tar.bz2 two.tar.bz2 
tar: Cannot update compressed archives
tar: Error is not recoverable: exiting now

However, should you be interested in a dirty sort-of-works solution that doesn't require decompression, I can provide one, based on the following observations:

  • Appending bzip2 streams using cat is supported and produces a valid bzip2 stream (the same is true of gzip);
  • appending tars using cat does not produce a valid tar file, which is why the --concatenate option exists, but we can ask tar to pretend it's valid:

It may seem more intuitive to you to want or try to use cat to concatenate two archives instead of using the --concatenate operation; after all, cat is the utility for combining files.

However, tar archives incorporate an end-of-file marker which must be removed if the concatenated archives are to be read properly as one archive. --concatenate removes the end-of-archive marker from the target archive before each new archive is appended. If you use cat to combine the archives, the result will not be a valid tar format archive. If you need to retrieve files from an archive that was added to using the cat utility, use the --ignore-zeros (-i) option.

Based on this knowledge, we can do, for example:

cat {one,two}.tar.bz2 >combined.tar.bz2

This results, as the documentation snippet above explains, in an invalid tar file, but using --ignore-zeros, it can still be read fully:

## Show contents of `one.tar.bz2'
$ tar tf one.tar.bz2
a
b

## Show contents of `two.tar.bz2'
$ tar tf two.tar.bz2
c

## Show contents of `combined.tar.bz2', bypassing the bad format
$ tar tif combined.tar.bz2
a
b
c

Note how the above lists all three files from the original two archives, whereas omitting -i (correctly) lists only the files from the first original archive:

$ tar tf combined.tar.bz2 
a
b

Once again, that's nothing more than a dirty trick, but it could be useful if you control both the writing and reading sides and can make sure that -i will be used when attempting to read from files created in this way.

| improve this answer | |
  • That very interesting thanks for the detailed response. Unfortunately I am not in control of the process that will ultimately read these archives. – BottleZero Feb 15 '18 at 17:34
  • This can be used to "hide" some files in an archive so that someone casually extracting will only get the contents of the original tarball, but those who know to add the i to the command line will see the whole thing. – Monty Harder Feb 15 '18 at 22:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.