rsync-like updating of archive file

Question

When updating a backup using rsync --include-from=files --exclude-deleted, you can get feedback something like:

Added file: new.file
Added file: other.file
Deleted file: deleted.file
etc.

Is it possible to achieve the same with a standard UNIX archiving tool (zip, tar, etc.), eg. new files are added to the archive, and deleted files are removed?

For example,zip -ru@ out < files updates an archive based on an inclusion list, but deleted files are not deleted. So you're left with a lot of leftover trash in the archive.

Of course, you can delete the archive and recreate the whole thing, but then you get like thousands lines of output which say that all files were added, so you get no "delta" output, so to speak.

(A side question is how you can include files based on wildcards in the inclusion file, but this may be suited for a separate question.)

Answer 1

You can use the option --filesync from zip.

From the zip(1) man page:

-FS

--filesync Synchronize the contents of an archive with the files on the OS. Normally when an archive is updated, new files are added and changed files are updated but files that no longer exist on the OS are not deleted from the archive. This option enables a new mode that checks entries in the archive against the file system. If the file time and file size of the entry matches that of the OS file, the entry is copied from the old archive instead of being read from the file system and compressed. If the OS file has changed, the entry is read and compressed as usual. If the entry in the archive does not match a file on the OS, the entry is deleted. Enabling this option should create archives that are the same as new archives, but since existing entries are copied instead of compressed, updating an existing archive with -FS can be much faster than creating a new archive. Also consider using -u for updating an archive. For this option to work, the archive should be updated from the same directory it was created in so the relative paths match. If few files are being copied from the old archive, it may be faster to create a new archive instead. Note that the timezone environment variable TZ should be set according to the local timezone in order for this option to work correctly. A change in timezone since the original archive was created could result in no times matching and recompression of all files.

This option deletes files from the archive. If you need to preserve the original archive, make a copy of the archive first or use the --out option to output the updated archive to a new file. Even though it may be slower, creating a new archive with a new archive name is safer, avoids mismatches between archive and OS paths, and is preferred.

Example:

user@host:~/$ mkdir compressme; echo "Lorem Ipsum" >compressme/file1; echo "Lorem Ipsum" >compressme/file2; echo "Lorem Ipsum" >compressme/file3
user@host:~/$ zip -r --filesync all.zip compressme
  adding: compressme/ (stored 0%)
  adding: compressme/file2 (stored 0%)
  adding: compressme/file1 (stored 0%)
  adding: compressme/file3 (stored 0%)
user@host:~/$ rm compressme/file2
user@host:~/$ zip -r --filesync all.zip compressme
updating: compressme/ (stored 0%)
deleting: compressme/file2

Answer 2

For (GNU) tar(1), there is -u (update). But it does not update in place, it just appends the new contents at the end (the result contains several versions of the file, real fun to sort out later...), -a (append) and -delete (the manual doesn't say if it is really deleted, or just marked non-existent, I suspect the later).

The ar(1) format (used for static libraries, most often; but a bona fide archive format) allows to delete, move and quick append.

Here cpio(1) has a big fat warning that it should not be used, as it isn't portable.

The Unix standard pax(1) has no provisions to modify archives (no wonder, here spax(1) handles a smattering of formats).

If the archive is compressed, you are out of luck. What you do with the above tools is to archive, and compress the result. Any modification of the archive contents requires recompressing the whole.

Note that the zip(1) format is a bunch of individually compressed files, adding/removing/moving stuff in there is (relatively) easy.