How can I get tar (or any other program that is commonly available on Linux, e.g. pax) to hardlink duplicate content on-the-fly during archive creation?

That is, I'd like to avoid hardlinking upfront and instead would like tar to take care of it.

Is that possible? How?

  • Could you add some context about this? What is your use case? – Faheem Mitha Apr 12 '15 at 12:32
  • @FaheemMitha: I have a bunch of different versions of a particular program and want to bundle those (unpacked) folders which are named after the version into a single archive. When I tried hardlinking explicitly on disk I had savings up to 90% over using plain tar with deflate (gzip). – 0xC0000022L Apr 12 '15 at 18:15
  • Is this particular program your own work? If so, it sounds like you should consider distributed version control, and using multiple branches. – Faheem Mitha Apr 12 '15 at 18:26
  • @FaheemMitha: it is not, and I know VCSs of various kinds and know it would be the right thing to do. However, this is no option as it cannot be automated, like the simple creation of an archive can be. You can add files and so on, but you'll quickly hit limitations when trying to automate renames and the likes. Also moved files/folders (technically renames or vice versa, whichever way you prefer to look at it). – 0xC0000022L Apr 12 '15 at 21:01

This is not possible as of now with GNU tar, but things exist :

Note that hardlinking and deduping does not have the same semantics, one would need another kind of tar node type to represent "dupe data" in order for the archive extraction process to properly recreate duplicate (and thus indendently-living) files; that would create incompatible tar archives with most standard tools (GNU tar, pax, etc) which would be a bold move.

Let me insist : if you were able to assimilate dupes with hardlinked files, you would have a problem at archive extraction : * are they really hardlinked files like those of a Git repo ? Then they should be really recreated as hardlinked files otherwise the restored Git repo won't work ? * or are they really identical files, and if they were restored as hardlinks, the restored archive would probably have major data leaks (imagine you have assimilated identical /etc/passwd files while archiving, restore them as a single file with multiple hardlinks : one modification in one VM becomes visible in the other!)

  • thanks. Does that imply that I also need a particular patched version of tar to decompress? – 0xC0000022L Apr 12 '15 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.