I have several directories/web site resources, each has total size of several gigabytes. They are more than 90% the same. Sometimes only a few files are modified.
I need to download these files for backup purposes periodically. Storage space is no concern, but it is taking too long to download.
I would like to:
Use one such directory as reference (d1).
1. For another directory(d2), list all files that are identical between d1 and d2, save it into common.txt. Identical means relative path(within d1 and d2), filename and contents are the same, ignoring the owner/group and timestamps.
2. Copy all files, that are in d2 but not in d1 (includes different files) into d3.
Then I will download tarballed d1, d3 and common.txt to another machine.
Copy d3 to d2.
3. Copy files listed in common.txt from d1 to d2.
Could you please suggest how to resolve steps (1), (2), (3)?