0

Working Syntax for Rsync Copy of a set of 2 or more Hard-linked Folder Trees while maintaining this specific archive structure?

PS: Would this be a better fit in SO or SU SE?

Template of Current/ Example Folder Structure:

Base Folder (Source): 

C:\iMazing
   \Backups

    \{device A UDID} // Last backup of Device A (AL)
    \{device B UDID} // Last backup of Device B (BL)
    \iMazing.Versions 

       \Versions
          \{device A UDID} // Backup history of Device A (AL) 
          \{device B UDID} // Backup history of Device B (BH)
Base Folder (Destination): 

S:\ExtHDD\APPLE\iMazing.Backups  

    \{device A UDID} // Last backup of Device A (AL)
    \{device B UDID} // Last backup of Device B (BL)
    \iMazing.Versions 

       \Versions
          \{device A UDID} // Backup history of Device A (AH) 
          \{device B UDID} // Backup history of Device B (BH)


Folder Trees under Folders with UDID Names

  • AL : {device A UDID} // Last backup of Device A
  • AH : {device A UDID} // Backup history of Device A
  • AL & AH Form a SINGLE 2 SET with Hard Links in between.
  • We'd like to work INDIVIDUAL SET at a time

How to use Rsync (preferable via Cygwin RSync or one of mentioned Windows variations) to :

  • Copy a Hard Linked INDIVIDUAL SET e.g. {AL + AH} from C drive to S drive while
    • Preserving Hard Links
    • Maintain the Sub-tree folder structure
  • Without causing issues on Windows or its NTFS Folder & File structure

Also:

  • Can we do only an INDIVIDUAL SET at a time (one single RSync command) or multiple SPECIFIED SETs?
  • Typically we have 2 SETs, how would commandline for Rsync change if it was a 3 SET (SET with 3 HL folders) within a structure

A Real/ Actual Example with UDID Folder name:

  • XL & XH which are Hard Linked folder trees, together make up a SET.
Base Folder (Source): 
C:\Users\UserName\AppData\Roaming\iMazing\Backups

  \f22ebcffcdd1b508d3f7564431a9db98d69208ef-20190414-170122 //..(XL)  
  \iMazing.Versions
   \Versions  
    \f22ebcffcdd1b508d3f7564431a9db98d69208ef-20190414-170122 //..(XH)

Reference list of Tools for Hard Link preserving Copying:
https://superuser.com/questions/997190/copy-to-another-ntfs-disk-and-preserve-hard-links/

  • Rsync variations under Windows
    • (Several *Nix & Windows: Cygwin variations, CW RSync, Delta Copy/ Syncrify/ Synaman, GRsync, RsyncBackup, Rsync.exe pkg, Rclone, AcroSync, YInterSync)
  • ln.exe - command line hardlinks - http://schinagl.priv.at/nt/ln/ln.html (Win)

  • Not sure why this should be a problem. Have you tried anything yet? – roaima Sep 11 '19 at 20:07
  • Yes. Single folder trees, Yes & abstract Syntax with HL, Yes (superuser.com/a/1001102/183467). But not sure how to do HLinked sets of Folder Trees and maintain that structure. The linked syntax is vague and I cant find a good example to do whats asked & maintain structure. I'm not an Rsync expert and not too many such examples around – Alex S Sep 11 '19 at 20:27
  • And add to it NTFS/ Windows/ Cygwin issues and from my reading there's more arguments for that. Would just linking all my reading/ pages/ articles & SE QnAs help you realize I've been looking & trying – Alex S Sep 11 '19 at 20:29
1

I've found it rather difficult to understand your earlier iterations of this question, so I'm going to restate a simplified version of what I think you mean, along with a corresponding worked example:

Structure

C:\iMazing\Backups\
    AL\
        files...
    BL\
        files...
    iMazing.Versions\Versions\
        AH\
            files...
        BH\

S:\ExtHDD\APPLE\iMazing.Backups\
    AL\
        destination files...
    BL\
        destination files...
    iMazing.Versions\Versions\
        AH\
            destination files...
        BH\
            destination files...

Requirement is to copy AL\... and AH\... from C: to S: while maintaining hard links and relative structure.

Worked example

mkdir -p iMazing/Backups/{AL,BL,iMazing.Versions/Versions/{AH,BH}} ExtHDD/APPLE/iMazing.Backups
touch iMazing/Backups/{AL/albackup,BL/blbackup}
touch iMazing/Backups/iMazing.Versions/Versions/{AH/ahbackup,BH/bhbackup}

At this point you can ls -R or find to see what I've done. There should be files albackup, ahbackup, blbackup and bhbackup in the AL, AH, BL, and BH directories. We're going to copy the AL and AH directories to the target:

rsync --dry-run -avHPR iMazing/Backups/./AL iMazing/Backups/./iMazing.Versions/Versions/AH ExtHDD/APPLE/iMazing.Backups/

The -R (--relative) flag retains the source path(s) from the /./ marker. In the example this means that iMazing/Backups/ is stripped, and the remainder appended to the destination path.

Once the --dry-run (-n) is removed this will copy files and directories from the source AL and AH directories to the target, maintaining internal hard links and structure.

Question solution

Since this is Cygwin you need to use forward slashes and paths starting with /cygdrive/{DriveLetter}. To maintain links you need to include the -H flag (--hard-links).

rsync -aHR /cygdrive/C/iMazing/Backups/./AL /cygdrive/C/iMazing/Backups/./iMazing.Versions/Versions/AH /cygdrive/ExtHDD/APPLE/iMazing.Backups/

You can include the -v (--verbose) and/or -P (--partial --progress) flags if you want to see what's going on. In this situation rsync will not bother with its infamous "delta" algorithm or compression (these only work when there are two servers involved in an rsync client-server configuration across a network). If you could run an instance of rsync on the server hosting your S: drive that could be far more efficient.

Add --dry-run to see what would happen without it actually doing anything.

| improve this answer | |
  • You’re missing a key part I don’t want entire base folder, only the SETs under & in their structure. Wouldn’t have to post this question with so much detail otherwise – Alex S Sep 11 '19 at 21:28
  • @AlexS this command won't copy the base folder path. Please try it. – roaima Sep 11 '19 at 23:20
  • @roima - Have you first understood what is COMPOSED of a SET? Please see the BlockQuoted (in Yellow) part above of what is a SET {AL - AH}. How would you copy an INDIVIDUAL SET {AL - AH} without referring to it? We do not want to copy ALL the SETS at one time. But choose which SET {AL - AH} or {BL - BH} at a given time and copy it WITH ITS STRUCTURE INTACT. – Alex S Sep 12 '19 at 12:06
  • 1
    @AlexS try that. I think I now understand what you're asking. – roaima Sep 12 '19 at 20:28
  • 1
    So....did this work? – B Layer Oct 30 '19 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.