Any way to sync directory structure when the files are already on both sides?

Question

I have two drives with the same files, but the directory structure is totally different.

Is there any way to 'move' all the files on the destination side so that they match the structure of the source side? With a script perhaps?

For example, drive A has:

/foo/bar/123.txt
/foo/bar/234.txt
/foo/bar/dir/567.txt

Whereas drive B has:

/some/other/path/123.txt
/bar/doo2/wow/234.txt
/bar/doo/567.txt

The files in question are huge (800GB), so I don't want to re-copy them; I just want to sync the structure by creating the necessary directories and moving the files.

I was thinking of a recursive script that would find each source file on the destination, then move it to a matching directory, creating it if necessary. But -- that's beyond my abilities!

Another elegant solution was given here: https://superuser.com/questions/237387/any-way-to-sync-directory-structure-when-the-files-are-already-on-both-sides/238086

Answer 1

I'll go with Gilles and point you to Unison as suggested by hasen j. Unison was DropBox 20 years before DropBox. Rock solid code that a lot of people (myself included) use every day -- very worthwhile to learn. Still, join needs all the publicity it can get :)

---

This is only half an answer, but I have to get back to work :)

Basically, I wanted to demonstrate the little-known join utility which does just that: joins two tables on a some field.

First, set up a test case including file names with spaces:

for d in a b 'c c'; do mkdir -p "old/$d"; echo $RANDOM > "old/${d}/${d}.txt"; done
cp -r old new

(edit some directory and/or file names in new).

Now, we want to build a map: hash -> filename for each directory and then use join to match up files with the same hash. To generate the map, put the following in makemap.sh:

find "$1" -type f -exec md5 -r "{}" \; \
  | sed "s/\([a-z0-9]*\) ${1}\/\(.*\)/\1 \"\2\"/" \

makemap.sh spits out a file with lines of the form, 'hash "filename"', so we just join on the first column:

join <(./makemap.sh 'old') <(./makemap.sh 'new') >moves.txt

This generates moves.txt which looks like this:

49787681dd7fcc685372784915855431 "a/a.txt" "bar/a.txt"
bfdaa3e91029d31610739d552ede0c26 "c c/c c.txt" "c c/c c.txt"

The next step would be to actually do the moves, but my attempts got stuck on quoting... mv -i and mkdir -p should come handy.

Answer 2

There's a utility called unison:

http://www.cis.upenn.edu/~bcpierce/unison/

Description from site:

Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.

Note that Unison only detects moved files on the first run if at least one of the roots is remote, so even if you're synchronizing local files, use ssh://localhost/path/to/dir as one of the roots.

Answer 3

Use Unison as suggested by hasen j. I'm leaving this answer up as a potentially useful scripting example or for use on a server with only basic utilities installed.

---

I'll assume that the file names are unique throughout the hierarchy. I'll also assume that no file name contains a newline, and that the directory trees only contain directories and regular files.

1. First collect the file names on the source side.

   (cd /A && find . \! -type d) >A.find
   

2. Then move the files into place on the destination side. First, create a flattened tree of files on the destination side. Use ln instead of mv if you want to keep hard links around in the old hierarchy.

   mkdir /B.staging /B.new
   find /B.old -type f -exec sh -c 'mv -- "$@" "$0"' /B.staging {} +
   

3. If some files may be missing in the destination, create a similarly flattened /A.staging and use rsync to copy the data from the source to the destination.

   rsync -au /A.staging/ /B.staging/
   

4. Now rename the files into place.

   cd /B.new &&
   <A.find perl -l -ne '
     my $dir = '.'; s!^\./+!!;
     while (s!^([^/]+)/+!!) {  # Create directories as needed
       $dir .= "/$1";
       -d $dir or mkdir $dir or die "mkdir $dir: $!"
     }
     rename "/B.staging/$_", "$dir/$_" or die "rename -> $dir/$_: $!"
   '
   

   Equivalently:

   cd /B.new &&
   <A.find python -c '
   import os, sys
   for path in sys.stdin.read().splitlines():
       dir, base = path.rsplit("/", 2)
       os.rename(os.path.join("/B.new", base), path)
   '
   

5. Finally, if you care about the metadata of the directories, call rsync with the files already in place.

   rsync -au /A/ /B.new/
   

Note that I haven't tested the snippets in this post. Use at your own risk. Please report any error in a comment.

Answer 4

Note that the tool Unison (suggested in other answers) appears to handle file renames, but what it actually does is detecting that the file already exists in the destination, and making a copy of it in the destination from the destination. This saves bandwidth, but is still slow and will stress hard drives for no reason.

A tool made by one single person is addressing this issue perfectly (although it only supports local directories for the moment): rsync-sidekick.

(Of course, read the code yourself before executing anything written by a stranger.)

Also, to ensure that the tool won't mistakenly delete files it isn't supposed to, we can run it in a Docker container with readonly volumes. It will output a list of commands to rename and move stuff in the destination to reproduce the renames/moves made in the source.

1. Get the repo from the link above.

2. Build the container:

   docker build -t rsync-sidekick .
   

3. Run it:

   docker run --rm \
    -v /<source-path>:/sync-src:ro \
    -v /<destination-path>:/sync-dst:ro \
    rsync-sidekick \
    /bin/bash -c "rsync-sidekick -shellscript /sync-src/ /sync-dst/ && echo && cat sync_actions_*.sh"
   

Answer 5

Particularly if on-going sync would be useful, you could try to figure out git-annex.

It's relatively new; I haven't tried to use it myself.

I'm able to suggest it because it avoids keeping a second copy of the files... this means it has to mark the files as read-only ("locked"), like certain non-Git version control systems.

Files are identified by sha256sum + file extension (by default). So it should be able to sync two repos with identical file content but different filenames, without having to perform writes (and over a low-bandwidth network, if desired). It will of course have to read all the files in order to checksum them.

Answer 6

How about something like this:

src=/mnt/driveA
dst=/mnt/driveB

cd $src
find . -name <PATTERN> -type f >/tmp/srclist
cd $dst
find . -name <PATTERN> -type f >/tmp/dstlist

cat /tmp/srclist | while read srcpath; do
    name=`basename "$srcpath"`
    srcdir=`dirname "$srcpath"`
    dstpath=`grep "/${name}\$" /tmp/dstlist`

    mkdir -p "$srcdir"
    cd "$srcdir" && ln -s "$dstpath" "$name"
done

This assumes that names of the files you want to sync are unique across the whole drive: otherwise there's no way it can be fully automated (however, you can provide a prompt for user to choose which file to pick if there's more that one.)

The script above will work in simple cases, but may fail if name happens to contain symbols which have special meaning for regexps. The grep on list of files can also take a lot of time if there's lot of files. You may consider translating this code to use hashtable which will map filenames to paths, e.g. in Ruby.

Answer 7

Assuming the base filenames are unique in the trees, it's fairly straightforward:

join <(cd A; find . -type f | while read f; do echo $(basename $f) $(dirname $f); done | sort) \
     <(cd B; find . -type f | while read f; do echo $(basename $f) $(dirname $f); done | sort) |\
while read name to from
do
        mkdir -p B/$to
        mv -v B/$from/$name B/$to/
done

If you wanted to clean up the old empty directories, use:

find B -depth -type d -delete

Answer 8

I also faced this problem. The md5sum -based solution did not work for me, because I sync my files to a webdav mount. Computing md5sum sums on the webdav destination would also mean large file operations.

I made a small script reorg_Remote_Dir_detect_moves.sh (on github) which is trying to detect the most moved files and then creates a new temporary shell-script with several commands to adjust the remote directory. Since I only take care of the file names, the script is no perfect solution.

For safety, several files will be ignored: A) Files with same (same beginning) names on every side, and B) Files which are only on the remote side. They will be ignored and skipped.

Skipped files will then be handled by your preferred sync tool (e.g. rsync, unison, ...), which you have to use after running the temporary shell-script.

So maybe my script is useful for someone? If so (to make it more clear) there are three steps:

1. Run the shell script reorg_Remote_Dir_detect_moves.sh (on github)
2. This will create the temporary shell-script /dev/shm/REORGRemoteMoveScript.sh => run this to do the moves (will be fast on mounted webdav)
3. Run your preferred sync tool (e.g. rsync, unison, ...)

Answer 9

Here's my attempt at an answer. As a forewarning, all my scripting experience comes from bash, so if you are using a different shell, the command names or syntax may be different.

This solution requires creating two seperate scripts.

This first script is responsible for actually moving the files on the destination drive.

md5_map_file="<absolute-path-to-a-temporary-file>"

# Given a single line from the md5 map file, list
# only the path from that line.
get_file()
{
  echo $2
}

# Given an md5, list the filename from the md5 map file
get_file_from_md5()
{
  # Grab the line from the md5 map file that has the
  # md5 sum passed in and call get_file() with that line.
  get_file `cat $md5_map_file | grep $1`
}

file=$1

# Compute the md5
sum=`md5sum $file`

# Get the new path for the file
new_file=`get_file_from_md5 $sum`

# Make sure the destination directory exists
mkdir -p `dirname $new_file`
# Move the file, prompting if the move would cause an overwrite
mv -i $file $new_file

The second script creates the md5 map file used by the first script and then calls the first script on every file in the destination drive.

# Do not put trailing /
src="<absolute-path-to-source-drive>"
dst="<absolute-path-to-destination-drive>"
script_path="<absolute-path-to-the-first-script>"
md5_map_file="<same-absolute-path-from-first-script>"


# This command searches through the source drive
# looking for files.  For every file it finds,
# it computes the md5sum and writes the md5 sum and
# the path to the found filename to the filename stored
# in $md5_map_file.
# The end result is a file listing the md5 of every file
# on the source drive
cd $src
find . -type f -exec md5sum "{}" \; > $md5_map_file

# This command searches the destination drive for files and calls the first
# script for every file it finds.
cd $dst
find . -type f -exec $script_path '{}' \; 

Basically, what is going on is the two scripts similuate an associative array with $md5_map_file. First, all the md5s for the files on the source drive are computed and stored. Associated with the md5s are the relative paths from the drive's root. Then, for each file on the destination drive, the md5 is computed. Using this md5, the path of that file on the source drive is looked up. The file on the destination drive is then moved to match the path of the file on the source drive.

There are a couple of caveats with this script:

• It assumes that every file in $dst is also in $src
• It does not remove any directories from $dst, only moves the files. I am currently unable to think of a safe way to do this automatically