I have two drives with the same files, but the directory structure is totally different.

Is there any way to 'move' all the files on the destination side so that they match the structure of the source side? With a script perhaps?

For example, drive A has:


Whereas drive B has:


The files in question are huge (800GB), so I don't want to re-copy them; I just want to sync the structure by creating the necessary directories and moving the files.

I was thinking of a recursive script that would find each source file on the destination, then move it to a matching directory, creating it if necessary. But -- that's beyond my abilities!

Another elegant solution was given here: https://superuser.com/questions/237387/any-way-to-sync-directory-structure-when-the-files-are-already-on-both-sides/238086

  • Are you sure the name uniquely determines the content of a file, otherwise you should consider comparing files by their checksums.
    – kasterma
    Commented Jan 25, 2011 at 19:22

9 Answers 9


I'll go with Gilles and point you to Unison as suggested by hasen j. Unison was DropBox 20 years before DropBox. Rock solid code that a lot of people (myself included) use every day -- very worthwhile to learn. Still, join needs all the publicity it can get :)

This is only half an answer, but I have to get back to work :)

Basically, I wanted to demonstrate the little-known join utility which does just that: joins two tables on a some field.

First, set up a test case including file names with spaces:

for d in a b 'c c'; do mkdir -p "old/$d"; echo $RANDOM > "old/${d}/${d}.txt"; done
cp -r old new

(edit some directory and/or file names in new).

Now, we want to build a map: hash -> filename for each directory and then use join to match up files with the same hash. To generate the map, put the following in makemap.sh:

find "$1" -type f -exec md5 -r "{}" \; \
  | sed "s/\([a-z0-9]*\) ${1}\/\(.*\)/\1 \"\2\"/" \

makemap.sh spits out a file with lines of the form, 'hash "filename"', so we just join on the first column:

join <(./makemap.sh 'old') <(./makemap.sh 'new') >moves.txt

This generates moves.txt which looks like this:

49787681dd7fcc685372784915855431 "a/a.txt" "bar/a.txt"
bfdaa3e91029d31610739d552ede0c26 "c c/c c.txt" "c c/c c.txt"

The next step would be to actually do the moves, but my attempts got stuck on quoting... mv -i and mkdir -p should come handy.

  • Sorry, I don't understand any of this!
    – d0g
    Commented Jan 26, 2011 at 4:24
  • 1
    join is really interesting. Thanks for bringing it to my attention.
    – Steven D
    Commented Jan 26, 2011 at 17:48
  • @Dan. Sorry. The problem is that I don't know what assumptions I can make about your file names. Scripting without assumptions is no fun, especially in this case where I chose to output the file names to a file dwheeler.com/essays/fixing-unix-linux-filenames.html .
    – Janus
    Commented Jan 28, 2011 at 7:38
  • 1
    This probably wastes a lot of time (and CPU load) because these huge files have to be read completely for creating the MD5 hashes. If the file name and file size match then it's probably overkill to hash the files. Hashing should be done in a second step and just for the files which match at least one (on the same disk) in name or size. Commented Feb 27, 2013 at 1:51
  • Don't you need to sort the files you use as join input?
    – cjm
    Commented Oct 8, 2013 at 15:46

There's a utility called unison:


Description from site:

Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.

Note that Unison only detects moved files on the first run if at least one of the roots is remote, so even if you're synchronizing local files, use ssh://localhost/path/to/dir as one of the roots.

  • @Gilles: Are you sure? I use unison for everything and often see it spotting files that have been renamed and/or moved far away. Are you saying that this only works for already synced files where unison has had a chance to record inode numbers (or whatever other tricks it uses)?
    – Janus
    Commented Jan 28, 2011 at 7:13
  • @Janus: Thanks for the correction, my comment was indeed wrong. Unison does detect files that were moved, even on the initial run. (It doesn't do this when both roots are local, which is why it didn't do it in my test.) So unison is a very good suggestion. Commented Jan 28, 2011 at 18:22
  • @Gilles. Good to know -- there seems to be quite a few places where the algorithm distinguishes between local and remote syncs. I actually didn't think it would work for first sync. +1 for unison!
    – Janus
    Commented Jan 29, 2011 at 15:43

Use Unison as suggested by hasen j. I'm leaving this answer up as a potentially useful scripting example or for use on a server with only basic utilities installed.

I'll assume that the file names are unique throughout the hierarchy. I'll also assume that no file name contains a newline, and that the directory trees only contain directories and regular files.

  1. First collect the file names on the source side.

    (cd /A && find . \! -type d) >A.find
  2. Then move the files into place on the destination side. First, create a flattened tree of files on the destination side. Use ln instead of mv if you want to keep hard links around in the old hierarchy.

    mkdir /B.staging /B.new
    find /B.old -type f -exec sh -c 'mv -- "$@" "$0"' /B.staging {} +
  3. If some files may be missing in the destination, create a similarly flattened /A.staging and use rsync to copy the data from the source to the destination.

    rsync -au /A.staging/ /B.staging/
  4. Now rename the files into place.

    cd /B.new &&
    <A.find perl -l -ne '
      my $dir = '.'; s!^\./+!!;
      while (s!^([^/]+)/+!!) {  # Create directories as needed
        $dir .= "/$1";
        -d $dir or mkdir $dir or die "mkdir $dir: $!"
      rename "/B.staging/$_", "$dir/$_" or die "rename -> $dir/$_: $!"


    cd /B.new &&
    <A.find python -c '
    import os, sys
    for path in sys.stdin.read().splitlines():
        dir, base = path.rsplit("/", 2)
        os.rename(os.path.join("/B.new", base), path)
  5. Finally, if you care about the metadata of the directories, call rsync with the files already in place.

    rsync -au /A/ /B.new/

Note that I haven't tested the snippets in this post. Use at your own risk. Please report any error in a comment.


Note that the tool Unison (suggested in other answers) appears to handle file renames, but what it actually does is detecting that the file already exists in the destination, and making a copy of it in the destination from the destination. This saves bandwidth, but is still slow and will stress hard drives for no reason.

A tool made by one single person is addressing this issue perfectly (although it only supports local directories for the moment): rsync-sidekick.

(Of course, read the code yourself before executing anything written by a stranger.)

Also, to ensure that the tool won't mistakenly delete files it isn't supposed to, we can run it in a Docker container with readonly volumes. It will output a list of commands to rename and move stuff in the destination to reproduce the renames/moves made in the source.

  1. Get the repo from the link above.

  2. Build the container:

    docker build -t rsync-sidekick .
  3. Run it:

    docker run --rm \
     -v /<source-path>:/sync-src:ro \
     -v /<destination-path>:/sync-dst:ro \
     rsync-sidekick \
     /bin/bash -c "rsync-sidekick -shellscript /sync-src/ /sync-dst/ && echo && cat sync_actions_*.sh"

Particularly if on-going sync would be useful, you could try to figure out git-annex.

It's relatively new; I haven't tried to use it myself.

I'm able to suggest it because it avoids keeping a second copy of the files... this means it has to mark the files as read-only ("locked"), like certain non-Git version control systems.

Files are identified by sha256sum + file extension (by default). So it should be able to sync two repos with identical file content but different filenames, without having to perform writes (and over a low-bandwidth network, if desired). It will of course have to read all the files in order to checksum them.


How about something like this:


cd $src
find . -name <PATTERN> -type f >/tmp/srclist
cd $dst
find . -name <PATTERN> -type f >/tmp/dstlist

cat /tmp/srclist | while read srcpath; do
    name=`basename "$srcpath"`
    srcdir=`dirname "$srcpath"`
    dstpath=`grep "/${name}\$" /tmp/dstlist`

    mkdir -p "$srcdir"
    cd "$srcdir" && ln -s "$dstpath" "$name"

This assumes that names of the files you want to sync are unique across the whole drive: otherwise there's no way it can be fully automated (however, you can provide a prompt for user to choose which file to pick if there's more that one.)

The script above will work in simple cases, but may fail if name happens to contain symbols which have special meaning for regexps. The grep on list of files can also take a lot of time if there's lot of files. You may consider translating this code to use hashtable which will map filenames to paths, e.g. in Ruby.

  • This looks promising -- but does it move the files, or just create symlinks?
    – d0g
    Commented Jan 26, 2011 at 4:21
  • I think I understand most of this; but what does the grep line do? Does it just find the full path of the matching file in dstlist?
    – d0g
    Commented Jan 26, 2011 at 4:29
  • @Dan: apparently by the use of ln it creates symlinks. You may employ mv to move the files, but beware of overwriting existing ones. Also, you might want to cleanup empty dirs if any, after moving the files away. Yes, that grep command searches for a line which ends on the filename, thus revealing the full path to it in on destination drive.
    – alex
    Commented Jan 26, 2011 at 7:13

Assuming the base filenames are unique in the trees, it's fairly straightforward:

join <(cd A; find . -type f | while read f; do echo $(basename $f) $(dirname $f); done | sort) \
     <(cd B; find . -type f | while read f; do echo $(basename $f) $(dirname $f); done | sort) |\
while read name to from
        mkdir -p B/$to
        mv -v B/$from/$name B/$to/

If you wanted to clean up the old empty directories, use:

find B -depth -type d -delete

I also faced this problem. The md5sum -based solution did not work for me, because I sync my files to a webdav mount. Computing md5sum sums on the webdav destination would also mean large file operations.

I made a small script reorg_Remote_Dir_detect_moves.sh (on github) which is trying to detect the most moved files and then creates a new temporary shell-script with several commands to adjust the remote directory. Since I only take care of the file names, the script is no perfect solution.

For safety, several files will be ignored: A) Files with same (same beginning) names on every side, and B) Files which are only on the remote side. They will be ignored and skipped.

Skipped files will then be handled by your preferred sync tool (e.g. rsync, unison, ...), which you have to use after running the temporary shell-script.

So maybe my script is useful for someone? If so (to make it more clear) there are three steps:

  1. Run the shell script reorg_Remote_Dir_detect_moves.sh (on github)
  2. This will create the temporary shell-script /dev/shm/REORGRemoteMoveScript.sh => run this to do the moves (will be fast on mounted webdav)
  3. Run your preferred sync tool (e.g. rsync, unison, ...)

Here's my attempt at an answer. As a forewarning, all my scripting experience comes from bash, so if you are using a different shell, the command names or syntax may be different.

This solution requires creating two seperate scripts.

This first script is responsible for actually moving the files on the destination drive.


# Given a single line from the md5 map file, list
# only the path from that line.
  echo $2

# Given an md5, list the filename from the md5 map file
  # Grab the line from the md5 map file that has the
  # md5 sum passed in and call get_file() with that line.
  get_file `cat $md5_map_file | grep $1`


# Compute the md5
sum=`md5sum $file`

# Get the new path for the file
new_file=`get_file_from_md5 $sum`

# Make sure the destination directory exists
mkdir -p `dirname $new_file`
# Move the file, prompting if the move would cause an overwrite
mv -i $file $new_file

The second script creates the md5 map file used by the first script and then calls the first script on every file in the destination drive.

# Do not put trailing /

# This command searches through the source drive
# looking for files.  For every file it finds,
# it computes the md5sum and writes the md5 sum and
# the path to the found filename to the filename stored
# in $md5_map_file.
# The end result is a file listing the md5 of every file
# on the source drive
cd $src
find . -type f -exec md5sum "{}" \; > $md5_map_file

# This command searches the destination drive for files and calls the first
# script for every file it finds.
cd $dst
find . -type f -exec $script_path '{}' \; 

Basically, what is going on is the two scripts similuate an associative array with $md5_map_file. First, all the md5s for the files on the source drive are computed and stored. Associated with the md5s are the relative paths from the drive's root. Then, for each file on the destination drive, the md5 is computed. Using this md5, the path of that file on the source drive is looked up. The file on the destination drive is then moved to match the path of the file on the source drive.

There are a couple of caveats with this script:

  • It assumes that every file in $dst is also in $src
  • It does not remove any directories from $dst, only moves the files. I am currently unable to think of a safe way to do this automatically
  • It must take a long while to compute the md5's: all the content must actually be read. While if Dan is sure the files are identical, simply moving them in the directory structure is very fast (no reading). So, md5sum seems not to be the thing to use here. (BTW, rsync has a mode in which it doesn't calculate checksums.) Commented Jan 29, 2011 at 23:48
  • It's a tradeoff between accuracy and speed. I wanted to provide a method that used a higher degree of accuracy than simply filenames.
    – cledoux
    Commented Jan 30, 2011 at 5:37
  • Maybe find files that are the same size first, and then only bother with md5's of those. Also, I gather from another thread that there are other hash functions that are faster.
    – Diagon
    Commented Nov 30, 2020 at 0:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .