Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have two drives with the same files, but the directory structure is totally different.

Is there any way to 'move' all the files on the destination side so that they match the structure of the source side? With a script perhaps?

For example, drive A has:

/foo/bar/123.txt
/foo/bar/234.txt
/foo/bar/dir/567.txt

Whereas drive B has:

/some/other/path/123.txt
/bar/doo2/wow/234.txt
/bar/doo/567.txt

The files in question are huge (800GB), so I don't want to re-copy them; I just want to sync the structure by creating the necessary directories and moving the files.

I was thinking of a recursive script that would find each source file on the destination, then move it to a matching directory, creating it if necessary. But that's beyond my abilities ...

Any help greatly appreciated!

Thanks

UPDATE: Another elegant solution was given here: http://superuser.com/questions/237387/any-way-to-sync-directory-structure-when-the-files-are-already-on-both-sides/238086

share|improve this question
    
Are you sure the name uniquely determines the content of a file, otherwise you should consider comparing files by their checksums. –  kasterma Jan 25 '11 at 19:22

6 Answers 6

Particularly if on-going sync would be useful, you could try to figure out git-annex.

It's relatively new; I haven't tried to use it myself.

I'm able to suggest it because it avoids keeping a second copy of the files... this means it has to mark the files as read-only ("locked"), like certain non-Git version control systems.

Files are identified by sha256sum + file extension (by default). So it should be able to sync two repos with identical file content but different filenames, without having to perform writes (and over a low-bandwidth network, if desired). It will of course have to read all the files in order to checksum them.

share|improve this answer

There's a utility called unison:

http://www.cis.upenn.edu/~bcpierce/unison/

Description from site:

Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.

Note that Unison only detects moved files on the first run if at least one of the roots is remote, so even if you're synchronizing local files, use ssh://localhost/path/to/dir as one of the roots.

share|improve this answer
    
@Gilles: Are you sure? I use unison for everything and often see it spotting files that have been renamed and/or moved far away. Are you saying that this only works for already synced files where unison has had a chance to record inode numbers (or whatever other tricks it uses)? –  Janus Jan 28 '11 at 7:13
    
@Janus: Thanks for the correction, my comment was indeed wrong. Unison does detect files that were moved, even on the initial run. (It doesn't do this when both roots are local, which is why it didn't do it in my test.) So unison is a very good suggestion. –  Gilles Jan 28 '11 at 18:22
    
@Gilles. Good to know -- there seems to be quite a few places where the algorithm distinguishes between local and remote syncs. I actually didn't think it would work for first sync. +1 for unison! –  Janus Jan 29 '11 at 15:43

I'll go with Gilles and point you to Unison as suggested by hasen j. Unison was DropBox 20 years before DropBox. Rock solid code that a lot of people (myself included) use every day -- very worthwhile to learn. Still, join needs all the publicity it can get :)


This is only half an answer, but I have to get back to work :)

Basically, I wanted to demonstrate the little-known join utility which does just that: joins two tables on a some field.

First, set up a test case including file names with spaces:

for d in a b 'c c'; do mkdir -p "old/$d"; echo $RANDOM > "old/${d}/${d}.txt"; done
cp -r old new

(edit some directory and/or file names in new).

Now, we want to build a map: hash -> filename for each directory and then use join to match up files with the same hash. To generate the map, put the following in makemap.sh:

find "$1" -type f -exec md5 -r "{}" \; \
  | sed "s/\([a-z0-9]*\) ${1}\/\(.*\)/\1 \"\2\"/" \

makemap.sh spits out a file with lines of the form, 'hash "filename"', so we just join on the first column:

join <(./makemap.sh 'old') <(./makemap.sh 'new') >moves.txt

This generates moves.txt which looks like this:

49787681dd7fcc685372784915855431 "a/a.txt" "bar/a.txt"
bfdaa3e91029d31610739d552ede0c26 "c c/c c.txt" "c c/c c.txt"

The next step would be to actually do the moves, but my attempts got stuck on quoting... mv -i and mkdir -p should come handy.

share|improve this answer
    
Sorry, I don't understand any of this! –  Ze'ev Jan 26 '11 at 4:24
1  
join is really interesting. Thanks for bringing it to my attention. –  Steven D Jan 26 '11 at 17:48
    
@Dan. Sorry. The problem is that I don't know what assumptions I can make about your file names. Scripting without assumptions is no fun, especially in this case where I chose to output the file names to a file dwheeler.com/essays/fixing-unix-linux-filenames.html . –  Janus Jan 28 '11 at 7:38
    
This probably wastes a lot of time (and CPU load) because these huge files have to be read completely for creating the MD5 hashes. If the file name and file size match then it's probably overkill to hash the files. Hashing should be done in a second step and just for the files which match at least one (on the same disk) in name or size. –  Hauke Laging Feb 27 '13 at 1:51
    
Don't you need to sort the files you use as join input? –  cjm Oct 8 '13 at 15:46

Use Unison as suggested by hasen j. I'm leaving this answer up as a potentially useful scripting example or for use on a server with only basic utilities installed.


I'll assume that the file names are unique throughout the hierarchy. I'll also assume that no file name contains a newline, and that the directory trees only contain directories and regular files.

  1. First collect the file names on the source side.

    (cd /A && find . \! -type d) >A.find
    
  2. Then move the files into place on the destination side. First, create a flattened tree of files on the destination side. Use ln instead of mv if you want to keep hard links around in the old hierarchy.

    mkdir /B.staging /B.new
    find /B.old -type f -exec sh -c 'mv -- "$@" "$0"' /B.staging {} +
    
  3. If some files may be missing in the destination, create a similarly flattened /A.staging and use rsync to copy the data from the source to the destination.

    rsync -au /A.staging/ /B.staging/
    
  4. Now rename the files into place.

    cd /B.new &&
    <A.find perl -l -ne '
      my $dir = '.'; s!^\./+!!;
      while (s!^([^/]+)/+!!) {  # Create directories as needed
        $dir .= "/$1";
        -d $dir or mkdir $dir or die "mkdir $dir: $!"
      }
      rename "/B.staging/$_", "$dir/$_" or die "rename -> $dir/$_: $!"
    '
    

    Equivalently:

    cd /B.new &&
    <A.find python -c '
    import os, sys
    for path in sys.stdin.read().splitlines():
        dir, base = path.rsplit("/", 2)
        os.rename(os.path.join("/B.new", base), path)
    '
    
  5. Finally, if you care about the metadata of the directories, call rsync with the files already in place.

    rsync -au /A/ /B.new/
    

Note that I haven't tested the snippets in this post. Use at your own risk. Please report any error in a comment.

share|improve this answer

Here's my attempt at an answer. As a forewarning, all my scripting experience comes from bash, so if you are using a different shell, the command names or syntax may be different.

This solution requires creating two seperate scripts.

This first script is responsible for actually moving the files on the destination drive.

md5_map_file="<absolute-path-to-a-temporary-file>"

# Given a single line from the md5 map file, list
# only the path from that line.
get_file()
{
  echo $2
}

# Given an md5, list the filename from the md5 map file
get_file_from_md5()
{
  # Grab the line from the md5 map file that has the
  # md5 sum passed in and call get_file() with that line.
  get_file `cat $md5_map_file | grep $1`
}

file=$1

# Compute the md5
sum=`md5sum $file`

# Get the new path for the file
new_file=`get_file_from_md5 $sum`

# Make sure the destination directory exists
mkdir -p `dirname $new_file`
# Move the file, prompting if the move would cause an overwrite
mv -i $file $new_file

The second script creates the md5 map file used by the first script and then calls the first script on every file in the destination drive.

# Do not put trailing /
src="<absolute-path-to-source-drive>"
dst="<absolute-path-to-destination-drive>"
script_path="<absolute-path-to-the-first-script>"
md5_map_file="<same-absolute-path-from-first-script>"


# This command searches through the source drive
# looking for files.  For every file it finds,
# it computes the md5sum and writes the md5 sum and
# the path to the found filename to the filename stored
# in $md5_map_file.
# The end result is a file listing the md5 of every file
# on the source drive
cd $src
find . -type f -exec md5sum "{}" \; > $md5_map_file

# This command searches the destination drive for files and calls the first
# script for every file it finds.
cd $dst
find . -type f -exec $script_path '{}' \; 

Basically, what is going on is the two scripts similuate an associative array with $md5_map_file. First, all the md5s for the files on the source drive are computed and stored. Associated with the md5s are the relative paths from the drive's root. Then, for each file on the destination drive, the md5 is computed. Using this md5, the path of that file on the source drive is looked up. The file on the destination drive is then moved to match the path of the file on the source drive.

There are a couple of caveats with this script:

  • It assumes that every file in $dst is also in $src
  • It does not remove any directories from $dst, only moves the files. I am currently unable to think of a safe way to do this automatically

Good luck and I hope this helped.

share|improve this answer
    
It must take a long while to compute the md5's: all the content must actually be read. While if Dan is sure the files are identical, simply moving them in the directory structure is very fast (no reading). So, md5sum seems not to be the thing to use here. (BTW, rsync has a mode in which it doesn't calculate checksums.) –  imz -- Ivan Zakharyaschev Jan 29 '11 at 23:48
    
It's a tradeoff between accuracy and speed. I wanted to provide a method that used a higher degree of accuracy than simply filenames. –  karategeek6 Jan 30 '11 at 5:37

How about something like this:

src=/mnt/driveA
dst=/mnt/driveB

cd $src
find . -name <PATTERN> -type f >/tmp/srclist
cd $dst
find . -name <PATTERN> -type f >/tmp/dstlist

cat /tmp/srclist | while read srcpath; do
    name=`basename "$srcpath"`
    srcdir=`dirname "$srcpath"`
    dstpath=`grep "/${name}\$" /tmp/dstlist`

    mkdir -p "$srcdir"
    cd "$srcdir" && ln -s "$dstpath" "$name"
done

This assumes that names of the files you want to sync are unique across the whole drive: otherwise there's no way it can be fully automated (however, you can provide a prompt for user to choose which file to pick if there's more that one.)

The script above will work in simple cases, but may fail if name happens to contain symbols which have special meaning for regexps. The grep on list of files can also take a lot of time if there's lot of files. You may consider translating this code to use hashtable which will map filenames to paths, e.g. in Ruby.

share|improve this answer
    
This looks promising -- but does it move the files, or just create symlinks? –  Ze'ev Jan 26 '11 at 4:21
    
I think I understand most of this; but what does the grep line do? Does it just find the full path of the matching file in dstlist? –  Ze'ev Jan 26 '11 at 4:29
    
@Dan: apparently by the use of ln it creates symlinks. You may employ mv to move the files, but beware of overwriting existing ones. Also, you might want to cleanup empty dirs if any, after moving the files away. Yes, that grep command searches for a line which ends on the filename, thus revealing the full path to it in on destination drive. –  alex Jan 26 '11 at 7:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.