Bash command to "search and replace" file in changing directory tree

Question

I've built a collection of files for research in a central repository on my server. I'm continually curating the repo changing the directory's tree structure by renaming and moving directories. And I'm continually updating and editing local copies of files on the repo. This creates the situation where I have a file in the repo which needs to be replaced by a newer file on the client, but the directory tree has since changed.

Is there a rsync option I haven't learned yet? I don't find rsync working with changing directory tree. I tested rsync --existing

Or, is there some commandline-fu using find and mv? Searching the net and other resources finds only search and replace in file examples.

-- UPDATE 1 If I could I'd give points to @meuh and @Lqueryvg for fantastic answers. I see @css1971 added to his answer and made a stronger case for subversion file management. Unfortunately I ran out of time and the award system simply choose it's favorite answer for me. I found all the response incredibly informative. I want to thank you all. I look forward to some time next next weekend to give them all more thoughtful review.

Answer 1

[expanded from my comment to the OP]. Create on the repo a directory flat which holds a copy of all the files, but in one flat list. The copies would be hard links, so take no space. Do the same on the local machine. You can then rsync from the local flat directory to the remote flat directory. This will update all the remote files as rsync preserves remote hard links if you use --inplace. The flat dirs could be created by script just before the rsync and removed after.

Here's a runnable proof of concept test script:

#!/bin/bash
dosync(){  # function to create flat trees and rsync them
    mkdir flat
    mkdir flatl
    find repo  -type f -exec ln {} flat \;
    find local -type f -exec ln {} flatl \;
    rsync -aHv --inplace flatl/ flat
    rm -fr flat flatl
}

# create repo and local dirs with same content. 3 empty files
cd /tmp || exit
mkdir repo
( cd repo; touch a b c )
mkdir local
rsync -a repo/ local

dosync
echo hi >local/a   # example change of local file a
dosync
mkdir repo/new     # example move of repo file b
mv repo/b repo/new/
echo hello >local/b  # change local file b
dosync

ls -lR repo local
# rm -fr flat flatl repo local

---

For the reverse direction, after dosync has transferred your local modifs to the repo, you can just rm -fr local and "rsync -a repo/ local" to copy the complete repository to the local filesystem. Instead, you can reduce the number of files needing to be transferred to only the new ones in the repo by using a similar technique:

reversesync(){
    mkdir flat
    mkdir flatl
    find repo  -type f -exec ln {} flat \;
    find local -type f -exec ln {} flatl \;
    mv flat repo/flat
    mv flatl local/flat # not flatl!
    rsync -aHv --delete repo/ local
    rm -fr repo/flat local/flat
}

This moves the flat trees into the repo and local dirs respectively, so that rsync can see the hard linked files and avoid copying them. (Obviously, the flat dirs must have the same name this time).

---

If you have only a single known file that you have changed, you can use find on the repository to get its new position in the tree and rsync the one file to there. Eg:

file=mychangedfile.abc
to=$(find repo -name "$file")
from=$(find local -name "$file")
rsync -av "$from" "$to"

This assumes repo is mounted, otherwise you can use ssh repo find.... If you cannot ssh to the repository, instead you can use rsync to a dummy local destination to get the list of files, and extract the one you want:

to=$(rsync -a --list-only repo dummy | awk '/\/'"$file"'$/{print $NF}')

Answer 2

I think rsync is the wrong tool, as are find and mv. My recommendation is instead to make use of a software configuration management system. These include Subversion, Git, Mercurial, Bazaar among others. All of which can easily handle changes of tree structure.

In the structure you describe, you have tree structure A on your client system and a repository tree structure B on a secondary location, possibly local, possibly remote.

Where you have updates to both, you now have competing changes which have to be applied in the correct order for your two repositories to remain consistent. If they are applied out of order you get into the situation you are in now, where changes in one cannot be applied directly to the other because the structures no longer exist.

As it stands, there are no options for rsync which can allow it to automatically know which changes have to be applied to bring both repositories to consistency. That isn't what it's designed to do. It can certainly make one repository look exactly like the other but it requires that only one side is changed at a time. Example being alternating changes to A, then B, then A. At any point, you need to designate one of the structures A or B as the master and sync changes in only one direction at a time.

I also don't believe there's a simple commandline-fu command which will achieve the result you're looking for, so now you're into the realm of shell programming.

Where you have changes only to the tree structure of B and only to the file contents of A, it's a relatively simple task to look for the file names of the changed files and acquire the new paths of those files in B, then modify the tree structure of A to match. This only works if the file names are unique.

The pseudo code which would bring the structure of A into consistency with B would look something like:

generate list of file names in A and their paths

For each of the names in A
    find that same name in B
    If the path of A is the same as B 
        continue to the next file
    if not then
        create the directory structure in A
        move the file to the new location.
    if the old path in A is now empty
        delete the directory.
        repeat 
            check if the parent directory is now empty, then delete it.
        until a non empty directory

Once the tree structures are in sync, then A can be copied directly to exactly the same paths in B. The --update option of rsync can be used to overwrite older files with newer ones in both directions.

Some example shell code to copy locally changed files into the existing repository, using find as the file name selector.

#!/bin/bash

set -xv

localRepo=/tmp/a
remoteRepo=/tmp/b

rm -rf $localRepo $remoteRepo

mkdir -p $localRepo/1/2/ $localRepo/1/3/
mkdir -p $remoteRepo/2/1/ $remoteRepo/3/1/

echo a12 > $localRepo/1/2/file
echo b21 > $remoteRepo/2/1/file

echo a13 > $localRepo/1/3/file1
echo b31 > $remoteRepo/3/1/file1

echo ex1
cat $localRepo/1/2/file $remoteRepo/2/1/file
echo ex2
cat $localRepo/1/3/file1 $remoteRepo/3/1/file1



localFileNameList=$(find $localRepo -type f -mtime -1 | xargs -L 1 basename)


for localFileName in $localFileNameList
do
    localFilePath=$(find $localRepo -name $localFileName | xargs dirname)
    backFile=$(find $remoteRepo -name $localFileName)
    repoDir=$(dirname $backFile)

    cp $localFilePath/$localFileName $repoDir

done

echo ex1
cat $localRepo/1/2/file $remoteRepo/2/1/file
echo ex2
cat $localRepo/1/3/file1 $remoteRepo/3/1/file1

To import a file system into subversion for example as one of the simpler to use SCMs:

e.g.
mkdir /tmp/svn
svnadmin create /tmp/svn/reponame

cd /tmp/b
svn import -m "The initial import " file:///tmp/svn/reponame
Adding         2
Adding         2/1
Adding         2/1/file
Adding         3
Adding         3/1
Adding         3/1/file1

Then check the repo out and make local changes.

$ cd /tmp
$ svn checkout file:///tmp/svn/reponame 
A    reponame/2
A    reponame/2/1
A    reponame/2/1/file
A    reponame/3
A    reponame/3/1
A    reponame/3/1/file1
Checked out revision 1.
/tmp:

$ cd reponame/
/tmp/reponame:
$ ls -ltr
total 8
drwxrwxr-x 3 css1971 css1971 4096 Apr 11 12:04 3
drwxrwxr-x 3 css1971 css1971 4096 Apr 11 12:04 2
/tmp/reponame:
$ svn move 3 4
A         4
D         3
D         3/1
D         3/1/file1
/tmp/reponame:

Commit the changes back to the repo.

$ svn commit -m "renamed dir"
Deleting       3
Adding         4

Committed revision 2.

From this point, use the svn tool as part of your normal workflow to manipulate the repository.

Useful commands:

svn import
svn update
svn commit
svn del
svn cp
svn mv

Command reference: http://svnbook.red-bean.com/en/1.7/svn.ref.html

Answer 3

I think that's a job for Unison ! I haven't played with it for years, but I think it could do exactly what you're asking for... The home page says:

Unison is a file-synchronization tool for OSX, Unix, and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.

Have a look and keep us posted ! ;)

Answer 4

You definitely can use rsync with a changing directory structure and there are some interesting options you might not be aware of, in particular -H which preserves hard links.

I'll describe my own scenario. This may or may not work for you, but I hope you at least find it interesting.

SCENARIO:

You have a large directory with lots of files in a directory tree which are copied via rsync to another directory (possibly on a remote machine or an external disk).

You want to restructure your directory, perhaps renaming files and/or moving them to different sub-directories, but without changing the contents of the files themselves.

But, the next time you run rsync it re-copies the data for all of the files which have been moved or whose names have changed, even though they already exist (albeit in different locations) on the target. The following is a way to sync up the locations of the files without copying the data again, and is quick.

Obviously, try this out on a test system first and be careful with your data.

Let's say your source data is in /tmp/src/dir/ and the target copy will be in /tmp/dst/. Note this works for remote rsync targets too.

0.

Example setup:

$ mkdir -p /tmp/src/dir; cd $_
$ fallocate -l 1000 a
$ fallocate -l 1000 b
$ fallocate -l 1000 c

$ tree /tmp/src
/tmp/src
`-- dir
    |-- a
    |-- b
    `-- c

1.

Initial copy:

$ mkdir /tmp/dst; rsync -havHP /tmp/src/dir /tmp/dst

At this point, /tmp/src/dir/ and all of the files and directories therein have been copied to /tmp/dst/.

$ tree /tmp/dst
/tmp/dst
`-- dir
    |-- a
    |-- b
    `-- c

2.

Make hard link copies of your directory structure on the source. Note: even with lots of files and directories this is very quick, because it's only copying meta-data:

$ cd /tmp/src/; mv dir dir.old
$ cp -rlp dir.old dir

# -l = link not copy
# -p = preserve permissions etc

3.

Make lots of changes in /tmp/src/dir, including moving and renaming files. In this example I'll move all files into a new subdirectory.

$ cd /tmp/src/dir; mkdir sub
$ mv a b c sub
$ tree /tmp/src/dir
/tmp/src/dir
`-- sub
    |-- a
    |-- b
    `-- c

4.

After swapping the old and the new directory structures around, the next rsync to the target is quick, because it only replicates hard links on the target, and doesn't copy data unless file contents have changed.

$ cd /tmp/src
$ mv dir dir.new; mv dir.old dir
$ rsync -havHP --delete-after --no-inc-recursive \
/tmp/src/dir /tmp/src/dir.new /tmp/dst

Note the output from rsync shows that it doesn't transfer the file contents again:

building file list ... 
9 files to consider
dir.new/
dir.new/sub/
dir.new/sub/a => dir/a
dir.new/sub/b => dir/b
dir.new/sub/c => dir/c

sent 165 bytes  received 45 bytes  420.00 bytes/sec
total size is 6.00K  speedup is 28.57

5.

Finally, tidy up by replacing the original dir with the new dir.new on both the source and the target:

$ cd /tmp/src
$ rm -rf dir
$ mv dir.new dir

$ cd /tmp/dst
$ rm -rf dir
$ mv dir.new dir

The above solution is a bit fiddly and you should be careful about others accessing the data during the directory swaps, but it's certainly an interesting feature of rsync which can save a lot of time in certain scenarios.