1

I've built a collection of files for research in a central repository on my server. I'm continually curating the repo changing the directory's tree structure by renaming and moving directories. And I'm continually updating and editing local copies of files on the repo. This creates the situation where I have a file in the repo which needs to be replaced by a newer file on the client, but the directory tree has since changed.

Is there a rsync option I haven't learned yet? I don't find rsync working with changing directory tree. I tested rsync --existing

Or, is there some commandline-fu using find and mv? Searching the net and other resources finds only search and replace in file examples.

-- UPDATE 1 If I could I'd give points to @meuh and @Lqueryvg for fantastic answers. I see @css1971 added to his answer and made a stronger case for subversion file management. Unfortunately I ran out of time and the award system simply choose it's favorite answer for me. I found all the response incredibly informative. I want to thank you all. I look forward to some time next next weekend to give them all more thoughtful review.

  • do your files all have unique names so you can track them as they move? – meuh Mar 5 '16 at 19:43
  • I'm forced to maintain unique names. These are OP's research papers, and if I change the name, then, for example, I can't be sure I'm not reading something I've already read. The changes are typically annotations to PDFs, so it's also important to not reread something I've already read. – xtian Mar 5 '16 at 21:48
  • 4
    You could create on the repo a directory ./flat/ which holds a copy of all the files, but in one flat list. The copies would be hard links, so take no space. You do the same on the local machine. You can then rsync from the local flat dir to the remote flat dir. This will update all the remote files as rsync preserves remote hard links if you use --inplace. The flat dirs could be created by script just before the rsync and removed after. – meuh Mar 5 '16 at 22:05
  • 5
    I'm inclined to suggest something like git or other version control software. – DopeGhoti Mar 5 '16 at 22:38
  • 1
    Modern version control software like git, subversion, mercurial are designed for textual files, they work best that way but all support binary files and store binary differences. – ColinSeligSmith Apr 6 '16 at 13:52
2

[expanded from my comment to the OP]. Create on the repo a directory flat which holds a copy of all the files, but in one flat list. The copies would be hard links, so take no space. Do the same on the local machine. You can then rsync from the local flat directory to the remote flat directory. This will update all the remote files as rsync preserves remote hard links if you use --inplace. The flat dirs could be created by script just before the rsync and removed after.

Here's a runnable proof of concept test script:

#!/bin/bash
dosync(){  # function to create flat trees and rsync them
    mkdir flat
    mkdir flatl
    find repo  -type f -exec ln {} flat \;
    find local -type f -exec ln {} flatl \;
    rsync -aHv --inplace flatl/ flat
    rm -fr flat flatl
}

# create repo and local dirs with same content. 3 empty files
cd /tmp || exit
mkdir repo
( cd repo; touch a b c )
mkdir local
rsync -a repo/ local

dosync
echo hi >local/a   # example change of local file a
dosync
mkdir repo/new     # example move of repo file b
mv repo/b repo/new/
echo hello >local/b  # change local file b
dosync

ls -lR repo local
# rm -fr flat flatl repo local

For the reverse direction, after dosync has transferred your local modifs to the repo, you can just rm -fr local and "rsync -a repo/ local" to copy the complete repository to the local filesystem. Instead, you can reduce the number of files needing to be transferred to only the new ones in the repo by using a similar technique:

reversesync(){
    mkdir flat
    mkdir flatl
    find repo  -type f -exec ln {} flat \;
    find local -type f -exec ln {} flatl \;
    mv flat repo/flat
    mv flatl local/flat # not flatl!
    rsync -aHv --delete repo/ local
    rm -fr repo/flat local/flat
}

This moves the flat trees into the repo and local dirs respectively, so that rsync can see the hard linked files and avoid copying them. (Obviously, the flat dirs must have the same name this time).


If you have only a single known file that you have changed, you can use find on the repository to get its new position in the tree and rsync the one file to there. Eg:

file=mychangedfile.abc
to=$(find repo -name "$file")
from=$(find local -name "$file")
rsync -av "$from" "$to"

This assumes repo is mounted, otherwise you can use ssh repo find.... If you cannot ssh to the repository, instead you can use rsync to a dummy local destination to get the list of files, and extract the one you want:

to=$(rsync -a --list-only repo dummy | awk '/\/'"$file"'$/{print $NF}')
  • Just to be clear, this is an rsync between a flat dir of hardlinks (let's call it the client) to another flat dir of hardlinks (call it the archive) both of which I make on the fly whenever I need to backup? – xtian Apr 9 '16 at 20:48
  • Yes. It is best to recreate and tear down the flat links each time because some tools will break a hard link, eg when editing a file. I've omitted any stage where you might want to copy the repo back to the client to get the new hierarchy. If you do this with the flat links still in place, then an rsync from a combined repo+flat to local+flatl will notice which files dont need to be transferred because they are already in the flatl dir, and will just add any missing dirs and a hard link to the existing file in flatl. You would need --delete in this reverse direction. – meuh Apr 9 '16 at 21:15
  • I have a short time to decide who will win this award. There are two key points to my bounty which I feel I've best expressed in my response to css1971 pseudo code solution. I feel your answer operates similarly between two directories and not a file and a directory, but you get extra points for actually providing code, which addresses the second aspect of this problem--time. Commandline solutions, and I would include bash scripts, are quick to implement, and perform one task quickly. – xtian Apr 10 '16 at 20:47
  • 1
    Perhaps I've looked for a too "complete" solution. If you just want to sync a single file to an unknown repo dir where an older version exists, then a simple find command should do. I've added this to my answer. – meuh Apr 10 '16 at 21:53
2
+25

I think rsync is the wrong tool, as are find and mv. My recommendation is instead to make use of a software configuration management system. These include Subversion, Git, Mercurial, Bazaar among others. All of which can easily handle changes of tree structure.

In the structure you describe, you have tree structure A on your client system and a repository tree structure B on a secondary location, possibly local, possibly remote.

Where you have updates to both, you now have competing changes which have to be applied in the correct order for your two repositories to remain consistent. If they are applied out of order you get into the situation you are in now, where changes in one cannot be applied directly to the other because the structures no longer exist.

As it stands, there are no options for rsync which can allow it to automatically know which changes have to be applied to bring both repositories to consistency. That isn't what it's designed to do. It can certainly make one repository look exactly like the other but it requires that only one side is changed at a time. Example being alternating changes to A, then B, then A. At any point, you need to designate one of the structures A or B as the master and sync changes in only one direction at a time.

I also don't believe there's a simple commandline-fu command which will achieve the result you're looking for, so now you're into the realm of shell programming.

Where you have changes only to the tree structure of B and only to the file contents of A, it's a relatively simple task to look for the file names of the changed files and acquire the new paths of those files in B, then modify the tree structure of A to match. This only works if the file names are unique.

The pseudo code which would bring the structure of A into consistency with B would look something like:

generate list of file names in A and their paths

For each of the names in A
    find that same name in B
    If the path of A is the same as B 
        continue to the next file
    if not then
        create the directory structure in A
        move the file to the new location.
    if the old path in A is now empty
        delete the directory.
        repeat 
            check if the parent directory is now empty, then delete it.
        until a non empty directory

Once the tree structures are in sync, then A can be copied directly to exactly the same paths in B. The --update option of rsync can be used to overwrite older files with newer ones in both directions.

Some example shell code to copy locally changed files into the existing repository, using find as the file name selector.

#!/bin/bash

set -xv

localRepo=/tmp/a
remoteRepo=/tmp/b

rm -rf $localRepo $remoteRepo

mkdir -p $localRepo/1/2/ $localRepo/1/3/
mkdir -p $remoteRepo/2/1/ $remoteRepo/3/1/

echo a12 > $localRepo/1/2/file
echo b21 > $remoteRepo/2/1/file

echo a13 > $localRepo/1/3/file1
echo b31 > $remoteRepo/3/1/file1

echo ex1
cat $localRepo/1/2/file $remoteRepo/2/1/file
echo ex2
cat $localRepo/1/3/file1 $remoteRepo/3/1/file1



localFileNameList=$(find $localRepo -type f -mtime -1 | xargs -L 1 basename)


for localFileName in $localFileNameList
do
    localFilePath=$(find $localRepo -name $localFileName | xargs dirname)
    backFile=$(find $remoteRepo -name $localFileName)
    repoDir=$(dirname $backFile)

    cp $localFilePath/$localFileName $repoDir

done

echo ex1
cat $localRepo/1/2/file $remoteRepo/2/1/file
echo ex2
cat $localRepo/1/3/file1 $remoteRepo/3/1/file1

To import a file system into subversion for example as one of the simpler to use SCMs:

e.g.
mkdir /tmp/svn
svnadmin create /tmp/svn/reponame

cd /tmp/b
svn import -m "The initial import " file:///tmp/svn/reponame
Adding         2
Adding         2/1
Adding         2/1/file
Adding         3
Adding         3/1
Adding         3/1/file1

Then check the repo out and make local changes.

$ cd /tmp
$ svn checkout file:///tmp/svn/reponame 
A    reponame/2
A    reponame/2/1
A    reponame/2/1/file
A    reponame/3
A    reponame/3/1
A    reponame/3/1/file1
Checked out revision 1.
/tmp:

$ cd reponame/
/tmp/reponame:
$ ls -ltr
total 8
drwxrwxr-x 3 css1971 css1971 4096 Apr 11 12:04 3
drwxrwxr-x 3 css1971 css1971 4096 Apr 11 12:04 2
/tmp/reponame:
$ svn move 3 4
A         4
D         3
D         3/1
D         3/1/file1
/tmp/reponame:

Commit the changes back to the repo.

$ svn commit -m "renamed dir"
Deleting       3
Adding         4

Committed revision 2.

From this point, use the svn tool as part of your normal workflow to manipulate the repository.

Useful commands:

svn import
svn update
svn commit
svn del
svn cp
svn mv

Command reference: http://svnbook.red-bean.com/en/1.7/svn.ref.html

  • I agree - OP should use repo management system, combined with cronjob or such to constantly commit/pull/push cycle. The only problem I see is that the OP needs to set up a repo server. – Otheus Apr 6 '16 at 22:28
  • What's lost here is a commandline-fu option is not a complete solution, as rsync, bash script, or DMS would be. The latter solutions sync all files between source dir and destination dir. The commandline-fu should only save one file at a time to a destination dir! The context is I'm working on some research for a week and I have a few docs/pdfs which I've altered and need to backup on the remote archive. The crisis is I have no time. My wish is for some fu to back up one file without making copies on my archive. It's not a 'solution' but a workflow. – xtian Apr 9 '16 at 20:37
  • @css1971, I was hoping you'd have a response to my comment before the time limit to the award is over...(>_<) – xtian Apr 10 '16 at 20:38
  • Sorry. My recommendation is still to use an SCM. Subversion, git work with local or remote repos no problem. From everything you've said, you appear to be re-inventing something very like an SCM, badly. The recommendation remains to fix the paths once for the inconsistencies you have, sync the repos you have and then import the file system into an SCM. Use the SCM as part of your workflow. e.g. svn mv, svn cp etc. – ColinSeligSmith Apr 11 '16 at 10:08
  • I agree, except for using svn. Just use git; it's actually designed to be distributed whereas svn is not. – Wildcard Apr 12 '16 at 1:07
0

I think that's a job for Unison ! I haven't played with it for years, but I think it could do exactly what you're asking for... The home page says:

Unison is a file-synchronization tool for OSX, Unix, and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.

Have a look and keep us posted ! ;)

  • I like the description of this package. Unfortunately it's more of a complete solution to the problem rather than the workflow or commandline fix I was hoping to get with this bounty. – xtian Apr 10 '16 at 20:40
  • how can you access the repo ? mount/ssh/rsyncd/other ? – CuriousFab Apr 11 '16 at 15:02
0

You definitely can use rsync with a changing directory structure and there are some interesting options you might not be aware of, in particular -H which preserves hard links.

I'll describe my own scenario. This may or may not work for you, but I hope you at least find it interesting.

SCENARIO:

You have a large directory with lots of files in a directory tree which are copied via rsync to another directory (possibly on a remote machine or an external disk).

You want to restructure your directory, perhaps renaming files and/or moving them to different sub-directories, but without changing the contents of the files themselves.

But, the next time you run rsync it re-copies the data for all of the files which have been moved or whose names have changed, even though they already exist (albeit in different locations) on the target. The following is a way to sync up the locations of the files without copying the data again, and is quick.

Obviously, try this out on a test system first and be careful with your data.

Let's say your source data is in /tmp/src/dir/ and the target copy will be in /tmp/dst/. Note this works for remote rsync targets too.

0.

Example setup:

$ mkdir -p /tmp/src/dir; cd $_
$ fallocate -l 1000 a
$ fallocate -l 1000 b
$ fallocate -l 1000 c

$ tree /tmp/src
/tmp/src
`-- dir
    |-- a
    |-- b
    `-- c

1.

Initial copy:

$ mkdir /tmp/dst; rsync -havHP /tmp/src/dir /tmp/dst

At this point, /tmp/src/dir/ and all of the files and directories therein have been copied to /tmp/dst/.

$ tree /tmp/dst
/tmp/dst
`-- dir
    |-- a
    |-- b
    `-- c

2.

Make hard link copies of your directory structure on the source. Note: even with lots of files and directories this is very quick, because it's only copying meta-data:

$ cd /tmp/src/; mv dir dir.old
$ cp -rlp dir.old dir

# -l = link not copy
# -p = preserve permissions etc

3.

Make lots of changes in /tmp/src/dir, including moving and renaming files. In this example I'll move all files into a new subdirectory.

$ cd /tmp/src/dir; mkdir sub
$ mv a b c sub
$ tree /tmp/src/dir
/tmp/src/dir
`-- sub
    |-- a
    |-- b
    `-- c

4.

After swapping the old and the new directory structures around, the next rsync to the target is quick, because it only replicates hard links on the target, and doesn't copy data unless file contents have changed.

$ cd /tmp/src
$ mv dir dir.new; mv dir.old dir
$ rsync -havHP --delete-after --no-inc-recursive \
/tmp/src/dir /tmp/src/dir.new /tmp/dst

Note the output from rsync shows that it doesn't transfer the file contents again:

building file list ... 
9 files to consider
dir.new/
dir.new/sub/
dir.new/sub/a => dir/a
dir.new/sub/b => dir/b
dir.new/sub/c => dir/c

sent 165 bytes  received 45 bytes  420.00 bytes/sec
total size is 6.00K  speedup is 28.57

5.

Finally, tidy up by replacing the original dir with the new dir.new on both the source and the target:

$ cd /tmp/src
$ rm -rf dir
$ mv dir.new dir

$ cd /tmp/dst
$ rm -rf dir
$ mv dir.new dir

The above solution is a bit fiddly and you should be careful about others accessing the data during the directory swaps, but it's certainly an interesting feature of rsync which can save a lot of time in certain scenarios.

  • I worked on this for several hours. First, I like this solution because it structures the way I can work with files in a very explicit way. I'm confused slightly with the workflow you propose. I've heard of hardlinks used in backups, particularly as permitting a changing structure in one dir pointing to a flat dir which is rsynced (correct me if I'm wrong). So what I'm saying is I'm not sure which directory I'm working in. – xtian Apr 9 '16 at 20:18
  • Step 2 moves the original to "dir.old" and creates the copy as "dir", so you make your changes (step 3) in the original "dir" location. There are 2 further swaps needed in the rsync step 4 and tidy step 5; but you should do these together. No, it's not "flat". It's exactly the original directory structure; hence this solution is particularly good if you are changing your directory structure & renaming files; the data doesn't need to be copied again; only the structure - so it's quick. – Lqueryvg Apr 10 '16 at 10:45
  • The intuition of the flat version is simple--it prevents duplicates to sync two flat dirs. I like that you say this solves my structure concerns. But, just to be clear, what is the intuition behind the 'swaps'? Since this is a workflow (and not a script) it's helpful to not miss a step. – xtian Apr 10 '16 at 17:07
  • I'm not going to lie to you, the swaps are not "intuitive". A less "swappy" approach would be to copy dir to dir.new in step 2 and make your changes in dir.new (step 3). Then you can remove the swap from step 4. I added the extra swaps so you can work in the orginal "dir" location. You'd probably want to wrap parts of this up in a script because it could get error prone and confusing. P.S. I'm not saying this solution is best for you, just something to consider. You should definitely look at git too as others have suggested. – Lqueryvg Apr 10 '16 at 17:37
  • Lastly, I don't understand why the rsync step has dir.work directory, as this directory doesn't exist in this scenario. And I've not seen any examples of rsync taking three arguments. The man page shows source and destination. What's the third location (and --no-inc-recursive doesn't take an argument)? – xtian Apr 10 '16 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.