Using rsync to move (not copy) files between directories?

Question

I have been using rsync to copy files for some time. My understanding is that rsync is faster than cp when some of the files to transfer are already in the destination directory, transferring only the incremental difference (i.e. the "deltas").

If this is correct, would there be any advantage to using rsync to moving the contents of a folder A, to say, a folder B, with B being empty?

The folder A has close to 1TB of data (and millions of files in it). The transfer would be done over a local network (A and B being on different filesystems, both mounted on a supercomputer, e.g. A is NFS and B is lustre).

Aside from that, what flags should I use to ask rsync to move (not copy) files from A to B (i.e. to delete A when the transfer has successfully finished)?

Answer 1

You can pass --remove-source-files to rsync to move files instead of copying them.

But in your case, there's no point in using rsync, since the destination is empty. A plain mv will do the job as fast as possible.

In your case, what could make a difference to performance is the choice of network protocol, if you have a choice among NFS, Samba, sshfs, sftp, rsync over ssh, tar piped into ssh, etc. The relative speed of these methods depends on the file sizes, the network and disk bandwidth, and other factors, so there's no way to give general advice, you'll need to run your own benchmarks.

Answer 2

Since --remove-source-files does not remove directories, I issue the following commands to move files over ssh:

rsync -avzh --remove-source-files --progress /source/ user@server:/target \
&&  find /source -type d -empty -delete

I personally like the --progress feature, as I do this transfer manually. Remove it if you're using a script. I expect that it slows down transfers marginally. The find command's delete option only deletes empty directories – do not use rm -rf, as it may delete non-empty directories in case a file was not transferred. The -delete option turns on the -depth option so that empty directory trees are deleted from the "bottom" up.

Answer 3

In general as Gilles said there is no advantage to using rsync to move files when mv will get the same job done simpler and there is no potential speed gain between ordinary file systems.

There are however some times when there is an advantage. In particular, if you have any doubts about the stability of either the source, the destination, or the machine doing the work, using rsync gives you resume ability. This can be a notable advantage if you transfer is very large and, say, your power grid is unreliable. Using rsync will be a more stable way to avoid data corruption in the event of a failure and pick up where you left off.

Answer 4

would there be any advantage to using rsync to moving the contents of a folder A, to say, a folder B, with B being empty?

I've found myself in a situation where rsync IS faster than mv simply because mv cannot handle the number of files in the directory. I have 1.8 million photos from a security camera that ran for 20 days and the mv command exits with a failure because it cannot allocate resources.

rsync however, seems to handle all the files without a problem.

Answer 5

If you want to recursively merge directories... move one directory into another directory with potentially duplicate directory names, then please see my answer here on serverfault.com. mv does a poor job when directories exist with the same name, and rsync copies (read + write full data) every file instead of just moving them (read and write only metadata).

Answer 6

I wrote a Bash script that implements an rsync-based mv:

#!/usr/bin/bash

echo -e "Would you like to use a relative path to your source?[y/n?]\n"
read ans
if [[ $ans == y ]]; then
    echo -e "Relative source?\n"
    read source
    source="`pwd`/$source"
elif [[ $ans == n ]]; then
    echo -e "Absolute path to your source?\n"
    read source
    source=${source/"~/"/"/home/jim/"}  
else
    echo -e "Use small cap 'y' or 'n' only."    
    exit 
fi

echo -e "Would you like to use a relative path to your destination?[y/n?]\n"
read ans2
if [[ $ans2 == y ]]; then
    echo -e "Relative destination?\n"
    read dest
    dest="`pwd`/$dest"
    dest=${dest/"~/"/"/home/jim/"}  
elif [[ $ans2 == n ]]; then
    echo -e "Absolute path to your destination?\n"
    read dest   
    dest=${dest/"~/"/"/home/jim/"}
else
    echo -e "Use small cap 'y' or 'n' only."    
    exit 
fi
rsync -avh --remove-source-files --info=progress2 ${source} ${dest} && find ${source} -type d -empty -delete

Rsync's --info=progress2 option shows statistics based on the whole transfer, rather than individual files. The script's find command only deletes empty directories. As others did already mention, Rsync's resume ability makes it a more stable way of moving files.

To alias the script as mv add the following lines to your .bashrc:

alias mv='~/ComputerScience/SoftwareDevelopment/MySoftware/Bash/mvRsync.sh'
alias sudo="sudo "

Replace my specific path in the first of the lines above with the one of your choosing. Aliasing sudo is necessary if you plan on using the alias with sudo.

Disadvantages of this script:

• No tab key completion of paths as with the mv command. This can be solved by rewriting this script so that it accepts CLI arguments instead of taking user's input with read.

Answer 7

--remove-source-files is perfectly fine if source and destination are on different volumes, but if they are on the same, you can use the following method to really move files.

Before:

find /test
/test
/test/empty_dir
/test/src
/test/src/50GB.bin

Execute rsync:

# SECONDS=0
# rsync --delete --recursive --backup --backup-dir=/test/dst /test/empty_dir/ /test/src
# echo "rsync needed $SECONDS seconds"
rsync needed 0 seconds

After:

find /test
/test
/test/dst
/test/dst/50GB.bin
/test/empty_dir
/test/src

This works because the --backup-dir feature moves and not copies files and as we compared /src against an empty dir it "deletes" everything to /dst. But note: It's still slower than mv as rsync executes a move on every file and not only on the root dir as mv would.

If you want to delete the (now empty) /src dir, use another rsync hack:

rsync --recursive --delete --include="/src**" --exclude="*" /test/empty_dir/ "/test"

Answer 8

I guess this smal script works:

copy_from=$(realpath "/tmp/a")
copy_to=$(realpath "/tmp/b")

while read from_file; do from_subdir=$(dirname $from_file | sed "s|$copy_from/||"); to_dir=$copy_to/$from_subdir; echo "Move: [$from_file] to [$to_dir]"; test ! -d $to_dir && mkdir -p $to_dir; mv $from_file $to_dir; done <<< $(find "$copy_from" -type f)