90

I have been using rsync to copy files for some time. My understanding is that rsync is faster than cp when some of the files to transfer are already in the destination directory, transferring only the incremental difference (i.e. the "deltas").

If this is correct, would there be any advantage to using rsync to moving the contents of a folder A, to say, a folder B, with B being empty?

The folder A has close to 1TB of data (and millions of files in it). The transfer would be done over a local network (A and B being on different filesystems, both mounted on a supercomputer, e.g. A is NFS and B is lustre).

Aside from that, what flags should I use to ask rsync to move (not copy) files from A to B (i.e. to delete A when the transfer has successfully finished)?

5
  • 8
    I don't think rsync can replace mv. I would expect mv to be faster on most file system types when the source and destination are within the same file system, because rsync would have to make a copy no matter what, and mv could probably get away with changing a few directory entries. The closest thing I can find to an rsync mv is the --remove-source-files command, but that does not remove directories.
    – jw013
    Jul 25, 2012 at 17:19
  • 2
    Thanks @jw013! Just to clarify, the files are on different filesystems, and the transfer would be done on a network. Do you know if that would still make mv faster? Jul 25, 2012 at 17:22
  • 1
    Well, mv can't operate across a network - it would have to rely a local mount (e.g. NFS). If the bottleneck is the network, rsync would probably be faster than mv because rsync can do compression.
    – jw013
    Jul 25, 2012 at 17:25
  • 2
    By the way cp has -u option to copy source file if it is newer than the destination file or when the destination file is missing
    – rush
    Jul 25, 2012 at 17:38
  • What about this: Create a timestamp, then rsync all the files to remote. When done, delete all local files older than timestamp.
    – U. Windl
    Feb 8 at 14:01

8 Answers 8

99

You can pass --remove-source-files to rsync to move files instead of copying them.

But in your case, there's no point in using rsync, since the destination is empty. A plain mv will do the job as fast as possible.

In your case, what could make a difference to performance is the choice of network protocol, if you have a choice among NFS, Samba, sshfs, sftp, rsync over ssh, tar piped into ssh, etc. The relative speed of these methods depends on the file sizes, the network and disk bandwidth, and other factors, so there's no way to give general advice, you'll need to run your own benchmarks.

12
  • 11
    Just to reiterate what Caleb says, if you are worried about corruption due to e.g. a flaky network, rsync can make sense, as it verifies every file it writes by checksumming the blocks as it writes them. Dec 1, 2015 at 21:31
  • 7
    the --remove-source-files only deletes the files in the source. if you want to clear our the source, wouldn't you have to do an rm -rf (or find all directories and pass -delete) on the source after rsync runs successfully? Nov 29, 2016 at 13:51
  • 3
    @DanielS.Sterling rsync doesn't checksum blocks after writing them (it uses checksums to find which parts of existing files were updated and need to be synchronized). You can do a second sync with --checksum to tell it to verify the results of the first synchronization.
    – Clément
    Apr 14, 2019 at 0:28
  • I am migrating data from one data pool to another. For me this flag makes sense because I want to run the data on the other pool for a week or so (they are new disks) before removing them from the original pool. This flag lets me do that, ensuring everything is in sync just before deleting them from the old pool.
    – deed02392
    Dec 16, 2019 at 17:24
  • 1
    @darw You mean if there's a system crash during a copy+delete move (move between different filesystems), which results in the source file being deleted but the target not being written? That can happen both with mv and with rsync. Dec 5, 2021 at 19:58
42

Since --remove-source-files does not remove directories, I issue the following commands to move files over ssh:

rsync -avzh --remove-source-files --progress /source/ user@server:/target \
&&  find /source -type d -empty -delete

I personally like the --progress feature, as I do this transfer manually. Remove it if you're using a script. I expect that it slows down transfers marginally. The find command's delete option only deletes empty directories – do not use rm -rf, as it may delete non-empty directories in case a file was not transferred. The -delete option turns on the -depth option so that empty directory trees are deleted from the "bottom" up.

6
  • 4
    -delete is much nicer than -exec rmdir {} + etc
    – lkraav
    Jul 14, 2016 at 6:11
  • 2
    I would skip the asterisk because and just have trailing slashes / with paths if doing this locally. If you use asterisk rsync will skip hidden files such as .htaccess or .htpasswd (if any) Apr 15, 2019 at 11:52
  • 1
    You should execute the find command only if rsync exists successfully. Otherwise, you'll remove empty directories in the source that you might actually want to preserve. Use the ampersand operator, so: rsync ... && find ...
    – deed02392
    Dec 16, 2019 at 17:28
  • 1
    NB: if your /source is actually a symlink, find will not delete any empty folders after rsync is finished. You have to cd into /source and then run find . -type d -empty -delete
    – GDP2
    Sep 3, 2020 at 17:02
  • 1
    @dsz when you use -delete the -depth action is automatically enabled
    – roaima
    Feb 3 at 23:52
22

In general as Gilles said there is no advantage to using rsync to move files when mv will get the same job done simpler and there is no potential speed gain between ordinary file systems.

There are however some times when there is an advantage. In particular, if you have any doubts about the stability of either the source, the destination, or the machine doing the work, using rsync gives you resume ability. This can be a notable advantage if you transfer is very large and, say, your power grid is unreliable. Using rsync will be a more stable way to avoid data corruption in the event of a failure and pick up where you left off.

3
  • 9
    I'd say this is a huge advantage. In fact, I'd say mv is only better if the target and source are in the same partition, so that mv only edits the file's metadata instead of doing a copy.
    – nomen
    May 30, 2018 at 23:30
  • 2
    One time I need rsync rather than mv is when I want to preserve the folder structure (if you use --relative). Oct 7, 2018 at 5:30
  • rsync can though preserve hard links within moved contents, if you ask it to do, while mv cannot. Jan 17 at 9:19
19

would there be any advantage to using rsync to moving the contents of a folder A, to say, a folder B, with B being empty?

I've found myself in a situation where rsync IS faster than mv simply because mv cannot handle the number of files in the directory. I have 1.8 million photos from a security camera that ran for 20 days and the mv command exits with a failure because it cannot allocate resources.

rsync however, seems to handle all the files without a problem.

5

If you want to recursively merge directories... move one directory into another directory with potentially duplicate directory names, then please see my answer here on serverfault.com. mv does a poor job when directories exist with the same name, and rsync copies (read + write full data) every file instead of just moving them (read and write only metadata).

1
  • Please add the sources here. I thin it's really good script so it might give your answer the attention it deserves.
    – kub1x
    Feb 10, 2020 at 10:15
1

I wrote a Bash script that implements an rsync-based mv:

#!/usr/bin/bash

echo -e "Would you like to use a relative path to your source?[y/n?]\n"

read ans

if [[ $ans == y ]]; then

    echo -e "Relative source?\n"

    read source

    source="`pwd`/$source"

elif [[ $ans == n ]]; then

    echo -e "Absolute path to your source?\n"

    read source
    
    source=${source/"~/"/"/home/jerzy/"}

    
else

    echo -e "Use small cap 'y' or 'n' only."
    
    exit 

fi


echo -e "Would you like to use a relative path to your destination?[y/n?]\n"

read ans2

if [[ $ans2 == y ]]; then

    echo -e "Relative destination?\n"

    read dest

    dest="`pwd`/$dest"
    
    dest=${dest/"~/"/"/home/jerzy/"}
    

elif [[ $ans2 == n ]]; then

    echo -e "Absolute path to your destination?\n"

    read dest
    
    dest=${dest/"~/"/"/home/jerzy/"}


else

    echo -e "Use small cap 'y' or 'n' only."
    
    exit 

fi

rsync -avh --remove-source-files --info=progress2 ${source} ${dest} && find ${source} -type d -empty -delete

Rsync's --info=progress2 option shows statistics based on the whole transfer, rather than individual files. The script's find command only deletes empty directories. As others did already mention, Rsync's resume ability makes it a more stable way of moving files.

To alias the script as mv add the following lines to your .bashrc:

alias mv='~/ComputerScience/SoftwareDevelopment/MySoftware/Bash/mvRsync.sh'
alias sudo="sudo "

Replace my specific path in the first of the lines above with the one of your choosing. Aliasing sudo is necessary if you plan on using the alias with sudo.

Disadvantages of this script:

  • No tab key completion of paths as with the mv command. This can be solved by rewriting this script so that it accepts CLI arguments instead of taking user's input with read.
0

I guess this smal script works:

copy_from=$(realpath "/tmp/a")
copy_to=$(realpath "/tmp/b")

while read from_file; do from_subdir=$(dirname $from_file | sed "s|$copy_from/||"); to_dir=$copy_to/$from_subdir; echo "Move: [$from_file] to [$to_dir]"; test ! -d $to_dir && mkdir -p $to_dir; mv $from_file $to_dir; done <<< $(find "$copy_from" -type f)
1
  • You fail to quote expansions, reading values that may contain backslashes wrongly, trimming whitespace from read values, using sed on data that is not text, using sed with unsanitized shell variables, needless testing for directory before mkdir -p, and reading the output from find with newline as the delimiter even though pathnames may contain newlines. There is also an issue if the source path is a substring of the destination path. You also do not consider the actual questions posed by the asker.
    – they
    Feb 3 at 19:15
-1

--remove-source-files is perfectly fine if source and destination are on different volumes, but if they are on the same, you can use the following method to really move files.

Before:

find /test
/test
/test/empty_dir
/test/src
/test/src/50GB.bin

Execute rsync:

# SECONDS=0
# rsync --delete --recursive --backup --backup-dir=/test/dst /test/empty_dir/ /test/src
# echo "rsync needed $SECONDS seconds"
rsync needed 0 seconds

After:

find /test
/test
/test/dst
/test/dst/50GB.bin
/test/empty_dir
/test/src

This works because the --backup-dir feature moves and not copies files and as we compared /src against an empty dir it "deletes" everything to /dst. But note: It's still slower than mv as rsync executes a move on every file and not only on the root dir as mv would.

If you want to delete the (now empty) /src dir, use another rsync hack:

rsync --recursive --delete --include="/src**" --exclude="*" /test/empty_dir/ "/test"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.