I have been using rsync to copy files for some time. My understanding is that rsync is faster than cp when some of the files to transfer are already in the destination directory, transferring only the incremental difference (i.e. the "deltas").

If this is correct, would there be any advantage to using rsync to moving the contents of a folder A, to say, a folder B, with B being empty?

The folder A has close to 1TB of data (and millions of files in it). The transfer would be done over a local network (A and B being on different filesystems, both mounted on a supercomputer, e.g. A is NFS and B is lustre).

Aside from that, what flags should I use to ask rsync to move (not copy) files from A to B (i.e. to delete A when the transfer has successfully finished)?

  • 7
    I don't think rsync can replace mv. I would expect mv to be faster on most file system types when the source and destination are within the same file system, because rsync would have to make a copy no matter what, and mv could probably get away with changing a few directory entries. The closest thing I can find to an rsync mv is the --remove-source-files command, but that does not remove directories. – jw013 Jul 25 '12 at 17:19
  • 2
    Thanks @jw013! Just to clarify, the files are on different filesystems, and the transfer would be done on a network. Do you know if that would still make mv faster? – Amelio Vazquez-Reina Jul 25 '12 at 17:22
  • 1
    Well, mv can't operate across a network - it would have to rely a local mount (e.g. NFS). If the bottleneck is the network, rsync would probably be faster than mv because rsync can do compression. – jw013 Jul 25 '12 at 17:25
  • 2
    By the way cp has -u option to copy source file if it is newer than the destination file or when the destination file is missing – rush Jul 25 '12 at 17:38

You can pass --remove-source-files to rsync to move files instead of copying them.

But in your case, there's no point in using rsync, since the destination is empty. A plain mv will do the job as fast as possible.

In your case, what could make a difference to performance is the choice of network protocol, if you have a choice among NFS, Samba, sshfs, sftp, rsync over ssh, tar piped into ssh, etc. The relative speed of these methods depends on the file sizes, the network and disk bandwidth, and other factors, so there's no way to give general advice, you'll need to run your own benchmarks.

| improve this answer | |
  • 8
    Just to reiterate what Caleb says, if you are worried about corruption due to e.g. a flaky network, rsync can make sense, as it verifies every file it writes by checksumming the blocks as it writes them. – Daniel S. Sterling Dec 1 '15 at 21:31
  • 6
    the --remove-source-files only deletes the files in the source. if you want to clear our the source, wouldn't you have to do an rm -rf (or find all directories and pass -delete) on the source after rsync runs successfully? – Trevor Boyd Smith Nov 29 '16 at 13:51
  • 2
    @DanielS.Sterling rsync doesn't checksum blocks after writing them (it uses checksums to find which parts of existing files were updated and need to be synchronized). You can do a second sync with --checksum to tell it to verify the results of the first synchronization. – Clément Apr 14 '19 at 0:28
  • I am migrating data from one data pool to another. For me this flag makes sense because I want to run the data on the other pool for a week or so (they are new disks) before removing them from the original pool. This flag lets me do that, ensuring everything is in sync just before deleting them from the old pool. – deed02392 Dec 16 '19 at 17:24
  • @Clement not according to the rsync man page discussing the -c flag: -c "Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred" So Rsync may do a better job than mv to a different volume / host if you are concerned with data loss. – openCivilisation Feb 22 at 12:48

Since --remove-source-files does not remove directories, I issue the following commands to move files over ssh:

rsync -avh --progress --remove-source-files /source/* user@server:/target \
&&  find /source -type d -empty -delete

I personally like the --progress feature, as I do this transfer manually. Remove it if you're using a script. I expect that it slows down transfers marginally. The find command's delete option only deletes empty directories – do not use rm -rf, as it may delete non-empty directories in case a file was not transferred. The -delete option turns on the -depth option so that empty directory trees are deleted from the "bottom" up.

| improve this answer | |
  • 4
    -delete is much nicer than -exec rmdir {} + etc – lkraav Jul 14 '16 at 6:11
  • 2
    I would skip the asterisk because and just have trailing slashes / with paths if doing this locally. If you use asterisk rsync will skip hidden files such as .htaccess or .htpasswd (if any) – Svetoslav Marinov Apr 15 '19 at 11:52
  • 1
    You should execute the find command only if rsync exists successfully. Otherwise, you'll remove empty directories in the source that you might actually want to preserve. Use the ampersand operator, so: rsync ... && find ... – deed02392 Dec 16 '19 at 17:28

In general as Gilles said there is no advantage to using rsync to move files when mv will get the same job done simpler and there is no potential speed gain between ordinary file systems.

There are however some times when there is an advantage. In particular, if you have any doubts about the stability of either the source, the destination, or the machine doing the work, using rsync gives you resume ability. This can be a notable advantage if you transfer is very large and, say, your power grid is unreliable. Using rsync will be a more stable way to avoid data corruption in the event of a failure and pick up where you left off.

| improve this answer | |
  • 5
    I'd say this is a huge advantage. In fact, I'd say mv is only better if the target and source are in the same partition, so that mv only edits the file's metadata instead of doing a copy. – nomen May 30 '18 at 23:30
  • 2
    One time I need rsync rather than mv is when I want to preserve the folder structure (if you use --relative). – Sridhar Sarnobat Oct 7 '18 at 5:30

would there be any advantage to using rsync to moving the contents of a folder A, to say, a folder B, with B being empty?

I've found myself in a situation where rsync IS faster than mv simply because mv cannot handle the number of files in the directory. I have 1.8 million photos from a security camera that ran for 20 days and the mv command exits with a failure because it cannot allocate resources.

rsync however, seems to handle all the files without a problem.

| improve this answer | |

If you want to recursively merge directories... move one directory into another directory with potentially duplicate directory names, then please see my answer here on serverfault.com. mv does a poor job when directories exist with the same name, and rsync copies (read + write full data) every file instead of just moving them (read and write only metadata).

| improve this answer | |
  • Please add the sources here. I thin it's really good script so it might give your answer the attention it deserves. – kub1x Feb 10 at 10:15

There is no way to actually move files using rsync like Linux mv command would do. By using --remove-source-files, you are essentially copying files to the destination and then deleting files (except directories) from the source. This might achieve your purpose to move files but you will NOT save time or i/o operation.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.