Using rsync to move (not copy) files between directories?

Question

I have been using rsync to copy files for some time. My understanding is that rsync is faster than cp when some of the files to transfer are already in the destination directory, transferring only the incremental difference (i.e. the "deltas").

If this is correct, would there be any advantage to using rsync to moving the contents of a folder A, to say, a folder B, with B being empty?

The folder A has close to 1TB of data (and millions of files in it). The transfer would be done over a local network (A and B being on different filesystems, both mounted on a supercomputer, e.g. A is NFS and B is lustre).

Aside from that, what flags should I use to ask rsync to move (not copy) files from A to B (i.e. to delete A when the transfer has successfully finished)?

Answer 1

You can pass --remove-source-files to rsync to move files instead of copying them.

But in your case, there's no point in using rsync, since the destination is empty. A plain mv will do the job as fast as possible.

In your case, what could make a difference to performance is the choice of network protocol, if you have a choice among NFS, Samba, sshfs, sftp, rsync over ssh, tar piped into ssh, etc. The relative speed of these methods depends on the file sizes, the network and disk bandwidth, and other factors, so there's no way to give general advice, you'll need to run your own benchmarks.

Answer 2

To answer this question, still some details are important. I'll assume some things, based on the information that was given:

• you want to copy files from one filesystem to another, then erase then from the original place.
• you could mount one filesystem on the other computer (be it on computer A or B)
• you have a fast network
• you have even faster processors, so dealing with compression / decompression won't be a problem
• you said it's a supercomputer, so I'll consider that "hard-disk" isn't a bottleneck and you're using some kind of storage.

Rsync have some advantage over a mounted fylesystem: it can compress data while copying. So it'll use less network, and will be faster.

But your problem isn't that: is that you have almost 1TB of files to copy. That will take some time, even in a fast network. Let's do very simple math about it:

• 1 TB = 1,000 GB = 1,000,000 MB of data.
• ethernet at 100 Mbits = 12,5 MB of data, per second.
• 1,000,000 / 12,5 = 80,000 seconds = 1,333 min = 22 hours.

Real-life network speed is a bit lower than that - ethernet 100 Mbips would be 8 MB of real transfer - so you would take a bit more than one day doing that.

One solution would be compressing your files before copying - for example, .tar.gz inside each dir - and then mouting one filesystem inside the other computer, and then simply copying / moving all files. In one TB of data, probably you'll achieve some compression by doing a tar before, better than compressing each file alone.

Edit: another idea: take a look at this site, it sugests using bbcp instead. BBCP can use multiple streams at the same time, filling your network completly.