rsync --progress -av -e "ssh" /archive/images/ username@[serverIP-or-domainname]:/archive/images --dry-run
rsync --progress -av -e "ssh" /archive/images/ email@example.com:/archive/images --dry-run
This is assuming the directory on both machines is /archive/images, and that you have set up keys, and that the remote system has sshd running, which I'm pretty sure it does.
--dry-run is always useful to see what the action would have done, helps avoid nasty errors.
-v adds output verbosity, which is useful to track where the operation is at.
--delete removes files that no longer exist on source from destination, which you generally want if you are creating a mirror of the data on the remote system. If your data is changing a lot, you might want to look into
--delete-during to see which meets your needs best. I find
--delete usually works fine however, but with TiB of data, that might matter.
--delete-before is useful if you are dealing with an almost full remote disk, for example.
BE CAREFUL WITH DELETE!! It will delete anything in the remote path not found in the local path, and that means, if you supply the wrong path, it will happily start deleting, or trying to delete, everything in that remote directory. Never use
--dry-run the first time at least to make sure you have not made a mistake!
-rtvz is a slightly faster way to sync files than
-a. I find this one is good enough for most applications.
-a basically creates almost a true mirror (
-aHAX is mostly a full mirror) of the source.
--archive is same as
--progress shows progress as the job is running, which can be useful.
-e "ssh" is executing ssh, that could be a longer command if you need more ssh options in the command or whatever, like specific ssh port. Sample:
-e "ssh -p 423"
-z : If you want to drop cpu usage, with not much change to bandwidth assuming binary files like images, then remove the
-z compress option.
--bwlimit : useful if you are worried about eating up too much of the network bandwidth between machines, smallest speed size is 1k, 1 KiB/s, can be 1m, aka, 1 MiB/s, etc. this is quite useful if you don't want to eat up all the bandwidth of the network doing the transfer. As man says, see
--max-size for syntax for different units.
The first letter of a units string can be B (bytes[not for
--bwlimit), K (kilo), M (mega), G (giga), T (tera), or P (peta). If the string is a single char or has "ib" added to it (e.g. "G" or "GiB") then the units are multiples of 1024. If you use a two-letter suffix that ends with a "B" (e.g. "kb") then you get units that are multiples of 1000. The string's letters can be any mix of upper and lower-case that you want to use.
--partial : useful if you think the transfer might get interrupted, this prevents the default of rsync deleting partial transfers on break.
Note that after your first complete sync, all subsequent syncs will be radically faster since only changed files are updated. Once you have the logic working, you always want to use
--delete on future syncs to keep the local and remote files in sync, deleting removed or renamed files, etc. In some configurations, only the changed data on the files is updated, for example if the file has metadata that can change, but binary core data that doesn't, only the meta data part changes. Not very applicable with images, but it is with other data types, can lead to 100x faster syncs.
rsync and nfs
Particularly if using ext4, rsync over nfs will fail because it doesn't support all filesystem attributes, if you are transferring those, which you do in the case of -a. It's also slow. nfs is fine for smaller transfers over a local network, where you are not running into extended file attribute issues, but I wouldn't use it in production. I used to do backups over nfs using rsync, and had to stop when ext4 came along because too many attributes wouldn't transfer.
Re rsync man page
Few things are more useful when working with these systems than spending some time reading the rysnc man page, for example, I had not realized
--partial was a thing until today, and had struggled with very large file transfers breaking and having to start all over again on that interrupted file on next start.
I won't whitewash this however, despite in my view being one of the best bits of cli software ever created, rysnc has terrible man page, sorely in need of reorganization, it's just too hard to find stuff in it, I didn't even know about some of these until reading it today, for example, not knowing
--partial for instance has cost me untold hours of lost restart of interrupted transfers of large files.
Send Andrew Tridgell a pizza, lol, that's what he used to ask for when people wanted to pay him for making rsync, but even better, help fix the man page to make it more usable, break it up into logical parts, it really is a struggle to read and use. But it's excellent documentation, but not excellently reorganized.
rsyncis natively suited for use via
ssh, so if you have
sshaccess to Server B, I would do something like:
rsync -HAXa /archive/images/ user@serverB:/archive/images/. If you want some verbosity to show which files are being copied, use
rsync -HAXav ....
rsync: Failed to exec \#342\#200\#234ssh\#342\#200\#235: No such file or directory (2). You've used smart quotes instead of ordinary boring double quotes. Better still, remove