Using Rsync to move then sync lots of data one way

Question

EDIT: Based on the two indepth answers I gave it a whirl:

rsync --progress -v -az -e “ssh” /archive/images/dcam/ root@10.x.x.xxx:/data/archive/images/dcam --dry-run

So --progress to see the results -v to make it verbose? -az to archive it (thus getting time stamps) and z to compress it to save network traffic. -e to login via ssh, of which the machine 10.x.x.xxx does have the sources ssh key in authorized key. Alas I got this error:

rsync: Failed to exec \#342\#200\#234ssh\#342\#200\#235: No such file or directory (2)
rsync error: error in IPC code (code 14) at pipe.c(84) [sender=3.0.6]
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in IPC code (code 14) at io.c(600) [sender=3.0.6]

Which is strange since remote machine has a /data/archive/images/dcam with data in it already.

I must not understand fully how rsync works. I have two servers...one has a crap ton of data I want to move to another. So...I NFS mounted the folder onto Server A from Server B (where the backup of sorts lives).

Then because this is an important live server, I was nervous to just let RSYNC run on 2TB of data... I ran by hand like this: (inside the /archive/images folder) rsync -r imageDateXX/ /mnt/backup/archive/images/imageDateXX and repeated this for 2TB of folders and data. I finally got it working. So then I was happy I didn't bring the server to the knees and then this data updates throughout the night. So to keep server B with the latest data I set up a cronjob:

0 8 * * * rsync -r /archive/images/ /mnt/backup/archive/images

This kicks off (I assume) but takes like 2 days to complete still. It looks like itis not just seeing what was new/or changed on server A and putting just that to server B but putting all the files back to server B again overwriting. I am not sure how to test this theory but it takes so long. Am I missing a switch in rsync or does my running rsync folder by folder make me then running rsync on the parent folder look 'different' to rsync where as it thinks its all new data and copies everything even if its to the same exact spot on Server b?

Not sure how to test this theory or be sure. Thought it was straight forward and rsync automagically just writes over the file or copies the file if it doesn't exists on serverB or has changed on Server A.

Answer 1

rsync --progress -av -e "ssh" /archive/images/ username@[serverIP-or-domainname]:/archive/images --dry-run

Sample:

rsync --progress -av -e "ssh" /archive/images/ fred@192.168.1.119:/archive/images --dry-run

This is assuming the directory on both machines is /archive/images, and that you have set up keys, and that the remote system has sshd running, which I'm pretty sure it does.

--dry-run is always useful to see what the action would have done, helps avoid nasty errors.

-v adds output verbosity, which is useful to track where the operation is at.

--delete removes files that no longer exist on source from destination, which you generally want if you are creating a mirror of the data on the remote system. If your data is changing a lot, you might want to look into --delete-before, --delete-after, --delete-during to see which meets your needs best. I find --delete usually works fine however, but with TiB of data, that might matter. --delete-before is useful if you are dealing with an almost full remote disk, for example.

BE CAREFUL WITH DELETE!! It will delete anything in the remote path not found in the local path, and that means, if you supply the wrong path, it will happily start deleting, or trying to delete, everything in that remote directory. Never use --delete without --dry-run the first time at least to make sure you have not made a mistake!

-rtvz is a slightly faster way to sync files than -a. I find this one is good enough for most applications.

-a basically creates almost a true mirror (-aHAX is mostly a full mirror) of the source. -a / --archive is same as -rlptgoD (no -H,-A,-X).

--progress shows progress as the job is running, which can be useful.

-e "ssh" is executing ssh, that could be a longer command if you need more ssh options in the command or whatever, like specific ssh port. Sample: -e "ssh -p 423"

-z : If you want to drop cpu usage, with not much change to bandwidth assuming binary files like images, then remove the -z compress option.

--bwlimit : useful if you are worried about eating up too much of the network bandwidth between machines, smallest speed size is 1k, 1 KiB/s, can be 1m, aka, 1 MiB/s, etc. this is quite useful if you don't want to eat up all the bandwidth of the network doing the transfer. As man says, see --max-size for syntax for different units.

The first letter of a units string can be B (bytes[not for --bwlimit), K (kilo), M (mega), G (giga), T (tera), or P (peta). If the string is a single char or has "ib" added to it (e.g. "G" or "GiB") then the units are multiples of 1024. If you use a two-letter suffix that ends with a "B" (e.g. "kb") then you get units that are multiples of 1000. The string's letters can be any mix of upper and lower-case that you want to use.

--partial : useful if you think the transfer might get interrupted, this prevents the default of rsync deleting partial transfers on break.

Note that after your first complete sync, all subsequent syncs will be radically faster since only changed files are updated. Once you have the logic working, you always want to use --delete on future syncs to keep the local and remote files in sync, deleting removed or renamed files, etc. In some configurations, only the changed data on the files is updated, for example if the file has metadata that can change, but binary core data that doesn't, only the meta data part changes. Not very applicable with images, but it is with other data types, can lead to 100x faster syncs.

rsync and nfs

Particularly if using ext4, rsync over nfs will fail because it doesn't support all filesystem attributes, if you are transferring those, which you do in the case of -a. It's also slow. nfs is fine for smaller transfers over a local network, where you are not running into extended file attribute issues, but I wouldn't use it in production. I used to do backups over nfs using rsync, and had to stop when ext4 came along because too many attributes wouldn't transfer.

Re rsync man page

Few things are more useful when working with these systems than spending some time reading the rysnc man page, for example, I had not realized --partial was a thing until today, and had struggled with very large file transfers breaking and having to start all over again on that interrupted file on next start.

I won't whitewash this however, despite in my view being one of the best bits of cli software ever created, rysnc has terrible man page, sorely in need of reorganization, it's just too hard to find stuff in it, I didn't even know about some of these until reading it today, for example, not knowing --partial for instance has cost me untold hours of lost restart of interrupted transfers of large files.

Send Andrew Tridgell a pizza, lol, that's what he used to ask for when people wanted to pay him for making rsync, but even better, help fix the man page to make it more usable, break it up into logical parts, it really is a struggle to read and use. But it's excellent documentation, but not excellently reorganized.

Answer 2

There are two major issues with your solution, which is why it's taking so long for each copy to complete:

• You're not copying the file times, so rsync has no way to identify and skip files it's already copied. Therefore each invocation will copy everything
• You're copying from what rsync sees as one part of a local filesystem to another. In this instance you don't get incremental copies, but any change to a file with cause the whole thing to be copied in its entirety

Fixes

• Include either --times (-t) or consider --archive (-a) to grab most of the metadata in one hit. Do this even if you have to continue using NFS
• Don't use NFS but instead use ssh transport to the NFS server (remoteHost in my example)
• Compress traffic over the network with --compress (-z)

Example

rsync -az /archive/images/ remoteHost:/mnt/backup/archive/images

I usually include --partial --progress --verbose (-Pv) if running interactively, too

The first time you run this modified command in your situation you'll find it still takes a long time to complete. This is because rsync has no quick way to identify which files are uptodate - and it does that with file times and sizes - so it has to compare every file pair (source and destination) to discover only the metadata is different. Thereafter, rsync will only consider a file for copying if its file size or time differs, so unchanged files will be skipped.