1

EDIT: Based on the two indepth answers I gave it a whirl:

rsync --progress -v -az -e “ssh” /archive/images/dcam/ [email protected]:/data/archive/images/dcam --dry-run

So --progress to see the results -v to make it verbose? -az to archive it (thus getting time stamps) and z to compress it to save network traffic. -e to login via ssh, of which the machine 10.x.x.xxx does have the sources ssh key in authorized key. Alas I got this error:

rsync: Failed to exec \#342\#200\#234ssh\#342\#200\#235: No such file or directory (2)
rsync error: error in IPC code (code 14) at pipe.c(84) [sender=3.0.6]
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in IPC code (code 14) at io.c(600) [sender=3.0.6]

Which is strange since remote machine has a /data/archive/images/dcam with data in it already.

I must not understand fully how rsync works. I have two servers...one has a crap ton of data I want to move to another. So...I NFS mounted the folder onto Server A from Server B (where the backup of sorts lives).

Then because this is an important live server, I was nervous to just let RSYNC run on 2TB of data... I ran by hand like this: (inside the /archive/images folder) rsync -r imageDateXX/ /mnt/backup/archive/images/imageDateXX and repeated this for 2TB of folders and data. I finally got it working. So then I was happy I didn't bring the server to the knees and then this data updates throughout the night. So to keep server B with the latest data I set up a cronjob:

0 8 * * * rsync -r /archive/images/ /mnt/backup/archive/images

This kicks off (I assume) but takes like 2 days to complete still. It looks like itis not just seeing what was new/or changed on server A and putting just that to server B but putting all the files back to server B again overwriting. I am not sure how to test this theory but it takes so long. Am I missing a switch in rsync or does my running rsync folder by folder make me then running rsync on the parent folder look 'different' to rsync where as it thinks its all new data and copies everything even if its to the same exact spot on Server b?

Not sure how to test this theory or be sure. Thought it was straight forward and rsync automagically just writes over the file or copies the file if it doesn't exists on serverB or has changed on Server A.

4
  • Personally, I wouldn't use rsync over NFS. rsync is natively suited for use via ssh, so if you have ssh access to Server B, I would do something like: rsync -HAXa /archive/images/ user@serverB:/archive/images/. If you want some verbosity to show which files are being copied, use rsync -HAXav ....
    – Jim L.
    Commented Sep 22, 2021 at 19:30
  • How many files under /archive/images?
    – BowlOfRed
    Commented Sep 23, 2021 at 7:26
  • rsync: Failed to exec \#342\#200\#234ssh\#342\#200\#235: No such file or directory (2). You've used smart quotes instead of ordinary boring double quotes. Better still, remove -e "ssh" entirely Commented Sep 23, 2021 at 16:46
  • 1
    As roamia notes, you used magic quotes. Break the habit of using any word processing software when it comes to noting commands etc, those love making these changes. Whatever software you used to store that command, get rid it, never use it for this purpose again. The "ssh" I had included more to give a feel for if you needed further options for ssh than default. There's no risk to the data on the source server, rsync isn't changing it, it's copying it.
    – Lizardx
    Commented Sep 24, 2021 at 0:15

2 Answers 2

3
rsync --progress -av -e "ssh" /archive/images/ username@[serverIP-or-domainname]:/archive/images --dry-run

Sample:

rsync --progress -av -e "ssh" /archive/images/ [email protected]:/archive/images --dry-run

This is assuming the directory on both machines is /archive/images, and that you have set up keys, and that the remote system has sshd running, which I'm pretty sure it does.

--dry-run is always useful to see what the action would have done, helps avoid nasty errors.

-v adds output verbosity, which is useful to track where the operation is at.

--delete removes files that no longer exist on source from destination, which you generally want if you are creating a mirror of the data on the remote system. If your data is changing a lot, you might want to look into --delete-before, --delete-after, --delete-during to see which meets your needs best. I find --delete usually works fine however, but with TiB of data, that might matter. --delete-before is useful if you are dealing with an almost full remote disk, for example.

BE CAREFUL WITH DELETE!! It will delete anything in the remote path not found in the local path, and that means, if you supply the wrong path, it will happily start deleting, or trying to delete, everything in that remote directory. Never use --delete without --dry-run the first time at least to make sure you have not made a mistake!

-rtvz is a slightly faster way to sync files than -a. I find this one is good enough for most applications.

-a basically creates almost a true mirror (-aHAX is mostly a full mirror) of the source. -a / --archive is same as -rlptgoD (no -H,-A,-X).

--progress shows progress as the job is running, which can be useful.

-e "ssh" is executing ssh, that could be a longer command if you need more ssh options in the command or whatever, like specific ssh port. Sample: -e "ssh -p 423"

-z : If you want to drop cpu usage, with not much change to bandwidth assuming binary files like images, then remove the -z compress option.

--bwlimit : useful if you are worried about eating up too much of the network bandwidth between machines, smallest speed size is 1k, 1 KiB/s, can be 1m, aka, 1 MiB/s, etc. this is quite useful if you don't want to eat up all the bandwidth of the network doing the transfer. As man says, see --max-size for syntax for different units.

The first letter of a units string can be B (bytes[not for --bwlimit), K (kilo), M (mega), G (giga), T (tera), or P (peta). If the string is a single char or has "ib" added to it (e.g. "G" or "GiB") then the units are multiples of 1024. If you use a two-letter suffix that ends with a "B" (e.g. "kb") then you get units that are multiples of 1000. The string's letters can be any mix of upper and lower-case that you want to use.

--partial : useful if you think the transfer might get interrupted, this prevents the default of rsync deleting partial transfers on break.

Note that after your first complete sync, all subsequent syncs will be radically faster since only changed files are updated. Once you have the logic working, you always want to use --delete on future syncs to keep the local and remote files in sync, deleting removed or renamed files, etc. In some configurations, only the changed data on the files is updated, for example if the file has metadata that can change, but binary core data that doesn't, only the meta data part changes. Not very applicable with images, but it is with other data types, can lead to 100x faster syncs.

rsync and nfs

Particularly if using ext4, rsync over nfs will fail because it doesn't support all filesystem attributes, if you are transferring those, which you do in the case of -a. It's also slow. nfs is fine for smaller transfers over a local network, where you are not running into extended file attribute issues, but I wouldn't use it in production. I used to do backups over nfs using rsync, and had to stop when ext4 came along because too many attributes wouldn't transfer.

Re rsync man page

Few things are more useful when working with these systems than spending some time reading the rysnc man page, for example, I had not realized --partial was a thing until today, and had struggled with very large file transfers breaking and having to start all over again on that interrupted file on next start.

I won't whitewash this however, despite in my view being one of the best bits of cli software ever created, rysnc has terrible man page, sorely in need of reorganization, it's just too hard to find stuff in it, I didn't even know about some of these until reading it today, for example, not knowing --partial for instance has cost me untold hours of lost restart of interrupted transfers of large files.

Send Andrew Tridgell a pizza, lol, that's what he used to ask for when people wanted to pay him for making rsync, but even better, help fix the man page to make it more usable, break it up into logical parts, it really is a struggle to read and use. But it's excellent documentation, but not excellently reorganized.

3
  • This is a great writeup, I am going to try this tomorrow when the system is not under heavy use. For 2.2TB of data, can I just fire and forget this or can it bring down the host system by consuming resources etc.
    – Codejoy
    Commented Sep 22, 2021 at 20:57
  • I'll update my answer, you can drop cpu a bit by removing -z, compress, you can reduce bandwidth by --bwlimit
    – Lizardx
    Commented Sep 22, 2021 at 23:26
  • --partial is really only relevant either when copying across a network, or when you can guarantee the source data hasn't changed so you're also planning to use --append. The second situation is much less common than you might initially think Commented Sep 24, 2021 at 8:31
1

There are two major issues with your solution, which is why it's taking so long for each copy to complete:

  • You're not copying the file times, so rsync has no way to identify and skip files it's already copied. Therefore each invocation will copy everything
  • You're copying from what rsync sees as one part of a local filesystem to another. In this instance you don't get incremental copies, but any change to a file with cause the whole thing to be copied in its entirety

Fixes

  • Include either --times (-t) or consider --archive (-a) to grab most of the metadata in one hit. Do this even if you have to continue using NFS
  • Don't use NFS but instead use ssh transport to the NFS server (remoteHost in my example)
  • Compress traffic over the network with --compress (-z)

Example

rsync -az /archive/images/ remoteHost:/mnt/backup/archive/images

I usually include --partial --progress --verbose (-Pv) if running interactively, too

The first time you run this modified command in your situation you'll find it still takes a long time to complete. This is because rsync has no quick way to identify which files are uptodate - and it does that with file times and sizes - so it has to compare every file pair (source and destination) to discover only the metadata is different. Thereafter, rsync will only consider a file for copying if its file size or time differs, so unchanged files will be skipped.

6
  • updated my question
    – Codejoy
    Commented Sep 23, 2021 at 16:34
  • :D cut and paste job, ty checking the dry run now! Thought I neeed that to tell it to use ssh. I kept it the same without the -e "ssh" and now am getting a command not found, if i change it back to the /mnt/ folder that is the nfs mount it seems to give me a list on the dry run
    – Codejoy
    Commented Sep 23, 2021 at 16:59
  • LOL I am in full derp mode, it was not. is now but I had already started it just pointing it to the nfs mount. I think since I used the -Pv I am safe to ctrl-break it when I need to at any time? It seems to be working well a few files (that are on the remote not the host) give a strange error when the host gets back to them again: rsync: chown "/mnt/archive_copy/archive/images/dcam/UT201006/.proc-d0355.fits.IGw8LT" failed: Operation not permitted (1) but other than that its all good.
    – Codejoy
    Commented Sep 23, 2021 at 18:14
  • The -a or -t is what makes it safe to interrupt and restart. The -Pv is purely for visual feedback on its progress Commented Sep 23, 2021 at 18:20
  • So I finally got back to this project. The final command I used was: rsync -Pv -az /archive/folder /mnt/archive_copy/folder It worked well but rerunning it after a break tehre are a lot of chown operation not permitted issues. I think I am not copying over the owner ship? Lastly, I can run this...it finishes. I can run it right away again and after a bunch of chown operation not permitted it finds more files to copy. These are not new ones either from like older folders????? It concerns me it seems to miss files. Not sure what I am messing up.
    – Codejoy
    Commented Oct 24, 2021 at 16:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .