1

I'm trying to setup a simple back-up script using rsync and ssh that is supposed to back up my stuff to a server.

To save space in the server drive, I use the --link-dest option with the latest backup so that repeated files are just hardlinked to previous ones, so the actual rsync command will look like

rsync -av -e ssh --link-dest=user@server:/home/user/backups/previous /home/user/tobackup/ user@server:/home/user/backups/current

This is just 'paraphrased' since the script takes care of what files to backup and how to deal with what the 'current' and 'previous' directories are.

My question is, since the comparison of a given file is made against the one backed in the server, I guess rsync needs to actually send it first, then compare it, and then either save it if it's new or changed, or delete it and save a hardlink if it's unchanged? Or is the comparison somehow made without sending the file?

The reason for my question is that if the backup is big, I'd like for big unchanged files to not have to be sent, only to find they're unchanged. I don't know how rsync does this and haven't been able to find it in its documentation, so it'd be great to know if it does it and I can expect all files to be sent over the network for every single backup, or if rsync is only going to send the unchanged ones.

Obviously alternative methods are welcome if my initial idea is for some reason flawed from the start.

7
  • I am not comfortable writing a full answer, but the answer is yes - I'm pretty sure (which is why I'm not quite comfortable writing a full answer). What rsync does is it looks for differences in things like date, time, and even checksum. When it finds a difference, it transfers those files to the new location. I dug out this page which explains it nicely.
    – KGIII
    Nov 24 '20 at 17:53
  • @KGIII Thank you for the comment. And thanks for the reference, it seems I've got a lot to learn about this. As for your answer, yes to what? to that the comparison is made without sending the file? Now that I think about it, checksums are all that would need to be compared, so I could try to hack that into the script if rsync doesn't do that. Nov 24 '20 at 18:11
  • Yes, it checks for differences over a network - as your titular question asked. Check man rsync as it's actually a versatile application. I often cheat and use luckyBackup to craft advanced rsync commands. Set it up, click validate, and it gives you the command. I'm just not comfortable trying to explain the algorithm as I've never really studied it.
    – KGIII
    Nov 24 '20 at 18:18
  • @KGIII Thanks again for the comment, I didn't know the luckybackup program. I ended up figuring it out experimentally by sending a big file with rsync and finding out that succesive backups take vitually no time if the file is unchanged, so you were right. Nov 24 '20 at 18:54
  • 1
    @KGIII This won't be the case, since I'm just trying to implement a simple and decent backup system for my own computer, but thanks anyway, seems like it may a useful tool in the future and I didn't know about it. Nov 25 '20 at 20:42
1

The answer to your titular question is a solid "Yes." The rsync, be it over a network, from drive to drive, or folder to folder, checks the existing files and only updates those that have changed.

From man rsync:

Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file’s data does not need to be updated.

rsync is even more capable than that describes, as it can detect differences within a file and update those. For an example of this, see the Ubuntu testing pages where you rsync the daily .iso builds for testing.

Some light reading material:

How to Use Rsync to Sync New or Changed/Modified Files in Linux.
How rsync Works.
The rsync Manual.

If you read any of those pages, the last one is probably the most important. It shows exactly how versatile the application really is.

1

Or is the comparison somehow made without sending the file?

rsync will NOT ever send the file to compare it. The default algorithm checks if the size or last modification time changed in order to decide if the file needs to be transferred.

But you can be more strict and do for example a checksum check (many algorithms to choose from) with -c or be more relaxed and just check for the size with --size-only. Those are only two of the options, check the man page for the full list.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.