0

I used the command "Rsync" to transfer large amount of data, from file system to nfs(about 1Tera).

After a while, the computer shut off, and the transfered canceled (it took about 10 hours for 600G to transfer).

Some of the filesystem, that have been uploaded, was changed/added by users. I want to know, that if I use Rsync again, without deleting the partial transfer, that it will know to ignore that transferres files, and reupload what has been changed.

P.s. if there is option to resume the Rsync, will the transferred files will be ignored much faster than they have been uploaded the first time? I concerned about it, because it took me 10 hours to transfer 600G, and I hope the next Rsync will be much faster.

Edit: Apperantly I can't comment on answers... so to @Kusalananda I used rsync -rtzvx.

  • 1
    What options to rsync did you use? This matters, because if you did not use -a (--archive) or the equivalent set of options, then the file meta data may not have transferred, and every file may be seen as "new". – Kusalananda May 28 '18 at 7:44
  • @Yagel Merge your accounts....... please.... unix.stackexchange.com/contact – peterh May 28 '18 at 11:10
-1

You did not use the -a options, so user and group ownerships, nor permissions, have not been preserved. However timestamps have been due to the use of -t.

When you now restart rsync, it won't be able to use the above metadata to determine whether the file on the destination is the same as on the source. If you're feeling lucky you could use --size-only, which tells rsync to assume the file is identical if the size is the same; this usually only works correctly for things like logfiles or photos.

However things may not be as bad as you might think: rsync will checksum each file on both source and destination, and only if the checksum shows a difference will the differing blocks be transferred. I.e. not the whole file is transferred again, only the changed blocks are. This is rsync's strength: optimizing network bandwidth usage at the cost of more disk IO.

This is of course assuming rsync is transferring over the network to another host. If the NFS filesystem you speak of is mounted locally, then rsync is probably not the tool for this, as rsync will actually use much more network bandwidth while checking the files. Also rsync switches to --whole-file mode when doing local transfers, as it makes no sense to first check the entire source and destination file, and then copy the source to the destination.

In general I would recommend the use of -a if possible. You may need to combine it with --numeric-ids if you have different users on source and destination; if you have the same users but the IDs may be different, then do NOT use --numeric-ids, rsync will map IDs according to the name.

  • It won't checksum, it'll just recopy. This is because the OP is copying to an NFS target, which rsync sees as a local destination. And if you try to use --no-whole-file you'll halve the speed (at least) again. – roaima May 28 '18 at 19:58
  • @roaima Why is --no-whole-file better on NFS? What is the problem with whole file? – Yagel May 28 '18 at 20:28
  • @Yagel the --whole-file option disables the incremental algorithm, and is the default for all transfers where the source and destination appear to be in the same server's filesystem. This includes NFS and SMB mounted filesystems. Do not try to cheat the program by specifying --no-whole-file because you will slow down the transfer even more. – roaima May 29 '18 at 7:16
  • @roalma It could be that the NFS filesystem is on a remote host, the OP is not explicit about that, not showing an example command line. But if it's a local NFS filesystem then you're entirely correct. Which is why it's not recommended to use rsync on NFS filesystems: as I wrote: This is rsync's strength: optimizing network bandwidth usage at the cost of more disk IO. With NFS disk IO == network bandwidth usage... – wurtel May 29 '18 at 10:30
  • It doesn't matter whether the NFS filesystem is on a local system or a remote system. It's mounted onto the local filesystem and therefore is treated as part of the local filesystem. At this point you lose the delta algorithm. Game over. – roaima May 29 '18 at 15:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.