0

I'm trying to synchronize literally thousands of files of various sizes and I would like to have a 1:1 copy of the files. That means that already present files should be checked for their integrity and if there's a wrong checksum, the file needs to be overwritten. A so-called delta transfer is only necessary at this point because of the partially failed transfer.

Apparently my mount is kinda unstable and it fails after 300-400GB of transfer using cp or rsync.

I did the following before this:

  1. I mounted the storage, and did cp -r src dest, it failed after like 300GB because the mount dropped and it errored out (don't have the error anymore apparently)
  2. I mounted the storage again and did rsync -aP src dest, it failed after like 400GB with rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7] because the mount failed again. Considering the file size it probably overwrite most of the files.
  3. I checked my kernel log and found nothing (sudo dmesg)

I found a reconnect flag for my mount, but it would not be instant.

  • There's a rsync flag named -c which calculates the checksums, but does it do a so-called delta transfer too or do I need to add more flags?

How could I best fix this problem at hand?

UPDATE 1

Correct me if I'm wrong but I think the issue at hand was that the storage had the owners and groups of different users and groups than in rsync. To elaborate: cp -r copied the files and changed their ownership and group ownership to the user copying, whereas rsync seems to copy the file 1:1 with the same user and group ownership... That's probably why the transfer was overwriting old files...

0

1 Answer 1

1

You sound like you are copying between two local filesystems. If so then rsync will not use its checksum scheme and will fall back to being cp.

Both cp and rsync will preserve owner/group if they are run as root. Otherwise the destination files will take the owner and (default) group of the account performing the copy.

Provided you copied timestamps with your original cp you can tell rsync just to check the file size and timestamp and use that to decide whether to update missing metadata or to recopy the file in its entirety:

rsync -rt src/ dst    # Timestamps only
rsync -a src/ dst     # Almost all metadata

Certain filesystems may not be able to preserve metadata. Or you may not be running as root. In these situations you can try to save metadata alongside each file in its extended attributes. You'd add -X for that:

rsync -aX src/ dst    # Almost all metadata

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .