0

I am doing a massive 200TB transfer with rsync. I used a shell script I found off the internet to spawn a few processes but here is the relevant rsync command itself:

        rsync --recursive \
              --whole-file \
              --inplace \
              --sparse \
              --no-compress \
              --max-alloc=8GiB \
              --size-only \
              --human-readable \
              --info=progress2 \
              --log-file="$rsync_file_basename.log" \
              --log-file-format="%o=%-7'''b | total=%-7'''l [%i] => %f%L" \
           "/mnt/disk${disk_id}/$share_name/$share_subdir/" \
           "$target_path" \
        >> "$rsync_file_basename.out"

The transfer is going pretty well but in watching the logs, I see two kinds of errors sporadically pop up.

mostly this:

rsync: [receiver] ftruncate failed on "/mnt/remotes/TS140_stuff/pictures/family_1/IMG345.jpg": Resource temporarily unavailable (11)

But also a few of these:

rsync: [receiver] write failed on "/mnt/remotes/TS140_stuff/shows/ep1.mp4": Resource temporarily unavailable (11) rsync error: error in file IO (code 11) at receiver.c(380) [receiver=3.2.7] rsync: [sender] write error: Broken pipe (32) rsync error: error in file IO (code 11) at io.c(1700) [sender=3.2.7]

Now, the target directory is an SMB mount and this is a lot of data so I expected some things like this.

But the reason for this post is that I can see the files at the target destination and they look fine. Does rsync have some kind of retry logic that it's using based on the command?

1 Answer 1

1

rsync doesn't really do retries. However there's nothing stopping you wrapping it in a loop:

#!/bin/sh
k=1 ss=1
while [ "$k" -le 5 ] 
do
    echo "Attempt $k" >&2
    rsync -rt ... &&
        ss=0 &&
        break
    sleep 60
    k=$((k+1))
done

if [ "$ss" -ne 0 ]
then
    echo "ERROR: rsync repeatedly failed" >&2
    exit 1
fi

Make sure you use the --times (-t) flag, otherwise this loop will keep repeating until $k runs out. Do you understand what the flags you've chosen actually do? They're a strange set, and I'd recommend removing most of them:

rsync -rtS
    --info=progress2 \
    --log-file="$rsync_file_basename.log" \
    --log-file-format="%o=%-7'''b | total=%-7'''l [%i] => %f%L" \
    "/mnt/disk$disk_id/$share_name/$share_subdir/" \
    "$target_path"

Finally, if you can access the server that provides the SMB share directly, you will gain a considerable speed improvement for (at least) retries by using rsync in its network mode:

rsync -rtSz local_path/ remote_host:remote_path 
3
  • thanks for your response. but in this case the files have been transferred and are in intact despite the errors. and im trying to understand why
    – john cs
    Mar 6 at 11:13
  • @johncs, oh. That's not actually what you asked. The errors are transient and may go away if you repeat the copy process for the errored files. The ftruncate issues are probably warnings that rsync couldn't complete its attempt to make the target file sparse. The write issues are more serious: you might have a file that looks to be the right size but doesn't contain the correct data Mar 6 at 11:38
  • thanks, i think ill do some checksum comparisons on the write error files to be sure!
    – john cs
    Mar 6 at 13:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .