2

I'm using rsync -rvhe ssh --partial-dir=path/to/partial/dir dir_to_copy remote_server:path/to/target/directory to attempt to copy a local directory to a remote server. I have also attempted this without using the --partial-dir arguement. The local copy of rsync is version 3.1.3 (included in WSL) and the remote version is 3.1.1 (included in the remote Ubuntu installation). I have already successfully completed this process for most of the files, which I did using compression (-z) and excluding the large files, and now I am attempting to complete the process by copying the large files as well.

However, rsync just hangs on certain files. This occurs both on 1) transfers and on 2) checking if a transfer has already completed. To further elaborate on this, for 1), I see the partial transfer appear in the /path/to/partial/dir (I've also seen the partial file in the target directory if I omit the --partial-dir command, as expected), and sometimes the transfer even completes in the partial directory before things stop moving. When 2) occurs, the transfer simply stops, even though the file is already present on the remote server.

Furthermore, rsync continues to run on my local machine and on the remote server, it just stops doing anything. As a result, it doesn't throw any errors for me to diagnose the issue. When I kill the local task using ^C, it returns rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(644) [sender=3.1.3], but it doesn't immediately return me to the terminal prompt--it hangs for a while after it says it received the SIGINT before printing [sender] io timeout after 60 seconds -- exiting and then returning me to the terminal prompt.

This sounds similar to what was described here: rsync keeps stopping on one directory with no errors , but I can't use their solution, which was to use cp, as I have over 10 GB to transfer and don't want to risk the copy getting interrupted and having to start from scratch.

Does anyone have any ideas how I can identify the root cause and fix this issue? Thanks!

3
  • Are you able to upgrade rsync on the remote side? 3.1.1 is from 2014.
    – Freddy
    Commented Aug 10, 2021 at 21:28
  • @Freddy not readily--this is a production system shared by many people, so we're pretty stuck with the OS and upgrades are non-trivial.
    – awho
    Commented Aug 10, 2021 at 22:08
  • I tried rsyncing the same files using the same command to a different directory and it worked with no problems. Could there have been some issue that would prevent the resumption of a transfer? If so, is there a way to address that if it happens again?
    – awho
    Commented Aug 24, 2021 at 18:39

1 Answer 1

0

You've missed the -t flag to capture times, so rsync has to fall back to checksums for verifying copies. This is extraordinarily slow and may well be the cause of the apparent slowdowns.

Try this instead,

rsync -rtvh dir_to_copy remote_server:path/to/target/directory

Rerun both your "small files only" copy and then the "all files" copy to see the effect.

3
  • The small files it's hanging on in scenario 1) I described above are ~10 MB and in scenario 2) < 1 MB, so I don't think the checksums are the bottleneck here.
    – awho
    Commented Aug 10, 2021 at 22:07
  • I just tried this just in case, and rsync has hung on a 400 kb file that is not yet on the remote server. Nothing in the partial directory either.
    – awho
    Commented Aug 10, 2021 at 22:11
  • @awho what filesystem is the target? Commented Dec 26, 2023 at 9:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .