4

I have around 50 gigabytes that I would like to move. I want to do it over TCP/IP (hence network in the title) optimized for a local area network. My problem is that the connection occasionally gets interrupted and I never seem to get all of the data reliably to its destination. I'd like this thing to

  1. not give up so easily
  2. keep retrying automatically (assuming that both machines are powered up).

My approach would be to use rsync.

SOURCE=/path/to/music/ # slash excludes "music" dir
DESTINATION=/path/to/destination 
rsync \
  --archive \ # archive mode; equals -rlptgoD (no -H,-A,-X)
  --compress \ # compress file data during the transfer
  --progress \ # show progress during transfer
  --partial \ # delete any partially transferred files after interrupt, on by default but I added it for kicks
  --human-readable \ #output numbers in a human-readable format
  "$SOURCE" \
  "$DESTINATION" \

Are there other parameters that I should consider?

  • And rsync doesn't work for you because ...? Please elaborate. – frostschutz Jun 20 '17 at 9:58
  • @frostschutz I'd like some usage advice by more experienced users of rsync. Specifically, what parameters would make the transfer of ≥50gb efficient and reliable? – Jonathan Komar Jun 20 '17 at 10:00
  • @JonathanKomar that will depend on your connection among other things. Is this being transferred over the internet? Between machines on the same LAN? Do you have any reason to assume that what you are showing won't be reliable? Are you just asking how to make this restart on failure? – terdon Jun 20 '17 at 13:34
  • @terdon On the same LAN is ideal, but sometimes I transfer larger directories over the internet over secure shell. I am not only asking how to make this restart on failure, I am also asking for the optimum parameters to make that process efficient and reliable. I have no reason to assume that it won't be reliable other than I do not do this kind of operation very often and I know there are some gurus out there who might provide some insight or orthodox way of getting the job done. – Jonathan Komar Jun 20 '17 at 13:42
  • OK, please edit your question and explain the network you will be using. for instance, compression makes little sense for LAN transfers (the overhead of compressing could end up slowing the transfer down) but makes a lot of sense over the internet. – terdon Jun 20 '17 at 13:57
0

Rsync Parameters

It would seem that my rsync parameters are fine.

I had to add a parameter to deal with files that exist after a connection failure. The choices were --ignore-existing or --update to avoid rewriting things already written. I am still not sure which one is better (perhaps someone knows) but in this case I went with with --update after reading this https://askubuntu.com/questions/399904/rsync-has-been-interrupted-copy-from-beginning

Compare:

  • --update skip files that are newer on the receiver
  • --ignore-existing skip updating files that already exist on receiver

Connection Interruptions

The connection problem conundrum (flaky wifi etc.) was solved by continually calling rsync when an exit code is not zero, thereby forcing my process to continue until the transfer is a success. (unless I cut the power, lightning strikes my power lines, or I kill it using a signal)

To handle network disconnects, I used a while loop.

while [ 1 ]
do
# STUFF
done

while [ 1 ] has a caveat: using a signal like ctrl c for an interrupt (SIGINT) will not work unless you add an explicit check for any exit codes above 128 that calls break.

if [ "$?" -gt 128 ] ; then break

then you can check for rsync's exit code. Zero means all files have been moved.

elif [ "$?" -eq 0 ] ; then exit

Otherwise, the transfer is not complete.

else sleep 5

Script Example sync-music.sh

The rsync script assumes ssh passwordless key authentication.

#!/bin/bash

SOURCE="/path/to/Music/"
DESTINATION="user@computer.local:/media/Music"

while [ 1 ]
do
  rsync -e 'ssh -p22'\
  --archive \
  --compress \
  --progress \
  --partial \
  --update \
  --human-readable \
  "$SOURCE" \
  "$DESTINATION"

  if [ "$?" -gt 128 ] ; then
    echo "SIGINT detected. Breaking while loop and exiting."
    break
  elif [ "$?" -eq 0 ] ; then
    echo "rsync completed normally"
    exit
  else
    echo "rsync failure. reestablishing connection after 5 seconds"
    sleep 5
  fi
done
  • You don't need -p22 on the ssh because that's the default. And since ssh is now the default transfer protocol for rsync you can skip the entire -e 'ssh -p22'. If you prefer you can reduce the rsync options to -ahvzP. This gives you a nice concise rsync -ahvzP "$SOURCE" "$DESTINATION". Unless you really have a specific reason to use --update or --ignore-existing don't use either; let rsync figure it out for you. The rest is a fairly standard loop-until-finished and should work well for you. – roaima Jul 4 '17 at 21:51
  • @roaima changing ports uses a strange syntax, so I included it to keep it easy to change. Also, I purposely use verbose parameters (here) only for transparency. And you do need --update or --ignore-existing. Otherwise rsync will start from the beginning at each network interruption. --partial only works on individual files. – Jonathan Komar Jul 5 '17 at 4:23
  • You should not need either of --update or --ignore-existing. They will not work with --partial. rsync is designed to start from the beginning each time but it skips correctly copied files. You've specified a remote target in your example; don't test with a local filesystem target because the algorithm changes under that scenario. – roaima Jul 5 '17 at 7:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.