1

I want to move a large number of large files over ssh.

Because I want to free space, I want to delete the files after they have been moved. Because the amount of data is large, I want to be able to interrupt the transfer any time and continue it with the same command later. (In fact my internet connection resets at least once per day and interrupts the transfer)

Unfortunately, I cannot use rsync, at least not like this:

rsync -avz --remove-source-files user@source:/path/ /destination_path/

rsync only deletes the source files after ALL have been copied - if the transfer is interrupted, no space is freed at all. Putting the command in cron will cause it to never finish without manually deleting files that have already been copied.

Is there a solution for this problem?

  • Shouldn't rsync be able to effectively continue where it left off, since it usually only transfers changes to the files? – ilkkachu Jun 6 '18 at 18:17
  • That's not the issue; the issue is that OP wants to remove anything copied ASAP due to storage constraints. – DopeGhoti Jun 6 '18 at 18:31
3

If you cannot use rsync, and want to remove source files only after each file has been successfully copied, you can do something like this for a directory-by-directory, file-by-file transfer:

if cd /path/to/files; then
for file in *; do
    if scp -pr "$file" user@remotehost.example.com:/destination/files/"$file"; then
        rm -fr "$file"
    else
        echo "Transfer of '$file' failed.  Not removing local copy." 1>&2
    fi
done; fi

If you want to do an additional sanity-check before deleting the local file, you can add a consistency check, but this will slow things down for very large files, and this quick-and-dirty checksum will only work on files; not directories:

if cd /path/to/files; then
for file in *; do
    if scp -pr "$file" user@remotehost.example.com:/destination/files/"$file"; then
        if [[ "$(md5sum < "$file" )" = "$( ssh user@remotehost.example.com md5sum < /destination/files/"$file" )" ]]; then
            rm -fr "$file"
        else
            echo "Unable to validate remote '$file'.  Not removing local copy" 1>&2
        fi
    else
        echo "Transfer of '$file' failed.  Not removing local copy." 1>&2
    fi
done; fi
  • 1
    I would wrap the whole thin in an if cd /path/to/files so that way it doesn't accidentally remove the wrong files. – Zachary Brady Jun 6 '18 at 17:56
  • A reasonable precaution. Implemented. – DopeGhoti Jun 6 '18 at 18:01
  • Hasn't your second block just reimplemented rsync-in-a-loop? – roaima Jun 6 '18 at 23:01
  • Yes, but because the OP specified that files should be deleted immediately after copying rather than eventually after copying as with rsync --delete. – DopeGhoti Jun 7 '18 at 15:28
1

rsync only deletes the source files after ALL have been copied

This is a false premise. It might appear to be true when you have only a few large files but it's certainly not true in the general case.

The rsync command queues a delete instruction as soon as a file has been successfully transferred. However, because instructions are multiplexed with other data it may take "a while" for the delete to be applied on the source side.

If you run rsync with a large number of files you will see files being deleted on the source before all the transfers have been completed. (I run rsync with tens or even hundreds of thousands of files in a session and I do see this behaviour.)

Furthermore, if you have a scenario where a transfer is interrupted, when the rsync is restarted it will delete the files it previously successfully transferred before moving on to the next set of files to transfer. (I also see this behaviour.)

Consider https://superuser.com/a/405795/332907 for reference to source code as evidence.

  • In my tests, even though the files take about 10-15 minutes to transfer, nothing was deleted after I pressed "Ctrl+C" to abort the transfer after three files were transferred – Roland Seuhs Jun 7 '18 at 7:42
  • @RolandSeuhs if you stop rsync running it can't compete its job (including deleting transferred files). But if you now go back and rerun it you'll find it starts to delete them. – roaima Jun 7 '18 at 7:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.