I have some photo files on my phone, and I am backing them up to an external drive. (Everything is mounted on a linux laptop.)

I have copied the files, when I should really have mv'ed them.

However, since I have done this, I would like to verify the data copied correctly for each file, and then remove the file from my phone if the file was copied without error.

So what I am really looking for is a cp or mv like command which:

  • Copies the file if it does not exist on the destination
  • Verifies the data between source and destination (after copying if the file did not already exist)
  • If both files are binary identical, remove the file from the source

I thought I might be able to do this with rsync, but I have had a look through the various delete-like options and none of them matched "delete the file on the source", rather these options delete files before/after/during(?) on the destination instead.

  • rsync has a --remove-source-files flag. (I just mentioned it in an answer here recently: unix.stackexchange.com/questions/378586/… ). However I'm not sure if it suits your needs because it requires that a source file be "part of the transfer" and "successfully duplicated on the receiving side". You might want to look into what the implications of those requirements are for your use case. – B Layer Jul 16 '17 at 12:37

(Note: rsync --remove-source-files is the command I was looking for, except that it does not verify data.)

In regards to binary compare, I have discovered that diff can do this:

In source directory:

for f in *.jpg; do diff $f "/path/to/destination/$f"; done

This compares binary files on the source and destination. No output = files do not differ.

This can be used to verify data.

Since diff returns 0 in the case where no difference was found, one can do

for f in *.jpg; do diff $f "/desination/$f" && rm $f; done

This will remove the file $f in the source folder if diff returns 0, ie; the files are binary the same.

It might be useful to make this look like a new command, let us call it verify-rm.

Since I have Dropbox installed, I am going to put my scripts in my Dropbox folder so I don't have to worry about backing them up when I re-install my OS in 6 months time...

$ cd ~/Dropbox && mkdir my-scripts && cd my-scripts
$ vim verify-rm

Here is my verify-rm script:

if [ "$#" -lt 2 ];
    echo "Bad arguments, use like cp command"
    for arg in ${@:1:$(($#-1))}
        #echo "$arg ${@:(-1):1}"
        diff "$arg" "${@:(-1):1}"
        if [ $ret -eq 0 ]
            echo "Verified file $arg"
            rm $arg
        elif [ $ret -eq 1 ]
            echo "Binary files $arg and ${@:(-1):1} differ"
            echo "diff exited with error: code=$ret"
    echo "Summary: $success files verified and ok"
    echo "         $fail files differ!"
    echo "         $error errors"
    echo "Files which differ not removed"
    echo "Files where diff returned error code not removed"

Make the script executable

$ chmod +x verify-rm

Then to use the new script ("command")

$ verify-rm /path/to/source/*.jpg /path/to/destination

I have noticed that files with spaces in the filename are not processed correctly, as bash treats each set of characters between spaces as a different argument. I am not sure what to do about this, or if it can be fixed.

However, it has been pointed out in the comments that rsync --remove-source-files will also work, and rsync works correctly when there are spaces in file names, and I've noticed it is a lot faster also, so I would recommend just using rsync.

| improve this answer | |
  • 1
    Since you answered your question yourself and made a script let me offer some tips. First you don't have a shebang. This might often not effect the outcome but in some cases it does so I'd suggest you to add #!/bin/SHELL at the beginning(replace shell with the shell you are using like #!/bin/bash). Also use tools like shellcheck.net to check whether you made some spelling, quoting or whatsoever mistakes. – ADDB Jul 16 '17 at 13:53
  • Where is the spelling error? – user3728501 Jul 16 '17 at 14:18
  • I said [to check whether you made...] I didn't find any spelling mistakes. But the quoting could be better at some points. – ADDB Jul 16 '17 at 14:19
  • Quoting of what? – user3728501 Jul 16 '17 at 14:45
  • Go to the link I posted earlier and enter the script there. That should you. – ADDB Jul 16 '17 at 14:52

You mentioned my comment about rsync --remove-source-files but if you want to take your scripting answer further here are some tips:

If you just want to do a boolean compare of any two files, including binary, there are some more direct routes than standard diff. For one, cmp does a byte by byte comparison of two files. Or you can just compare their checksums with, for example,

diff -q <(cksum file1) <(cksum file2)

As for spaces in your filenames this is a well known issue with a variety of possible solutions. This is a pretty comprehensive rundown of things: https://www.dwheeler.com/essays/filenames-in-shell.html. If you're not planning on sharing your script just choose whatever working alternative suits you best.

| improve this answer | |

rsync --remove-source-files will accomplish this exact task for you. (The one thing I have not figured out how to do, however, is have it delete the source files if the file already exists (and matches) on the destination.)

I see that you are under the impression that it does not verify files, but it does, according to the manpage:

Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred, but that automatic after-the-transfer verification has nothing to do with this option's before-the-transfer "Does this file need to be updated?" check.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.