28

I was copying hundreds of files to another computer using the scp command that I got the stalled error. Now I am going to copy the files again. Is there any way to avoid copying the already copied files?

48

You can use rsync for it. rsync is really designed for this type of operation.

Syntax:

rsync -avh /source/path/ host:/destination/path

or

rsync -a --ignore-existing /local/directory/ host:/remote/directory/

When you run it first time it will copy all content then it will copy only new files.

If you need to tunnel the traffic through a SSH connection (for example, for confidentiality purposes), as indicated by you originally asking for a SCP-based solution, simply add -e ssh to the parameters to rsync. For example:

rsync -avh -e ssh /source/path/ host:/destination/path
| improve this answer | |
  • Is --ignore-existing really required? – B Faley Dec 14 '13 at 12:41
  • Because in this example of manual: rsync -t *.c foo:src/ -> "This would transfer all files matching the pattern *.c from the current directory to the directory src on the machine foo. If any of the files already exist on the remote system then the rsync remote-update protocol is used to update the file by sending only the differences" – B Faley Dec 14 '13 at 12:46
  • 1
    I typically use rsync -avuzh /source/path/ host:/destination/path. That does the whole job in one command, and on subsequent invocations, only transfers the files that need it. – Edward Falk Jun 24 '16 at 21:07
6

If you want to stick with scp for any reason, you might remove w permissions on your local data (if you have the right to do it) and scp won't touch it. To be more precise:

  1. chmod a-w local/*.tar.gz (in your local dir)
  2. scp remote/*.tar.gz local

This is not very efficient, nice but might help if you need a fast temporary solution without changing to something else than scp. (Kudos: scp without replacing existing files in the destination )

| improve this answer | |
  • I really like this solution. I am collecting files from remote sensors into an archive. The whole working directories- each has an Archive subdir. Once files go into Archive they will never be changed. It would be really easy to mark archived files as -w so this will be what I do. And the working files outside of the archive directories can be left as-is so if I run the update program during the day it will still copy the in-progress files. This is precisely what I was looking for. TNX much! – SDsolar Mar 31 '17 at 17:31
4

There was no exact answer on how to do this with SCP, since that was what the original question was asking.

sudo find /files/ -type f -exec chmod a-w {} \;

scp -r username@server.com:/file/location/* /files

When it's done copying the files change the permissions back:

sudo chmod a+w -R /files

| improve this answer | |
  • 1
    Well, arguably, the original question was "how can I solve this problem", for which rsync is the answer. – Edward Falk Jun 24 '16 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.