182

How do I copy an entire directory into a directory of the same name without replacing the content in the destination directory? (instead, I would like to add to the contents of the destination folder)

  • 1
    Yes, it is annoying when a useful program/utility does not have that one "if only it could ..." option ! In this case "--noclobber" ! – MikeW Feb 7 '18 at 10:57
164

Use rsync, and pass -u if you want to only update files that are newer in the original directory, or --ignore-existing to skip all files that already exist in the destination.

rsync -au /local/directory/ host:/remote/directory/
rsync -a --ignore-existing /local/directory/ host:/remote/directory/

(Note the / on the source side: without it rsync would create /remote/directory/directory.)

| improve this answer | |
  • @Anthon I don't understand your comment and I don't see an answer or comment by chandra. --ignore-existing does add without replacing, what data loss do you see? – Gilles 'SO- stop being evil' Nov 27 '13 at 9:59
  • Sorry, I only looked at your first example that is where you can have data loss (and is IMHO not what the OP asked for), if you include --ignore-existing data-loss should not happen. – Anthon Nov 27 '13 at 10:08
  • This does not help if the remote system does not have rsync easily available.... (Like Win32-OpenSSH) – Gert van den Berg Oct 25 '16 at 8:00
  • @GertvandenBerg rsync is pretty easy to install on Windows, no harder than SSH. – Gilles 'SO- stop being evil' Oct 25 '16 at 11:51
  • @Gilles: True, but all of the options seems to involve Cygwin DLLs... (The current state of the MS port of OpenSSH is such that enabling compression on scp is enough to break SCP...) (Getting rsync functional over Win32-OpenSSH also seems non-trivial - hopefully that improves over time) (Solaris 10 is the other example, where a third party package and --rsync-path is needed) – Gert van den Berg Oct 25 '16 at 13:01
95

scp will overwrite the files if you have write permissions to them.

In other words: You can make scp effectively skip said files by temporarily removing the write permissions on them (if you are the files' owner, that is).

| improve this answer | |
  • 6
    Thanks for this. Was exactly the trick I was looking for. – saccharine Jul 16 '13 at 21:02
  • make sure you copy the files back you add a * to do so. Example scp -r user@server.com:/location/of/files/* /local/location/ – Rick May 27 '15 at 19:16
  • 1
    find . -type f -exec chown root:root {} \; – ling Aug 21 '16 at 19:58
  • 1
    In the uncommon case that only some of the files in the directory are expected to be overwritten (if not "protected"), and file permissions are not uniform across files, this solution may not work. Otherwise, it is quite simple and effective. – sancho.s ReinstateMonicaCellio May 20 '18 at 12:55
  • To do this on windows select all the files, right click -> properties -> read only – BenJammin Sep 6 '18 at 15:17
12

You can copy only new files by date. Use find

scp  `find /data/*.gz -type f -mtime +7` USER@SERVER:/backup/
| improve this answer | |
  • 1
    what does +7 mean? – Francis Kim May 1 '17 at 13:07
  • @FrancisKim it means modified 7 days ago. So this query select any *.gz files modified within last 7 days – Vadim Eremeev May 24 '17 at 21:34
  • 4
    Doesn't +7 actually imply more than 7 days ago? – Jeff Mar 21 '18 at 22:51
  • 2
    Yes it means "more than 7 days ago". No idea who upvoted this. – Stefan Reich Dec 12 '18 at 14:59
11

If you can make the destination file contents read-only:

find . -type f -exec chmod a-w '{}' \;

before running scp (it will complain and skip the existing files).

And change them back afterward ( chmod +w to get umask based value). If the files do not all have write permission according to your umask, you would somehow have to store the permissions so that you can restore them.

(Gilles' answer overwrites existing files if locally they are newer, I lost valuable data that way. Do not understand why that wrong and harmful answer has so many up votes).

| improve this answer | |
  • 2
    I don't get it: how did rsync --ignore-existing cause you to lose data? – Gilles 'SO- stop being evil' Nov 27 '13 at 10:01
  • 4
    I got the error find: missing argument to '-exec' using this command, and instead had to use: find . -type f -exec chmod a-w {} \;. My linux is bad, ymmv. – wpearse Apr 6 '15 at 0:16
  • This is a better and safer answer. The correct command to change the permission is as @wpearse mentioned: find . -type f -exec chmod a-w {} \; – Amir Oct 17 '17 at 16:41
8

I had a similar task, in my case I could not use rsync, csync, or FUSE because my storage has only SFTP. rsync could not change the date and time for the file, some other utilities (like csync) showed me other errors: "Unable to create temporary file Clock skew detected".

If you have access to the storage-server - just install openssh-server or launch rsync as a daemon here.

In my case - I could not do this and the solution was: lftp. lftp's usage for syncronization is below:

lftp -c "open -u login,password sftp://sft.domain.tld/; \
    mirror -c --verbose=9 -e -R -L /srs/folder /rem/folder"

/src/folder - is the folder on my PC, /rem/folder - is sftp://sft.domain.tld/rem/folder.

You may find man pages by the link: http://lftp.yar.ru/lftp-man.html

| improve this answer | |
  • Fabulous! While it's not using scp (as the binary), but sftp (as the same protocol), it helps to achieve a synchronization when sftp is the only protocol available - hence no ssh and therefore no rsync, ls, tar or any of the other proposed solutions. Thank you! Btw, if you want to mirror the remote folder to a local folder, just drop the "-R" flag. – Martin Rüegg Jun 14 at 10:59
  • Awesome, this worked for me as well. Just some notes. Careful about using -R, that deletes items if they're not present (believe its similar to the --delete flags in rsync). The /rem/folder is a relative path. So if you wanted to copy something to the base path, you'd do: sftp://stf.domain.tld/; ...... /src/folder ../../ – Sean Breckenridge Jun 29 at 1:31
7

To copy a whole bunch of files, it's faster to tar them. By using -k you also prevent tar from overwriting files when unpacking it on the target system.

tar -c <source-dir> | ssh <name>@<host> 'tar -kxzf - -C <target-dir>' 
| improve this answer | |
  • 1
    It does make a remote connection. First it tar's the source, pipes it into the ssh connection and unpacks it on the remote system. – huembi Aug 22 '16 at 21:17
-1

Another way to achieve this is to do a ls on the destination folder:

On remote destination folder:

ls | awk '{print "mv " $1 " ../copied_data/"}' > mv_copied_data
scp mv_copied_data user@source.server.com:/path/to/source/folder

On source:

cd /path/to/source/folder
chmod 777 mv_copied_data
./mv_copied_data

On destination:

scp -r user@source.server.com:/path/to/source/folder /path/to/destination/foldeer
| improve this answer | |
  • 1
    Please don't do this. 1. Don't parse ls. 2. The answer is dangerous if any filename contains - > space newline or other special characters. 3. Hard-code a for-loop in a python script. Generating a program at runtime is hard and tricky (even when done by professional programmers), for a lot of reasons. Generating a shell script is even worse - an awful idea. – ignis Jun 1 '19 at 21:12
-2

scp does overwrite files and there's no switch to stop it doing that, but you can copy things out the way, do the scp and then copy the existing files back. Examples:

  1. Copy all existing files out the way

    mkdir original_files ; cp -r * original_files/
    
  2. Copy everything using scp

    scp -r user@server:dir/* ./
    
  3. Copy the original files over anything scp has written over:

    cp -r original_files/* ./
    
| improve this answer | |
  • 4
    This method doesn't help when you're trying to pull files over from a remote and pick up where you left off. I.e. if the whole purpose is to save time. – Oliver Williams Dec 1 '16 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy