228

How do I copy an entire directory into a directory of the same name without replacing the content in the destination directory? (instead, I would like to add to the contents of the destination folder)

1
  • 1
    Yes, it is annoying when a useful program/utility does not have that one "if only it could ..." option ! In this case "--noclobber" !
    – MikeW
    Feb 7, 2018 at 10:57

9 Answers 9

201

Use rsync, and pass -u if you want to only update files that are newer in the original directory, or --ignore-existing to skip all files that already exist in the destination.

rsync -au /local/directory/ host:/remote/directory/
rsync -a --ignore-existing /local/directory/ host:/remote/directory/

(Note the / on the source side: without it rsync would create /remote/directory/directory.)

8
  • @Anthon I don't understand your comment and I don't see an answer or comment by chandra. --ignore-existing does add without replacing, what data loss do you see? Nov 27, 2013 at 9:59
  • Sorry, I only looked at your first example that is where you can have data loss (and is IMHO not what the OP asked for), if you include --ignore-existing data-loss should not happen.
    – Anthon
    Nov 27, 2013 at 10:08
  • 1
    This does not help if the remote system does not have rsync easily available.... (Like Win32-OpenSSH) Oct 25, 2016 at 8:00
  • @GertvandenBerg rsync is pretty easy to install on Windows, no harder than SSH. Oct 25, 2016 at 11:51
  • @Gilles: True, but all of the options seems to involve Cygwin DLLs... (The current state of the MS port of OpenSSH is such that enabling compression on scp is enough to break SCP...) (Getting rsync functional over Win32-OpenSSH also seems non-trivial - hopefully that improves over time) (Solaris 10 is the other example, where a third party package and --rsync-path is needed) Oct 25, 2016 at 13:01
109

scp will overwrite the files if you have write permissions to them.

In other words: You can make scp effectively skip said files by temporarily removing the write permissions on them (if you are the files' owner, that is).

9
  • 7
    Thanks for this. Was exactly the trick I was looking for.
    – saccharine
    Jul 16, 2013 at 21:02
  • 1
    find . -type f -exec chown root:root {} \;
    – ling
    Aug 21, 2016 at 19:58
  • 2
    In the uncommon case that only some of the files in the directory are expected to be overwritten (if not "protected"), and file permissions are not uniform across files, this solution may not work. Otherwise, it is quite simple and effective. May 20, 2018 at 12:55
  • 1
    To do this on windows select all the files, right click -> properties -> read only
    – BenJammin
    Sep 6, 2018 at 15:17
  • 1
    Nice trick. You can add 2>/dev/null at the end to disregard the complaints. My attempt was to copy over remote 19200 files ignoring the existing files, so the screen was a mess.
    – David Jung
    Nov 19, 2018 at 4:46
18

You can copy only new files by date. Use find

scp  `find /data/*.gz -type f -mtime -7` USER@SERVER:/backup/

From the manpage (-atime is for last accessed time, but the principle is the same):

-atime n

File was last accessed n*24 hours ago. When find figures out how many 24-hour periods ago the file was last accessed, any fractional part is ignored, so to match -atime +1, a file has to have been accessed at least two days ago.>

0
16

If you can make the destination file contents read-only:

find . -type f -exec chmod a-w '{}' \;

before running scp (it will complain and skip the existing files).

And change them back afterward ( chmod +w to get umask based value). If the files do not all have write permission according to your umask, you would somehow have to store the permissions so that you can restore them.

(Gilles' answer overwrites existing files if locally they are newer, I lost valuable data that way. Do not understand why that wrong and harmful answer has so many up votes).

3
  • 2
    I don't get it: how did rsync --ignore-existing cause you to lose data? Nov 27, 2013 at 10:01
  • 4
    I got the error find: missing argument to '-exec' using this command, and instead had to use: find . -type f -exec chmod a-w {} \;. My linux is bad, ymmv.
    – wpearse
    Apr 6, 2015 at 0:16
  • This is a better and safer answer. The correct command to change the permission is as @wpearse mentioned: find . -type f -exec chmod a-w {} \;
    – Amir
    Oct 17, 2017 at 16:41
9

I had a similar task, in my case I could not use rsync, csync, or FUSE because my storage has only SFTP. rsync could not change the date and time for the file, some other utilities (like csync) showed me other errors: "Unable to create temporary file Clock skew detected".

If you have access to the storage-server - just install openssh-server or launch rsync as a daemon here.

In my case - I could not do this and the solution was: lftp. lftp's usage for syncronization is below:

lftp -c "open -u login,password sftp://sft.domain.tld/; \
    mirror -c --verbose=9 -e -R -L /srs/folder /rem/folder"

/src/folder - is the folder on my PC, /rem/folder - is sftp://sft.domain.tld/rem/folder.

You may find man pages by the link: http://lftp.yar.ru/lftp-man.html

2
  • Fabulous! While it's not using scp (as the binary), but sftp (as the same protocol), it helps to achieve a synchronization when sftp is the only protocol available - hence no ssh and therefore no rsync, ls, tar or any of the other proposed solutions. Thank you! Btw, if you want to mirror the remote folder to a local folder, just drop the "-R" flag. Jun 14, 2020 at 10:59
  • Awesome, this worked for me as well. Just some notes. Careful about using -R, that deletes items if they're not present (believe its similar to the --delete flags in rsync). The /rem/folder is a relative path. So if you wanted to copy something to the base path, you'd do: sftp://stf.domain.tld/; ...... /src/folder ../../ Jun 29, 2020 at 1:31
8

To copy a whole bunch of files, it's faster to tar them. By using -k you also prevent tar from overwriting files when unpacking it on the target system.

tar -c <source-dir> | ssh <name>@<host> 'tar -kxzf - -C <target-dir>' 

Note: If your tar -c command creates a POSIX tar archive (GNU), you will have to run the extract like this tar -kxvf - otherwise you'll get error messages like "gzip: stdin: not in gzip format"

1
  • 1
    It does make a remote connection. First it tar's the source, pipes it into the ssh connection and unpacks it on the remote system.
    – huembi
    Aug 22, 2016 at 21:17
0

Another way to achieve this is to do a ls on the destination folder:

On remote destination folder:

ls | awk '{print "mv " $1 " ../copied_data/"}' > mv_copied_data
scp mv_copied_data user@source.server.com:/path/to/source/folder

On source:

cd /path/to/source/folder
chmod 777 mv_copied_data
./mv_copied_data

On destination:

scp -r user@source.server.com:/path/to/source/folder /path/to/destination/foldeer
1
  • 1
    Please don't do this. 1. Don't parse ls. 2. The answer is dangerous if any filename contains - > space newline or other special characters. 3. Hard-code a for-loop in a python script. Generating a program at runtime is hard and tricky (even when done by professional programmers), for a lot of reasons. Generating a shell script is even worse - an awful idea.
    – ignis
    Jun 1, 2019 at 21:12
0

An other option not using rsync (perhaps for portability) is using sftp.

get -a -r FILES will attempt to resume copies and will only bother to copy differences.

This is in case you want to avoid copying duplicate files, but this will not protect files that are different and need to remain different

-1

scp does overwrite files and there's no switch to stop it doing that, but you can copy things out the way, do the scp and then copy the existing files back. Examples:

  1. Copy all existing files out the way

    mkdir original_files ; cp -r * original_files/
    
  2. Copy everything using scp

    scp -r user@server:dir/* ./
    
  3. Copy the original files over anything scp has written over:

    cp -r original_files/* ./
    
1
  • 4
    This method doesn't help when you're trying to pull files over from a remote and pick up where you left off. I.e. if the whole purpose is to save time. Dec 1, 2016 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy