How do I copy an entire directory into a directory of the same name without replacing the content in the destination directory? (instead, I would like to add to the contents of the destination folder)

migrated from stackoverflow.com Jun 2 '11 at 0:50

This question came from our site for professional and enthusiast programmers.

  • Yes, it is annoying when a useful program/utility does not have that one "if only it could ..." option ! In this case "--noclobber" ! – MikeW Feb 7 at 10:57

Use rsync, and pass -u if you want to only update files that are newer in the original directory, or --ignore-existing to skip all files that already exist in the destination.

rsync -au /local/directory/ host:/remote/directory/
rsync -a --ignore-existing /local/directory/ host:/remote/directory/

(Note the / on the source side: without it rsync would create /remote/directory/directory.)

  • @Anthon I don't understand your comment and I don't see an answer or comment by chandra. --ignore-existing does add without replacing, what data loss do you see? – Gilles Nov 27 '13 at 9:59
  • Sorry, I only looked at your first example that is where you can have data loss (and is IMHO not what the OP asked for), if you include --ignore-existing data-loss should not happen. – Anthon Nov 27 '13 at 10:08
  • This does not help if the remote system does not have rsync easily available.... (Like Win32-OpenSSH) – Gert van den Berg Oct 25 '16 at 8:00
  • @GertvandenBerg rsync is pretty easy to install on Windows, no harder than SSH. – Gilles Oct 25 '16 at 11:51
  • @Gilles: True, but all of the options seems to involve Cygwin DLLs... (The current state of the MS port of OpenSSH is such that enabling compression on scp is enough to break SCP...) (Getting rsync functional over Win32-OpenSSH also seems non-trivial - hopefully that improves over time) (Solaris 10 is the other example, where a third party package and --rsync-path is needed) – Gert van den Berg Oct 25 '16 at 13:01

scp will overwrite the files if you have write permissions to them.

In other words: You can make scp effectively skip said files by temporarily removing the write permissions on them (if you are the files' owner, that is).

  • 5
    Thanks for this. Was exactly the trick I was looking for. – saccharine Jul 16 '13 at 21:02
  • make sure you copy the files back you add a * to do so. Example scp -r user@server.com:/location/of/files/* /local/location/ – Rick May 27 '15 at 19:16
  • 1
    find . -type f -exec chown root:root {} \; – ling Aug 21 '16 at 19:58
  • In the uncommon case that only some of the files in the directory are expected to be overwritten (if not "protected"), and file permissions are not uniform across files, this solution may not work. Otherwise, it is quite simple and effective. – sancho.s May 20 at 12:55
  • To do this on windows select all the files, right click -> properties -> read only – BenJammin Sep 6 at 15:17

You can copy only new files by date. Use find

scp  `find /data/*.gz -type f -mtime +7` USER@SERVER:/backup/
  • what does +7 mean? – Francis Kim May 1 '17 at 13:07
  • @FrancisKim it means modified 7 days ago. So this query select any *.gz files modified within last 7 days – Vadim Eremeev May 24 '17 at 21:34
  • 3
    Doesn't +7 actually imply more than 7 days ago? – Jeff H. Mar 21 at 22:51
  • Yes it means "more than 7 days ago". No idea who upvoted this. – Stefan Reich 22 hours ago

If you can make the destination file contents read-only:

find . -type f -exec chmod a-w '{}' \;

before running scp (it will complain and skip the existing files).

And change them back afterward ( chmod +w to get umask based value). If the files do not all have write permission according to your umask, you would somehow have to store the permissions so that you can restore them.

(Gilles' answer overwrites existing files if locally they are newer, I lost valuable data that way. Do not understand why that wrong and harmful answer has so many up votes).

  • 2
    I don't get it: how did rsync --ignore-existing cause you to lose data? – Gilles Nov 27 '13 at 10:01
  • 3
    I got the error find: missing argument to '-exec' using this command, and instead had to use: find . -type f -exec chmod a-w {} \;. My linux is bad, ymmv. – wpearse Apr 6 '15 at 0:16
  • This is a better and safer answer. The correct command to change the permission is as @wpearse mentioned: find . -type f -exec chmod a-w {} \; – Amir Oct 17 '17 at 16:41

I had a similar task, in my case I could not use rsync, csync, or FUSE because my storage has only SFTP. rsync could not change the date and time for the file, some other utilities (like csync) showed me other errors: "Unable to create temporary file Clock skew detected".

If you have access to the storage-server - just install openssh-server or launch rsync as a daemon here.

In my case - I could not do this and the solution was: lftp. lftp's usage for syncronization is below:

lftp -c "open -u login,password sftp://sft.domain.tld/; \
    mirror -c --verbose=9 -e -R -L /srs/folder /rem/folder"

/src/folder - is the folder on my PC, /rem/folder - is sftp://sft.domain.tld/rem/folder.

You may find man pages by the link: http://lftp.yar.ru/lftp-man.html

To copy a whole bunch of files, it's faster to tar them. By using -k you also prevent tar from overwriting files when unpacking it on the target system.

tar -c <source-dir> | ssh <name>@<host> 'tar -kxzf - -C <target-dir>' 
  • 1
    It does make a remote connection. First it tar's the source, pipes it into the ssh connection and unpacks it on the remote system. – huembi Aug 22 '16 at 21:17

scp does overwrite files and there's no switch to stop it doing that, but you can copy things out the way, do the scp and then copy the existing files back. Examples:

  1. Copy all existing files out the way

    mkdir original_files ; cp -r * original_files/
    
  2. Copy everything using scp

    scp -r user@server:dir/* ./
    
  3. Copy the original files over anything scp has written over:

    cp -r original_files/* ./
    
  • 3
    This method doesn't help when you're trying to pull files over from a remote and pick up where you left off. I.e. if the whole purpose is to save time. – Oliver Williams Dec 1 '16 at 17:58

Another way to achieve this is to do a ls on the destination folder:

On remote destination folder: 1.ls | awk '{print "mv " $1 " ../copied_data/"}' > mv_copied_data

2.scp mv_copied_data user@source.server.com:/path/to/source/folder

On source: 3. cd /path/to/source/folder

  1. chmod 777 mv_copied_data
  2. ./mv_copied_data

On destination: scp -r user@source.server.com:/path/to/source/folder /path/to/destination/foldeer

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.