I would like to copy a set of files from directory A to directory B, with the caveat that if a file in directory A is identical to a file in directory B, that file should not be copied (and thus its modification time should not be updated). Is there a way to do that with existing tools, without writing my own script to do it?

To elaborate a bit on my use-case: I am autogenerating a bunch of .c files in a temporary directory (by a method that has to generate all of them unconditionally), and when I re-generate them, I'd like to copy only the ones that have changed into the actual source directory, leaving the unchanged ones untouched (with their old creation times) so that make will know that it doesn't need to recompile them. (Not all the generated files are .c files, though, so I need to do binary comparisons rather than text comparisons.)

(As a note: This grew out of the question I asked on https://stackoverflow.com/questions/8981552/speeding-up-file-comparions-with-cmp-on-cygwin/8981762#8981762, where I was trying to speed up the script file I was using to do this operation, but it occurs to me that I really should ask if there's a a better way to do this than writing my own script -- especially since any simple way of doing this in a shell script will invoke something like cmp on every pair of files, and starting all those processes takes too long.)

  • 1
    You can use diff -qr dirA dirB to see what files are unique to dirA and dirB, repectively. – user13742 Jan 24 '12 at 8:38
  • 1
    @brooks-moses this is really a job suited for ccache! – aculich Jan 31 '12 at 0:02
  • 3
    @hesse if you want to show the unique files you can use diff, but if you want to see just what has changed then use rsync -avnc or the long way rsync --archive --verbose --dry-run --checksum. – aculich Jan 31 '12 at 0:04

rsync is probably the best tool for this. There are a lot of options on this command so read man page. I think you want the --checksum option or the --ignore-times

| improve this answer | |
  • I should have noted that I already tried that, with no success. Both of those options only affect whether rsync does a copy -- but, even when it does not do a copy, it either updates the target file's modification time to the same as the source (if the -t option is specified) or to the synchronization time (if -t is not specified). – Brooks Moses Jan 24 '12 at 5:52
  • 4
    @Brooks Moses: It doesn't. At least my version of rsync doesn't. If I do this: mkdir src dest; echo a>src/a; rsync -c src/* dest; sleep 5; touch src/a; rsync -c src/* dest, then stat dest/a shows its mtime and ctime are 5 secs older than the ones of src/a. – angus Jan 24 '12 at 8:48
  • 2
    @BrooksMoses: I think you can rely on it: rsync's workflow is: 1) check if the file need to be updated; 2) if so, update the file. The --checksum option say it should not be updated, so rsync should not proceed to step 2). – enzotib Jan 24 '12 at 10:25
  • 2
    @BrooksMoses: --ignore-times without --checksum would copy every file, and so also update the timestamp, even if the files are identical. – enzotib Jan 24 '12 at 21:22
  • 1
    This answer's high rating is unwarranted. Saying "read the man page" is barely more helpful than saying "go google the answer". – Dennis Aug 22 '19 at 17:22

You can use the -u switch to cp like so:

$ cp -u [source] [destination]

From the man page:

   -u, --update
       copy only when the SOURCE file is newer than the destination file or 
       when the destination file is missing
| improve this answer | |
  • 4
    Hi and welcome to the site. We expect answers to be a bit more substantial here. For example, you could have included an explanation of what the -u flag does and how it works and how this would help the OP. However, in this particular case, it wouldn't help the OP since it would copy identical files if they were newer and so change their time stamps which is precisely what the OP wants to avoid. – terdon Jun 27 '14 at 17:37
  • 1
    From a comment on a similar A that was already deleted: "This will not work since it would copy also identical files, if timestamp of source is newer (and so update timestamp of destination, against the OP request)." – slm Jun 27 '14 at 17:48
  • Doesn't answer the question at all, but I still found it useful. – user31389 Oct 21 '16 at 11:19

While using rsync --checksum is a good general way to "copy if changed", in your particular case there is an even better solution!

If you want to avoid unnecessarily recompiling files you should use ccache which was built for exactly this purpose! In fact, not only will it avoid unnecessary recompiles of your auto-generated files, it will also speed things up whenever you do make clean and re-compile from scratch.

Next I'm sure you'll ask, "Is it safe?" Well, yes, as the website points out:

Is it safe?

Yes. The most important aspect of a compiler cache is to always produce exactly the same output that the real compiler would produce. This includes providing exactly the same object files and exactly the same compiler warnings that would be produced if you use the real compiler. The only way you should be able to tell that you are using ccache is the speed.

And it's easy to use it by just adding it as a prefix in the CC= line of your makefile (or you can use symlinks, but the makefile way is probably better).

| improve this answer | |
  • 1
    I initially misunderstood and thought you were suggesting I use ccache to do part of the generating, but now I understand -- your suggestion was that I simply copy all the files, and then use ccache in the build process, thereby avoiding rebuilding the ones that hadn't changed. It's a good idea, but it won't do well in my case -- I have hundreds of files, usually only change one or two at a time, and am running under Cygwin where simply starting the hundreds of ccache processes to look at each file would take several minutes. Nonetheless, upvoted because it's a good answer for most people! – Brooks Moses Jan 31 '12 at 6:14
  • No, I was not suggesting that you copy all the files, rather you can just autogenerate your .c files in-place (remove the copy step and write to them directly). And then just use ccache. I don't know what you mean by starting hundreds of ccache processes... it is just a light-weight wrapper around gcc that is quite fast and will speed up re-building other parts of your project, too. Have you tried using it? I would like to see a comparison of the timing between using your copy-method vs ccache. You could, in fact, combine the two methods to get the benefits of both. – aculich Jan 31 '12 at 16:07
  • 1
    Right, ok, I understand now about the copying. To clarify, what I mean is this: If I generate the files in place, I have to then call ccache file.c -o file.o or the equivalent, several hundreds of times because there are several hundred file.c files. When I was doing that with cmp, rather than ccache, it took several minutes -- and cmp is as lightweight as ccache. The problem is that, on Cygwin, starting a process takes non-negligible time, even for a completely trivial process. – Brooks Moses Feb 1 '12 at 3:57
  • 1
    As a datapoint, for f in src/*; do /bin/true.exe; done takes 30 seconds, so yeah. Anyway, I prefer my Windows-based editor, and aside from this sort of timing issue Cygwin works quite well with my workflow as the lightweight place to test things locally if I'm not uploading to the build servers. It's useful to have my shell and my editor in the same OS. :) – Brooks Moses Feb 1 '12 at 4:43
  • 1
    If you want to use your Windows-based editor you can do that quite easily with Shared Folders if you install Guest Additions... but hey, if Cygwin suits you, then who am I to say any different? It just seems a shame to have to jump through weird hoops like this... and compilation in general would be faster in a VM, too. – aculich Feb 1 '12 at 4:55

This should do what you need

diff -qr ./x ./y | awk '{print $2}' | xargs -n1 -J% cp % ./y/


  • x is your updated/new folder
  • y is the destination you want to copy to
  • awk will take the second argument of the each line from the diff command (maybe you will need some extra stuff for filenames with space - can't try it now)
  • xargs -J% will insert the file name to cp at the proper place
| improve this answer | |
  • 1
    -1 because this is overly-complicated, non-portable (-J is bsd-specific; with GNU xargs it is -I), and does not work correctly if the same set of files do not exist in both locations already (if I touch x/boo then grep gives me Only in ./x: boo which causes errors in the pipeline). Use a tool built for the job, like rsync --checksum. – aculich Jan 30 '12 at 23:24
  • Or better yet, for this specific case use ccache. – aculich Jan 31 '12 at 0:06
  • +1 because its a set of well known commands that I can break to use on similar tasks (came here for doing a diff), still rsync may be better for this particular task – ntg Nov 20 '17 at 8:22

I like to use unison in favor of rsync because it supports multiple masters, having already setup my ssh keys and vpn separately.

So in my crontab of only one host I let them synchronize every 15 minutes:

*/15 * * * * [ -z "$(pidof unison)" ] && (timeout 25m unison -sortbysize -ui text -batch -times /home/master ssh:// -path dev -logfile /tmp/sync.master.dev.log) &> /tmp/sync.master.dev.log

Then I can be developing on either side and the changes will propagate. In fact for important projects I have up to 4 servers mirroring the same tree (3 run unison from cron, pointing to the one that doesn't). In fact, Linux and Cygwin hosts mixed--except don't expect sense out of soft links in win32 outside the cygwin environment.

If you go this route, make the initial mirror on the empty side without the -batch, i.e.

unison -ui text  -times /home/master ssh:// -path dev

Of course there is a config to ignore backup files, archives, etc.:

 ~/.unison/default.prf :
# Unison preferences file
ignore = Name {,.}*{.sh~}
ignore = Name {,.}*{.rb~}
ignore = Name {,.}*{.bak}
ignore = Name {,.}*{.tmp}
ignore = Name {,.}*{.txt~}
ignore = Name {,.}*{.pl~}
ignore = Name {.unison.}*
ignore = Name {,.}*{.zip}

    # Use this command for displaying diffs
    diff = diff -y -W 79 --suppress-common-lines

    ignore = Name *~
    ignore = Name .*~
    ignore = Path */pilot/backup/Archive_*
    ignore = Name *.o
| improve this answer | |
  • I looked at that, but I couldn't find a unison option that means "don't update file-last-modified dates". Is there one? Otherwise, this is a great answer to an entirely different problem. – Brooks Moses Feb 2 '12 at 20:46
  • 1
    -times does that for me. Unison has a dry-run mode too, me thinks. – Marcos Feb 2 '12 at 22:29
  • Well, setting times=false (or leaving off -times) would do that. I don't know how I missed that in the documentation before. Thanks! – Brooks Moses Feb 2 '12 at 23:56
  • Glad to help. I'm a stickler when it comes to preserving things like modtimes, permissions and soft links. Often overlooked – Marcos Feb 3 '12 at 16:44

While rsync --checksum is the correct answer, note that this option is incompatible with --times, and that --archive includes --times, so if you want to rsync -a --checksum, you really need to rsync -a --no-times --checksum.

| improve this answer | |
  • What do you mean by saying 'incompatible'? – o.v Jan 10 '19 at 12:44
  • What do you mean by "is the correct answer"? – thoni56 Feb 3 '19 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.