Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I would like to copy a set of files from directory A to directory B, with the caveat that if a file in directory A is identical to a file in directory B, that file should not be copied (and thus its modification time should not be updated). Is there a way to do that with existing tools, without writing my own script to do it?

To elaborate a bit on my use-case: I am autogenerating a bunch of .c files in a temporary directory (by a method that has to generate all of them unconditionally), and when I re-generate them, I'd like to copy only the ones that have changed into the actual source directory, leaving the unchanged ones untouched (with their old creation times) so that make will know that it doesn't need to recompile them. (Not all the generated files are .c files, though, so I need to do binary comparisons rather than text comparisons.)

(As a note: This grew out of the question I asked on http://stackoverflow.com/questions/8981552/speeding-up-file-comparions-with-cmp-on-cygwin/8981762#8981762, where I was trying to speed up the script file I was using to do this operation, but it occurs to me that I really should ask if there's a a better way to do this than writing my own script -- especially since any simple way of doing this in a shell script will invoke something like cmp on every pair of files, and starting all those processes takes too long.)

share|improve this question
1  
You can use diff -qr dirA dirB to see what files are unique to dirA and dirB, repectively. –  Herman Torjussen Jan 24 '12 at 8:38
    
@brooks-moses this is really a job suited for ccache! –  aculich Jan 31 '12 at 0:02
1  
@hesse if you want to show the unique files you can use diff, but if you want to see just what has changed then use rsync -avnc or the long way rsync --archive --verbose --dry-run --checksum. –  aculich Jan 31 '12 at 0:04
add comment

4 Answers

up vote 17 down vote accepted

rsync is probably the best tool for this. There are a lot of options on this command so read man page. I think you want the --checksum option or the --ignore-times

share|improve this answer
    
I should have noted that I already tried that, with no success. Both of those options only affect whether rsync does a copy -- but, even when it does not do a copy, it either updates the target file's modification time to the same as the source (if the -t option is specified) or to the synchronization time (if -t is not specified). –  Brooks Moses Jan 24 '12 at 5:52
3  
@Brooks Moses: It doesn't. At least my version of rsync doesn't. If I do this: mkdir src dest; echo a>src/a; rsync -c src/* dest; sleep 5; touch src/a; rsync -c src/* dest, then stat dest/a shows its mtime and ctime are 5 secs older than the ones of src/a. –  angus Jan 24 '12 at 8:48
    
@angus: Huh. Okay, you're right. The key seems to be the --checksum option, and although linux.die.net/man/1/rsync contains absolutely nothing that would imply that it has any affect on whether the modification date is updated, it nonetheless causes the destination modification date to be left untouched. (On the other hand, the --ignore-times option does not have this effect; with it the modification date is still updated.) Given that this seems to be entirely undocumented, though, can I rely on it? –  Brooks Moses Jan 24 '12 at 9:32
2  
@BrooksMoses: I think you can rely on it: rsync's workflow is: 1) check if the file need to be updated; 2) if so, update the file. The --checksum option say it should not be updated, so rsync should not proceed to step 2). –  enzotib Jan 24 '12 at 10:25
1  
@BrooksMoses: --ignore-times without --checksum would copy every file, and so also update the timestamp, even if the files are identical. –  enzotib Jan 24 '12 at 21:22
show 2 more comments

I like to use unison in favor of rsync because it supports multiple masters, having already setup my ssh keys and vpn separately.

So in my crontab of only one host I let them synchronize every 15 minutes:

*/15 * * * * [ -z "$(pidof unison)" ] && (timeout 25m unison -sortbysize -ui text -batch -times /home/master ssh://192.168.1.12//home/master -path dev -logfile /tmp/sync.master.dev.log) &> /tmp/sync.master.dev.log

Then I can be developing on either side and the changes will propagate. In fact for important projects I have up to 4 servers mirroring the same tree (3 run unison from cron, pointing to the one that doesn't). In fact, Linux and Cygwin hosts mixed--except don't expect sense out of soft links in win32 outside the cygwin environment.

If you go this route, make the initial mirror on the empty side without the -batch, i.e.

unison -ui text  -times /home/master ssh://192.168.1.12//home/master -path dev

Of course there is a config to ignore backup files, archives, etc.:

 ~/.unison/default.prf :
# Unison preferences file
ignore = Name {,.}*{.sh~}
ignore = Name {,.}*{.rb~}
ignore = Name {,.}*{.bak}
ignore = Name {,.}*{.tmp}
ignore = Name {,.}*{.txt~}
ignore = Name {,.}*{.pl~}
ignore = Name {.unison.}*
ignore = Name {,.}*{.zip}

    # Use this command for displaying diffs
    diff = diff -y -W 79 --suppress-common-lines

    ignore = Name *~
    ignore = Name .*~
    ignore = Path */pilot/backup/Archive_*
    ignore = Name *.o
share|improve this answer
    
I looked at that, but I couldn't find a unison option that means "don't update file-last-modified dates". Is there one? Otherwise, this is a great answer to an entirely different problem. –  Brooks Moses Feb 2 '12 at 20:46
1  
-times does that for me. Unison has a dry-run mode too, me thinks. –  Marcos Feb 2 '12 at 22:29
    
Well, setting times=false (or leaving off -times) would do that. I don't know how I missed that in the documentation before. Thanks! –  Brooks Moses Feb 2 '12 at 23:56
    
Glad to help. I'm a stickler when it comes to preserving things like modtimes, permissions and soft links. Often overlooked –  Marcos Feb 3 '12 at 16:44
add comment

While using rsync --checksum is a good general way to "copy if changed", in your particular case there is an even better solution!

If you want to avoid unnecessarily recompiling files you should use ccache which was built for exactly this purpose! In fact, not only will it avoid unnecessary recompiles of your auto-generated files, it will also speed things up whenever you do make clean and re-compile from scratch.

Next I'm sure you'll ask, "Is it safe?" Well, yes, as the website points out:

Is it safe?

Yes. The most important aspect of a compiler cache is to always produce exactly the same output that the real compiler would produce. This includes providing exactly the same object files and exactly the same compiler warnings that would be produced if you use the real compiler. The only way you should be able to tell that you are using ccache is the speed.

And it's easy to use it by just adding it as a prefix in the CC= line of your makefile (or you can use symlinks, but the makefile way is probably better).

share|improve this answer
    
I initially misunderstood and thought you were suggesting I use ccache to do part of the generating, but now I understand -- your suggestion was that I simply copy all the files, and then use ccache in the build process, thereby avoiding rebuilding the ones that hadn't changed. It's a good idea, but it won't do well in my case -- I have hundreds of files, usually only change one or two at a time, and am running under Cygwin where simply starting the hundreds of ccache processes to look at each file would take several minutes. Nonetheless, upvoted because it's a good answer for most people! –  Brooks Moses Jan 31 '12 at 6:14
    
No, I was not suggesting that you copy all the files, rather you can just autogenerate your .c files in-place (remove the copy step and write to them directly). And then just use ccache. I don't know what you mean by starting hundreds of ccache processes... it is just a light-weight wrapper around gcc that is quite fast and will speed up re-building other parts of your project, too. Have you tried using it? I would like to see a comparison of the timing between using your copy-method vs ccache. You could, in fact, combine the two methods to get the benefits of both. –  aculich Jan 31 '12 at 16:07
    
Right, ok, I understand now about the copying. To clarify, what I mean is this: If I generate the files in place, I have to then call ccache file.c -o file.o or the equivalent, several hundreds of times because there are several hundred file.c files. When I was doing that with cmp, rather than ccache, it took several minutes -- and cmp is as lightweight as ccache. The problem is that, on Cygwin, starting a process takes non-negligible time, even for a completely trivial process. –  Brooks Moses Feb 1 '12 at 3:57
    
Cygwin is all kinds of problematic, so that doesn't surprise me. I am curious, though, what keeps you tied to Cygwin? Seems to me it would be easier to spin up a Linux VM with VirtualBox and do the primary development there, but then test it on Cygwin when you need to... you can still have Cygwin as a target platform (though with VM tech what it is, I don't see why Cygwin is still around anymore), but develop elsewhere. –  aculich Feb 1 '12 at 4:35
    
As a datapoint, for f in src/*; do /bin/true.exe; done takes 30 seconds, so yeah. Anyway, I prefer my Windows-based editor, and aside from this sort of timing issue Cygwin works quite well with my workflow as the lightweight place to test things locally if I'm not uploading to the build servers. It's useful to have my shell and my editor in the same OS. :) –  Brooks Moses Feb 1 '12 at 4:43
show 1 more comment

This should do what you need

diff -qr ./x ./y | awk '{print $2}' | xargs -n1 -J% cp % ./y/

Where:

  • x is your updated/new folder
  • y is the destination you want to copy to
  • awk will take the second argument of the each line from the diff command (maybe you will need some extra stuff for filenames with space - can't try it now)
  • xargs -J% will insert the file name to cp at the proper place
share|improve this answer
    
-1 because this is overly-complicated, non-portable (-J is bsd-specific; with GNU xargs it is -I), and does not work correctly if the same set of files do not exist in both locations already (if I touch x/boo then grep gives me Only in ./x: boo which causes errors in the pipeline). Use a tool built for the job, like rsync --checksum. –  aculich Jan 30 '12 at 23:24
    
Or better yet, for this specific case use ccache. –  aculich Jan 31 '12 at 0:06
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.