given a file, find where within a subdirectory it was copied to

Question

Given a file that I know has been copied into a given directory, I want to find the exact path where the file copy now resides. It is guaranteed the file has not been edited and will look exactly like the copy.

It wouldn't be necessary to check the entire directory and all its subdirectories, since I know some things about where the file might be. The files have two characteristics: RUN and VERSION, which are known to us and narrow down where the file copy might be.

The solution would likely use diff to compare files, and either grep or find to select the directories that I actually want to look at. I have no idea, however, of how to put it together.

So, we have a file (MYFILE=data.txt) and want to know the path of its copy (for example, Jun-09/15/version3/run1) and we already know, for example, some directories that can be avoided. For example, for some given file we might know that RUN=run1, in which case we should not look at the "run2" directories. Likewise we might know VERSION=version3, in which case we should not look at the version1 or version2 directories. It is worth noting that there is also the possibility of the file not having a copy, in which case I'd like to know that too.

Explanation of what the file structure looks like: The directory in question has one folder for each month for the past 7 years (called, for example, "Jun-09"), and each of these subdirectories has a folder for each day (for example, "11" for the 11th day of the month). Then, each of these "day" folders has a folder for each "version" (the data in question has 3 "versions") and each of these have two "runs." However, despite the fact that the folders are to organize the files time-wise, there is no guarantee the file was created or last edited that month.

MY ATTEMPT: I tried to, from the root directory of the above-described structure, running find . -type f -name data.txt | diff ~/myOtherdirectory/files/data.txt but I keep getting `missing operand after '~/myOtherdirectory/files/data.txt'. What this would do, ideally, is find the differences between the file I'm looking at and all other files that it could be. It does not narrow down the directories to look at at all, and it does not actually get the path of the copy.

Answer 1

I suggest you --depend of your so-- fslint, duff, fdupes, dmerge, rmlint, rdfind --which both are able to find twins much faster than fdupes or dupseek.

I'd execute:

$ find /path -type f -printf "%p - %s\n" | sort -nr -k3 | uniq -D -f1

Answer 2

myfile=/full/path/to/data.txt

mysearchpath=/my/search/path/root/directory

for file in $(find ${mysearchpath} -type f)
do
  diff ${myfile} ${file} > /dev/null 
  result=${?}
  if [ $result -eq 0 ]
  then
    echo "Identical file found at ${file}"
  fi
done

Although this is a very expensive way (computing resources wise, especially if you are sharing this server with other people doing other things) of doing it. You can create checksum of these files and maybe run a job to create checksum of files added on any given day and put them in a flat file somewhere. When you need to find files, create checksum of that file and compare it to your checksum database. Just some food for thought.

Answer 3

If this is something that needs to be checked on a regular basis, you could setup a cron job to create a digest of the md5 hashes of the files, something like

echo > $digest_file; find $search_path -type f | xargs md5sum >> $digest_file

Assuming this has been ran since your file was copied, find the hash of your known file first, then check the list for other files with the same hash. Performance will, of course, depend on the number and size of the files in your search path.

This has the added benefit that, if you copy the digest before each run and compare the old and new digests, you can detect data corruption/changes.

Answer 4

With find and cmp:

find . -type f -exec sh -c 'cmp -s data.txt "$0" 2>/dev/null && echo $0' {} \;

You might want to add some more criteria to limit find.