How do I find which files are missing from a list?

Question

I have a list of files that I want to check if they exist on my filesystem. I thought of doing this using find as in:

for f in $(cat file_list); do
find . -name $f > /dev/null || print $f
done

(using zsh) but that doesn't work as find seems to exit 0 whether or not it finds the file. I guess I could pass it through some other test which tests to see if find produces any output (crude but effective would be to replace the > /dev/null with |grep '') but this feels like using a troll to catch a goat (other nationalities might say something about sledgehammers and walnuts).

Is there a way to coerce find in to giving me a useful exit value? Or at least to get a list of those files that weren't found? (I can imagine the latter being perhaps easier by some cunning choice of logical connectives, but I seem to always get tied up in knots when I try to figure it out.)

Background/Motivation: I have a "master" backup and I want to check that some files on my local machine exist on my master backup before deleting them (to create a bit of space). So I made a list of the files, sshed them to the master machine, and was then at a loss for figuring out the best way to find the missing files.

Answer 1

You could use stat to determine if a file exists on the file system.

You should use the built in shell functions to test if files exist.

while read f; do
   test -f "$f" || echo $f
done < file_list

The "test" is optional and the script will actually work without it, but I left it there for readability.

Edit: If you really have no option but to work for a list of filenames without paths, I suggest you build a list of files once with find, then iterate over it with grep to figure out which files are there.

find -type f /dst > $TMPFILE
while read f; do
    grep -q "/$f$" $TIMPFILE || echo $f
done < file_list

Note that:

• the file list only includes files not directories,
• the slash in the grep match pattern is so we compare full file names not partials,
• and the last '$' in the search pattern is to match the end of the line so you don't get directory matches, only full file name patches.

Answer 2

find considers finding nothing a special case of success (no error occurred). A general way to test whether files match some find criteria is to test whether the output of find is empty. For better efficiency when there are matching files, use -quit on GNU find to make it quit at the first match, or head (head -c 1 if available, otherwise head -n 1 which is standard) on other systems to make it die of a broken pipe rather than produce long output.

while IFS= read -r name; do
  [ -n "$(find . -name "$name" -print | head -n 1)" ] || printf '%s\n' "$name"
done <file_list

In bash ≥4 or zsh, you don't need the external find command for a simple name match: you can use **/$name. Bash version:

shopt -s nullglob
while IFS= read -r name; do
  set -- **/"$name"
  [ $# -ge 1 ] || printf '%s\n' "$name"
done <file_list

Zsh version on a similar principle:

while IFS= read -r name; do
  set -- **/"$name"(N)
  [ $# -ge 1 ] || print -- "$name"
done <file_list

Or here's a shorter but more cryptic way of testing the existence of a file matching a pattern. The glob qualifier N makes the output empty if there is no match, [1] retains only the first match, and e:REPLY=true: changes each match to expand to 1 instead of the matched file name. So **/"$name"(Ne:REPLY=true:[1]) false expands to true false if there is a match, or to just false if there is no match.

while IFS= read -r name; do
  **/"$name"(Ne:REPLY=true:[1]) false || print -- "$name"
done <file_list

It would be more efficient to combine all your names into one search. If the number of patterns is not too large for your system's length limit on a command line, you can join all the names with -o, make a single find call, and post-process the output. If none of the names contain shell metacharacters (so that the names are find patterns as well), here's a way to post-process with awk (untested):

set -o noglob; IFS='
'
set -- $(<file_list sed -e '2,$s/^/-o\
/')
set +o noglob; unset IFS
find . \( "$@" \) -print | awk -F/ '
    BEGIN {while (getline <"file_list") {found[$0]=0}}
    wanted[$0]==0 {found[$0]=1}
    END {for (f in found) {if (found[f]==0) {print f}}}
'

Another approach would be to use Perl and File::Find, which makes it easy to run Perl code for all the files in a directory.

perl -MFile::Find -l -e '
    %missing = map {chomp; $_, 1} <STDIN>;
    find(sub {delete $missing{$_}}, ".");
    print foreach sort keys %missing'

An alternate approach is to generate a list of file names on both sides and work on a text comparison. Zsh version:

comm -23 <(<file_list sort) <(print -rl -- **/*(:t) | sort)

Answer 3

A first, simplistic approach, could be:

a) sort your filelist:

sort file.lst > sorted.lst 
for f in $(< sortd.lst) ; do find -name $f -printf "%f\n"; done > found.lst
diff sorted.lst found.lst

to find missings, or

comm sorted.lst found.lst

to find matches

• Pitfalls:
  • Newlines in filenames are very hard to handle
  • blanks and similar things in filenames aren't nice too. But since you have control over the files in the list of files, maybe this solution is already sufficient, however ...

• Drawbacks:

  • When find finds a file, it keeps running to find another one, and another one. It would be nice to skip further search.

  • find could search for multiple files at once, with some preparation:

    find -name a.file -or -name -b.file -or -name c.file ...

Could locate be an option? Again, a presorted list of files assumed:

 for f in $(< sorted.tmp) ; do locate --regexp "/"$f"$" > /dev/null || echo missing $f ; done

A search for foo.bar will not match a a file foo.ba, or oo.bar with the --regexp-construct (not to be confuesed by regex without p).

You may specify a specific database for locate, and you have to update it before searching, if you need most recent results.

Answer 4

I think this can be useful too.

This is a one line solution, in case you opt for your "list" be real files that you want to synchronize with another folder:

function FUNCsync() { local fileCheck="$synchronizeTo/$1"; if [[ ! -f "$fileCheck" ]];then echo "$fileCheck";fi; };export -f FUNCsync;find "$synchronizeFrom/" -maxdepth 1 -type f -not -iname "*~" -exec bash -c 'FUNCsync "{}"' \; |sort

to help reading:

function FUNCsync() {
  local fileCheck="$synchronizeTo/$1";
  if [[ ! -f "$fileCheck" ]];then 
    echo "$fileCheck";
  fi; 
};export -f FUNCsync;
find "$synchronizeFrom/" -maxdepth 1 -type f -not -iname "*~" -exec bash -c 'FUNCsync "{}"' \; |sort

this example excludes backup "*~" files and limits to regular file type "-type f"

Answer 5

FIND_EXP=". -type f \( "
while read f; do
   FIND_EXP="${FIND_EXP} -iname $f -or"
done < file_list
FIND_EXP="${var%-or}"
FIND_EXP="${FIND_EXP} \)"
find ${FIND_EXP}

Maybe?

Answer 6

Why not simply comparing the length of the query list with the length of the result list?

while read p; do
  find . -name $p 2>/dev/null
done < file_list.txt | wc -l
wc -l file_list.txt