Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a list of files that I want to check if they exist on my filesystem. I thought of doing this using find as in:

for f in $(cat file_list); do
find . -name $f > /dev/null || print $f
done

(using zsh) but that doesn't work as find seems to exit 0 whether or not it finds the file. I guess I could pass it through some other test which tests to see if find produces any output (crude but effective would be to replace the > /dev/null with |grep '') but this feels like using a troll to catch a goat (other nationalities might say something about sledgehammers and walnuts).

Is there a way to coerce find in to giving me a useful exit value? Or at least to get a list of those files that weren't found? (I can imagine the latter being perhaps easier by some cunning choice of logical connectives, but I seem to always get tied up in knots when I try to figure it out.)

Background/Motivation: I have a "master" backup and I want to check that some files on my local machine exist on my master backup before deleting them (to create a bit of space). So I made a list of the files, sshed them to the master machine, and was then at a loss for figuring out the best way to find the missing files.

share|improve this question
    
I updated my solution to use the far faster locate. –  user unknown Apr 25 '11 at 22:08
    
@userunknown locate is not showing the current state of the filesystem, it could be a day, or even a week old. That is suitable as a base for testing backups. –  Volker Siegel Sep 26 at 12:56

5 Answers 5

up vote 3 down vote accepted

find considers finding nothing a special case of success (no error occurred). A general way to test whether files match some find criteria is to test whether the output of find is empty. For better efficiency when there are matching files, use -quit on GNU find to make it quit at the first match, or head (head -c 1 if available, otherwise head -n 1 which is standard) on other systems to make it die of a broken pipe rather than produce long output.

while IFS= read -r name; do
  [ -n "$(find . -name "$name" -print | head -n 1)" ] || printf '%s\n' "$name"
done <file_list

In bash ≥4 or zsh, you don't need the external find command for a simple name match: you can use **/$name. Bash version:

shopt -s nullglob
while IFS= read -r name; do
  set -- **/"$name"
  [ $# -ge 1 ] || printf '%s\n' "$name"
done <file_list

Zsh version on a similar principle:

while IFS= read -r name; do
  set -- **/"$name"(N)
  [ $# -ge 1 ] || print -- "$name"
done <file_list

Or here's a shorter but more cryptic way of testing the existence of a file matching a pattern. The glob qualifier N makes the output empty if there is no match, [1] retains only the first match, and e:REPLY=true: changes each match to expand to 1 instead of the matched file name. So **/"$name"(Ne:REPLY=true:[1]) false expands to true false if there is a match, or to just false if there is no match.

while IFS= read -r name; do
  **/"$name"(Ne:REPLY=true:[1]) false || print -- "$name"
done <file_list

It would be more efficient to combine all your names into one search. If the number of patterns is not too large for your system's length limit on a command line, you can join all the names with -o, make a single find call, and post-process the output. If none of the names contain shell metacharacters (so that the names are find patterns as well), here's a way to post-process with awk (untested):

set -o noglob; IFS='
'
set -- $(<file_list sed -e '2,$s/^/-o\
/')
set +o noglob; unset IFS
find . \( "$@" \) -print | awk -F/ '
    BEGIN {while (getline <"file_list") {found[$0]=0}}
    wanted[$0]==0 {found[$0]=1}
    END {for (f in found) {if (found[f]==0) {print f}}}
'

Another approach would be to use Perl and File::Find, which makes it easy to run Perl code for all the files in a directory.

perl -MFile::Find -l -e '
    %missing = map {chomp; $_, 1} <STDIN>;
    find(sub {delete $missing{$_}}, ".");
    print foreach sort keys %missing'

An alternate approach is to generate a list of file names on both sides and work on a text comparison. Zsh version:

comm -23 <(<file_list sort) <(print -rl -- **/*(:t) | sort)
share|improve this answer
    
I'm accepting this one for two reasons. I like the zsh solution with the ** syntax. It's a very simple solution and whilst it may not be the most efficient in terms of the machine, it is probably the most efficient in terms of me actually remembering it! Also, the first solution here answers the actual question in that it twists find into something where the exit code distinguishes "I got a match" from "I didn't get a match". –  Loop Space Apr 26 '11 at 7:43

You could use stat to determine if a file exists on the file system.

You should use the built in shell functions to test if files exist.

while read f; do
   test -f "$f" || echo $f
done < file_list

The "test" is optional and the script will actually work without it, but I left it there for readability.

Edit: If you really have no option but to work for a list of filenames without paths, I suggest you build a list of files once with find, then iterate over it with grep to figure out which files are there.

find -type f /dst > $TMPFILE
while read f; do
    grep -q "/$f$" $TIMPFILE || echo $f
done < file_list

Note that:

  • the file list only includes files not directories,
  • the slash in the grep match pattern is so we compare full file names not partials,
  • and the last '$' in the search pattern is to match the end of the line so you don't get directory matches, only full file name patches.
share|improve this answer
    
stat needs the exact location, does it not? I'm using find because I just have a list of file names and they could be in numerous directories. Sorry if that wasn't clear. –  Loop Space Apr 25 '11 at 19:00
    
Hmmm. Ya you didn't say you had filenames without paths! Maybe you can fix THAT problem instead? It would be way more efficient than running find a bunch of times across the same dataset. –  Caleb Apr 25 '11 at 19:39
    
Thanks for the edit, and sorry again for not being specific. The file name/path is not something that I'm going to fix - files may be in different places on the two systems so I want a solution that is robust enough to work around that. The computer should work to my specifications, not the other way around! Seriously, this isn't something I do often - I was looking for some old files to delete to make space and just wanted a "quick 'n' dirty" way to ensure that they were in my backups. –  Loop Space Apr 25 '11 at 20:28
    
First of all you wouldn't need to full path, just a relative path to whatever directory structure you were backing up. Allow me to suggest that if the path is not the same, there is a good chance the file is not the same and you might get false positives out of your test. It sounds like your solution might be more dirty than quick; I wouldn't want to see you burned by thinking you had something you didn't. Also, if files are valuable enough to backup in the first place, you shouldn't delete the primaries, otherwise you need to backup your backups! –  Caleb Apr 25 '11 at 20:35
    
Ak! I left out a load of details to try to focus the question and you're filling those in with a load of assumptions which - I should say - are perfectly reasonable but happen to be completely wrong! Suffice it to say that I know that if the file is there and is in a directory with a particular type of name then I know that it is the original file and it is safe to delete the copy on my machine. –  Loop Space Apr 25 '11 at 20:45

A first, simplistic approach, could be:

a) sort your filelist:

sort file.lst > sorted.lst 
for f in $(< sortd.lst) ; do find -name $f -printf "%f\n"; done > found.lst
diff sorted.lst found.lst

to find missings, or

comm sorted.lst found.lst

to find matches

  • Pitfalls:
    • Newlines in filenames are very hard to handle
    • blanks and similar things in filenames aren't nice too. But since you have control over the files in the list of files, maybe this solution is already sufficient, however ...
  • Drawbacks:

    • When find finds a file, it keeps running to find another one, and another one. It would be nice to skip further search.
    • find could search for multiple files at once, with some preparation:

      find -name a.file -or -name -b.file -or -name c.file ...

Could locate be an option? Again, a presorted list of files assumed:

 for f in $(< sorted.tmp) ; do locate --regexp "/"$f"$" > /dev/null || echo missing $f ; done

A search for foo.bar will not match a a file foo.ba, or oo.bar with the --regexp-construct (not to be confuesed by regex without p).

You may specify a specific database for locate, and you have to update it before searching, if you need most recent results.

share|improve this answer

I think this can be useful too.

This is a one line solution, in case you opt for your "list" be real files that you want to synchronize with another folder:

function FUNCsync() { local fileCheck="$synchronizeTo/$1"; if [[ ! -f "$fileCheck" ]];then echo "$fileCheck";fi; };export -f FUNCsync;find "$synchronizeFrom/" -maxdepth 1 -type f -not -iname "*~" -exec bash -c 'FUNCsync "{}"' \; |sort

to help reading:

function FUNCsync() {
  local fileCheck="$synchronizeTo/$1";
  if [[ ! -f "$fileCheck" ]];then 
    echo "$fileCheck";
  fi; 
};export -f FUNCsync;
find "$synchronizeFrom/" -maxdepth 1 -type f -not -iname "*~" -exec bash -c 'FUNCsync "{}"' \; |sort

this example excludes backup "*~" files and limits to regular file type "-type f"

share|improve this answer
FIND_EXP=". -type f \( "
while read f; do
   FIND_EXP="${FIND_EXP} -iname $f -or"
done < file_list
FIND_EXP="${var%-or}"
FIND_EXP="${FIND_EXP} \)"
find ${FIND_EXP}

Maybe?

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.