Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a list of tens of thousands of file names. I want to find which of those files actually exist on disk (in a particular directory). I'm not sure how to start. I could try it with either Python or bash.

The list of file names is an ascii file with one file name per line and no other content.

share|improve this question

4 Answers 4

Let's assume your file names are absolute paths, and all files are in one directory, and you want to find only files (not directories, special files, etc), and you don't have spaces or special characters in your file names.

sort < yourlist >yourlist_sorted 
find <absolute path to dir> -type f |sort |comm -1 -3 - yourlist_sorted

Will print lines of your file which can't be found with the find, ie. which aren't on your disk.

The choice of which files are displayed is controlled by the comm command and the first two options. the comm command, sees two files stdin (list from find) and your list the options control which set are filtered (removed). -1 filters lines only in file 1, -2 only in file 2, -3 lines in both file 1 and 2

So,

  • -2 -3 Prints files that are found on disk and not in your list
  • -1 -2 Prints files that are found on disk and in your list. <== What you want
  • -1 -3 Prints files that are only in your list and not on disk.
share|improve this answer
    
I want to print the files which ARE found –  MountainX Feb 8 at 7:14
    
@MountainX I have added alternative cases. –  X Tian Feb 8 at 12:07

If list.txt contains a list of file names (not paths) and assuming none of the file names contain newline characters:

find . | awk -F/ 'NR==FNR{a[$0];next}; $NF in a' list.txt -
share|improve this answer

I misunderstood your question. To know if the files in the list actually exist, you will need to pass their name to find command. A naive solution would be to iterate over the list and pass each item to the find command:

IFS=''
while read i ; do
   find "$i" 2>/dev/null
   RC=$?
   if [ $RC -eq 0 ] ; then
        echo "file $i exists" >> /tmp/exists.txt
   fi
done < list.txt

This will put all files that exist in the file /tmp/exists.txt.

share|improve this answer
    
This may not work with a list.txt as large as indicated. Nor will it work with spaces or other special characters in the filenames. –  Ricky Beam Feb 8 at 4:38
    
I addressed the space issue. About processing large file: wouldn't it depend on machine resources? –  Ketan Feb 8 at 4:46
    
@RickyBeam this should work with a list of any size, it will just take a long, long time. –  terdon Feb 8 at 15:01
    
It will now, look at the original pre-edit answer... for x in $(cat large-file) won't. (exceeds the shell max cmdline length.) –  Ricky Beam Feb 8 at 21:30

Here's another way, just find all files and pass the result through grep using the -f option to make it read patterns from a file, -w to make it match ojnly if the pattern is an entire "word" and -F so it doesn't treat the patterns as regular expressions:

find /path/to/dir -type -f | grep -wFf list.txt

This also assumes that you have no newlines in your file names.

share|improve this answer
    
nice. looks easy and simple. –  MountainX Feb 8 at 16:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.