I need to find a list of files in multiple subdirectories and save the path of each file in a txt file. I have all the filenames in a list in filenames.txt. I am trying to do this:

while read -r fname; do
    find . -name "$fname" > paths.txt
done < filenames.txt

Is this correct? It seems to take quite more time than I was expecting, even though I know I have a lot of subdirectories (200k) to search. Are there better ways of doing this? Thanks

  • Filenames may contain newline characters. Can we assume that none of your filenames contain newline characters? – Quasímodo Jun 30 '20 at 22:34
  • @Quasimodo each filename is on a new line. I don't know if this means that there is a newline character after each filename, but I suppose so. How can I check? – ginopino Jun 30 '20 at 22:56
  • That is what I meant, filenames with no newline, but after each filename a newline. (TBH I think it would be impossible to have filenames in a text file if we allowed all the unusual but allowed characters to be part of the filenames.) Thanks for the clarification. – Quasímodo Jun 30 '20 at 23:00

You are running find multiple times, once for each line in filenames.txt - that can be the bottleneck.

You can try to reverse the process - first find all files, then compare it to the files from the list. Something like

find * > allfiles.txt
grep -f filenames.txt allfiles.txt
  • find * will not output any files that start with a . such as .ssh – fpmurphy Jul 1 '20 at 15:40

You could do something like:

find . -print0 | gawk -F/ '
   ARGIND == 1 {selected[$0]; next}
   $NF in selected' filenames.txt RS='\0' -

That is, have find reports all the files, but do the filtering with gawk using a hash table built from the contents of filenames.txt.

That way, you run find only once and do the matching in a more efficient manner.

That assumes filenames.txt is meant to contain a list of literal filenames, not a list of wildcard patterns (find's -name expects a wildcard pattern, not a literal filename).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.