find: How to efficiently search for big list of filenames

Question

I need to find a few hundred files, where the base names are provided by some list (let's call it baseNames). I then need to search for these base names + three given extensions.

Example: Assume one of the base names extracted from the input list is FOO and the given extensions are .txt, .csv, .py. Then I need to find FOO.txt, FOO.csv, FOO.py.

The current approach in my bash script is as follows:

for bn in ${baseNames}; do
  find ${searchDir} '(' -name "$bn.txt" -o -name "$bn.csv" -o -name "$bn.py" ')'
done

This works but is quite inefficient. For every base name I need to run find on the whole searchDir again, which contains quite a lot of files and hence takes a while.

Is there a way to provide a list of files that find should search for, via options or pipes?

Obviously I am aware of -name ... -or, but it's also obvious that this approach is unrealistic if I have a couple of hundred files. For simplicity, you may also leave aside the extensions. Just assume I have a huge list of files that I want to search for as input to find.

Answer 1

Use an array. e.g.

#!/bin/bash

baseNames=(FOO BAR BAZ)

findNames=('(')
for bn in "${baseNames[@]}"; do
  for ext in txt csv py; do
    findNames+=("$bn.$ext" '-o' '-name')
  done
done
# replace the final '-o' and '-name' in the array with a close parenthesis
unset 'findNames[-1]'
findNames[-1]=')'
# If using a version of bash before v4.3, use:
#unset 'findNames[${#findNames[@]}-1]'
#findNames[${#findNames[@]}-1]=')'


declare -p findNames

Output from the declare -p is (some line feeds and spaces added to break it up and make it more readable):

declare -a findNames=(
  [0]="("
    [1]="-name" [2]="FOO.txt" [3]="-o" [4]="-name" [5]="FOO.csv"
    [6]="-o" [7]="-name" [8]="FOO.py" [9]="-o" [10]="-name" [11]="BAR.txt"
    [12]="-o" [13]="-name" [14]="BAR.csv" [15]="-o" [16]="-name" [17]="BAR.py"
    [18]="-o" [19]="-name" [20]="BAZ.txt" [21]="-o" [22]="-name" [23]="BAZ.csv"
    [24]="-o" [25]="-name" [26]="BAZ.py"
  [27]=")"
)

---

To use the array with find, you'd do something like:

searchDir="./"
find "$searchDir" "${findNames[@]}"

This would result in the following find command being executed (line breaks added for readability):

find ./ ( -name FOO.txt -o -name FOO.csv -o -name FOO.py \
  -o -name BAR.txt -o -name BAR.csv -o -name BAR.py \
  -o -name BAZ.txt -o -name BAZ.csv -o -name BAZ.py )

The ( and ) don't need to be escaped here because the shell treats them as just literal arguments (the array has already been expanded by bash), not an instruction to start a sub-shell. If you were typing them into a shell, you'd have to escape or quote them.

Answer 2

The following sh script reads base names from the file names, which contains one name per line (names should be quoted if they contain spaces etc.), and calls an in-line sh -c script with batches of these (50 at a time). I'm dividing the input into batches, just in case the data expands to a list that is too long for a single invocation of find (we need to construct commands with a total combined length of more than n time the length of the input data, where n is the number of filename suffixes to look for).

The in-line script constructs an "OR list" of -name tests based on the given base names for find. Each base name is entered into the list with variations for the three filenames suffixes .txt, .csv, and .py.

The list is maintained in the list of positional parameters, "$@".

Once the list is complete, find is called to find the regular files matching these names, in or under some directory $topdir.

topdir=$HOME

<names xargs -L 50 sh -c '
        topdir=$1; shift

        for name do
                for suffix in txt csv py; do
                        set -- "$@" -o -name "$name.$suffix"
                done
                shift  # shift off current base name
        done
        shift  # shift off the initial "-o"

        find "$topdir" -type f \( "$@" \) -print
' sh "$topdir"

Run with a lower number than 50 and with sh -x -c in place of sh -c to see what commands the in-line script is actually executing.

---

If you want to use named arrays and the bash shell:

topdir=$HOME

<names xargs -L 50 bash -c '
        topdir=$1; shift
        unset tests

        for name do
                for suffix in txt csv py; do
                        tests+=( -o -name "$name.$suffix" )
                done
        done

        find "$topdir" -type f \( "${tests[@]:1}" \) -print
' bash "$topdir"

Here, an array, tests, is used instead of the list of positional parameters. The strange-looking "${tests[@]:1}" expands to the list of elements of the array, except for the first element (which will be -o).

Although, if you're using bash, you may just as well use its globbing facilities (originally inherited from the ksh shell):

shopt -s extglob globstar dotglob nullglob

topdir=$HOME

printf -v pattern '%s/**/@(%s).@(txt|csv|py)' "$topdir" "$(paste -s -d '|' - <names)"

eval "pathnames=( $pattern )"

# The following loop is only for illustration.
# If you really just wanted to list the names, use
#     printf '%s\n' "${pathnames[@]}"

for pathname in "${pathnames[@]}"; do
        printf '%s\n' "$pathname"
done

This constructs an extended globbing pattern from the contents of the names file. This pattern may end up looking like

/home/myself/**/@(name1|name2|name3).@(txt|csv|py)

... which would match the names that you presumably are interested in. Note though that you would have to do any file type tests yourself in the loop (to sort out regular files from directories etc.)

The shell options set at the top of the script enables the use of the extend pattern @(...|...) (extglob), the use of ** to match down into subdirectories (globstar), allows us to names that are hidden or located in hidden subdirectories (dotglob). We also set nullglob ta make the pattern disappear if there are no matches at all.

Answer 3

With zsh (you are using zsh syntax in your code when you're leaving those parameter expansions unquoted):

names=(foo bar baz)
exts=(txt csv py)
print -rC1 - **/(${(~j[|])names}).${(~j[|])exts})(ND)

Where ${(j[|])array} joins the elements of the array with |, a | considered as a glob operator thanks to ~. N for nullglob, D for dotglob.

Or of course do directly:

print -rC1 - **/(foo|bar|baz).(cvs|py|txt)(ND)

If names and extensions are found one per line in some files, use

names=( ${(f)"$(< names.txt)"} )
 exts=( ${(f)"$(< exts.txt)"}  )

You could also do:

print -rC1 - **/$^names.$^exts(ND)

But that would be a lot less efficient as that would expand a recursive glob for each combination of name + ext.

To use find to do the searching:

cmd=(find . '(') or=()
for name ($^names.$^exts) cmd+=($or -name ${(b)name}) or=(-o)
cmd+=(')')
$cmd