find and globbing (and wildcards)

Question

I tried finding some files (*.e*) that are in the same directory as another file (md.tpr). I needed to list them (for further processing) using the following:

find . -name md.tpr -execdir ls *.e* \;

I tried a few variants of this command and some others (including single quoting the command passed to -execdir or passing it as sh -c 'ls *.e*' or eval 'ls *.e*' to name a few). It appears that globbing is not working when passed to -exec or -execdir. The error I get when running the above command is:

ls: *.e*: No such file or directory

Just as a sanity check, I did -execdir pwd and it prints what it should, so it appears that it's a problem with globbing, as those *.e* files do exist in the directories listed with this test.

Now, I was able to solve this problem in a much less elegant way but it just baffled me why globbing and wildcards wouldn't work here. Any ideas? Or am I completely off the track?

I use bash 3.2.25 (old but I don't have admin rights on that system).

Also, interestingly, if I do

find ~ -name .bashrc -maxdepth 2 -execdir ls -d .b* \;

it doesn't work unless it's done from $HOME.

Answer 1

Both the find command and the shell are capable of file globbing.

This is unusual - most commands are NOT capable of globbing and rely entirely on the shell to expand globs. But find is a super duper mega power user tool that you can very easily hurt yourself with!

Example: when you do the command

   find /path -iname *.txt

The first thing that happens is the shell attempts to find all files that match *.txt in the current directory. If it finds any, it substitutes the names of all the matching files for the glob, and then calls the find command. The find command never sees the glob if this happens, the shell has expanded it out of existence.

But if there are no files in the current directory that match the glob, the shell shrugs its metaphorical shoulders and passes the glob unchanged to find. So at that point, the find command (which understands globs, remember) will output the names of all files it finds under /path that match the glob.

So using globs this way means that find will behave differently depending on the content of the current directory. This is almost certainly not what you want!

To prevent the shell from tampering with globs before find sees them, escape the globs with appropriate shell metacharacter quoting. Usually this just means putting your globstrings in single quotes like this

   find /path -iname '*.txt'

Remember, GLOBS ARE NOT REGEXES - the glob ".*" and the regex ".*" are very different and do not match the same strings!

Answer 2

When you type a command that includes an unquoted glob like .b* or *.e*, the shell will expand that for you. This happens before find ever sees it.

You probably have files like .bashrc, .bash_history, etc. in your $HOME directory. So when run from $HOME, your command turns into find ... -execdir ls -d .bashrc .bash_history ... \;. When run from other places, the .b* glob doesn't match anything, so it gets passed through. This still doesn't work, as find -exec doesn't do anything with * globs. If you wanted the glob to be expanded for the -exec, you would need to invoke a shell to do it:

find ... -execdir sh -c 'echo globs: *' \;

Answer 3

When the bash shell can't find a file that matches a given globbing pattern, it leaves the pattern unexpanded. This leads to, in your find command, that ls gets the unexpanded pattern *.e* as its argument. The ls utility does not do expansions of filename globbing patterns by itself but relies on the shell to have done that already.

This is most likely your issue with your last find command that only seems to be able to work correctly from your home directory, probably because your home directory contains files matching the pattern (.bashrc, for example, matches .b*). When the patterns don't match anything in the current directory, the pattern will be handed to ls as it is, and since ls does not expand globbing patterns by itself, it would fail to list any files.

In short, you can't call ls directly with -execdir or -exec and give it a filename globbing pattern.

You additionally say that you want to list the files matching *.e* for "further processing". I would urge not to do that, and instead to do that processing in the actual find command itself. The reason for this is given in the question/answer "Why is looping over find's output bad practice?".

So, instead of what you're currently doing, consider

find . -type f -name md.tpr -exec bash -O nullglob -O dotglob -c '
    for pathname do
        dirpath=${pathname%/md.tpr}
        for e in "$dirpath"/*.e*; do
            # process "$e" here!
        done
    done' bash {} +

This is assuming that md.tpr is a regular file that should be found. The find command would find the pathnames of all these md.tpr files and feed them in batches to an inline bash script. The bash script is short:

for pathname do
    dirpath=${pathname%/md.tpr}
    for e in "$dirpath"/*.e*; do
        # process "$e" here!
    done
done

This simply takes the given arguments, extracts the directory component of each (by removing the /md.tpr suffix string, which we know is there, from the pathname) and loops over the files matching *.e* in each directory (with $e holding the pathname of each matched file in turn).

The in-line script is run with the nullglob and dotglob options set, so that the *.e* pattern would be removed completely if it doesn't match, and so that the pattern would match hidden names.

A bit more info about using -exec with find can be found in "Understanding the -exec option of `find`".

---

Since you've tagged this question with bash, this is how to do the same thing in a plain bash loop (this requires bash release 4 or later):

shopt -s globstar nullglob dotglob

for pathname in ./**/md.tpr; do
    dirpath=${pathname%/md.tpr}
    for e in "$dirpath"/*.e*; do
        # process "$e" here!
    done
done

Apart from there not being any checks for whether the matched md.tpr files are regular files or not, this should look very similar to the in-line script called by find above. The globstar shell option in bash enables the ** glob, which matches "recursively" down into subdirectories.

I would expect this to be slightly slower than using find, but it may be a more convenient way of writing the code.