Piping find into another find does not work

Question

Here's my use case:

Search the entire machine, and find any directory that is named X

Of course, this line works:

find / -type d -name "X"

However, it's a little slow and I assume it uses a lot of resources.

To increase its speed, I thought about piping finds into one another to filter down possible search results. For example, only find directories that have an uppercase first letter in the root, then inside them find directories that are named X.

find / -maxdepth 1 -type d -name "/[A-Z]*" | xargs find -type d -name "X"

However, this did not work.

I already looked at how to pipe find's output into another find but I could not find piping instructions for a couple of finds.

How can I pipe find into another find?

Answer 1

You can run find on the result of another find, but not with that syntax.

find / -maxdepth 1 -type d -name "/[A-Z]*" | xargs find -type d -name "X"

First, you can't use xargs like that on the output of find for any command, find or otherwise, unless none of the file names it outputs contain any whitespace characters, quotes, or backslashes (or non-characters with some implementations).

You can only use xargs with the non-standard -0 option on the output of find -print0 (-print0 also non-standard) if you want to be able to process arbitrary files. The output of find -print itself is simply not post processable (see Why is looping over find's output bad practice?).

Also, here, xargs will append the file paths to the second find command, and put them after the predicates that make up the filtering criteria. The list of files find is to operate on must be given before any predicate.

More generally, using xargs on the output of find is rarely needed, as find has its own builtin (and more reliable and often more efficient) support to run commands on the files it finds with its -exec/-ok (and for some -execdir/-okdir) predicates.

But like with xargs, you need to make sure the list of files for the second find comes before the predicates, so it would have to be:

find / -maxdepth 1 -name '[[:upper:]]*' -type d -exec sh -c '
   exec find "$@" -name X -type d' sh {} +

Where we use the -exec cmd {} + form of -exec which like xargs passes as many possible paths as possible to cmd but also can only pass them at the end. sh is used to move those in the right place for the second find.

Also note that -name matches on the name of the file, not its full path (for which you need -path). So we need [[:upper:]]* not /[[:upper:]]* (nor [A-Z]* where what it matches on is often rather random depending on the locale) to match on filenames that start with an uppercase letter.

With the next version of GNU find (or its current development version), you'd also be able to do:

find / -maxdepth 1 -name '[[:upper:]]*' -type d -print0 |
  find -files0-from - -name X -type d

Here, you could do it whole with one find invocation though:

find / -path '/[![:upper:]]*' -prune -o -name X -type d -print

Where we tell find to prune the branches of the tree that start with directories in / whose name starts by any character other than an uppercase letter before looking for directory named X.

Note that with some find implementations on some systems (including GNU find on GNU systems), * may fail to match parts of file names that are not valid text in the current locale.

For example, the command above would find /stéphane/X even though s is not an uppercase letter if that é was encoded in iso8859-1 and the current locale was using UTF-8 as its charmap (where the 0xe9 byte there could not be decoded into a character so * couldn't match across it), and the first command would fail to find /Stéphane/X for the same reason.

zsh globs don't have this kind of problem as the treat each byte that can't be decoded into a character as if it was an undefined character, so you could do:

print -rC1 /[[:upper:]]*/**/X(ND/)

Or as a slight optimisation if you don't need the list to be ordered:

print -rC1 /[[:upper:]]*/**/X(ND/oN)

Note that it would include /SymLink/.../X dirs. To avoid that:

(){print -rC1 $^@/**/X(ND/oN)} /[[:upper:]]*(N/oN)

or:

print -rC1 /[[:upper:]]*(N/oNe['reply=($REPLY/**/X(ND/oN)'])

Which are like the two stage find approaches: find the directories whose name starts with an uppercase letter in one glob and then all the X dirs in them as separate globs.

Answer 2

Your task is to find subdirectories under / that have an upper-case first letters, and then extract the pathnames of subdirectories in these that are called X.

With find, I would do this like so, assuming there are not many thousands of directories in / with upper-case first letter,

find /[[:upper:]]*/X -prune -type d -print 

Above, we call find with a set of top-level search paths. These pathnames happen to correspond the the directory names that we are actually looking for. The sole task of find becomes to investigate each of these and print the ones that are directories.

You may also skip find completely an instead just use

printf '%s\n' /[[:upper:]]*/X/

The only difference here is that the pattern may match symbolic links to directories. If that matters to you, you could do a more explicit test on the matching names, and tell the shell (which I'm going to assume is bash) to also match hidden names:

shopt -s nullglob

for name in /[[:upper:]]*/X/; do
    [ -L "$name" ] && continue
    printf '%s\n' "$name"
done

---

After re-reading you question several times, I realized that I don't know whether you want to make the search for X recursive or not. In the above discussion, I have assumed that you do not want to search recursively.

If you do want to search recursively, then you would use

find /[[:upper:]]* -name X -type d -print

There is nothing that can make this faster unless you restrict the search to even fewer top-level search paths, or prune known paths under which you know you don't want to search, e.g. to avoid entering any tmp directory,

find /[[:upper:]]* -name X -type d -print -o -name tmp -prune

You may want to restrict the search to a single file-system:

find /[[:upper:]]* -xdev -name X -type d -print -o -name tmp -prune

Here, the the search from a top-level search path would never cross a file-system boundary.