find "paths must precede expression" when substituting command results

Question

I am trying to run find to exclude a list of directories that I have in a file. This file is used by a different command as well so I am stuck with the format of the file, which is each line contains the directory name "as is" followed by a slash, e.g.

My Files/

I tried various things such as xargs which I could not get to work but finally settled on pre-processing the blacklist to produce the arguments to find like this:

find . -type f $(cat .blacklist.txt | while read -r line; do printf "! -path './%s*' -prune " "$line"; done)

However, whenever I encounter a directory name in the blacklist which has spaces in it, I get the error:

find: paths must precede expression: `<Last Part Of Name>/*''

I used set -x to try to figure out what is going on, and it appears that the separate parts of the directory name are being individually quoted. For instance, suppose I have a directory name in the list called "My Files". I see this output from set -x for this particular line of the file:

++ printf '! -path '\''./%s*'\'' -prune ' 'My Files/'

So far so good I think. But when I see the output of this in the final assembled command, it looks like this:

'!' -path ''\''./My' 'Files/*'\''' -prune 

The problem is obvious: it's turning my directory name which I worked so hard to keep in one piece, into two pieces! But I can't see why it would be doing this. I have tried a few variations, for instance instead of:

printf "! -path './%s*' -prune " "$line"

I think that somehow I need to make sure that the resulting output of the path is quoted when it comes out of the printf, but I have tried all of these and none of them worked:

printf "! -path \"./%s*\" -prune " "$line"
printf "! -path "'./$line*'" -prune "
printf "! -path '"./$line*"' -prune "

But none of these stopped the individual words in the directory name from being split apart.

I also tried this:

printf "! -path ./%s* -prune " "$line"

This avoids splitting the directory name, but as it's no longer quoted it is getting expanded into all the sub-directories under that path which is wrong and also breaks the command as now there are multiple sub-directories where only one directory is expected.

It also does not seem to matter if I use %s followed by the argument as a separate parameter to printf, or if I substitute %s with $line directly.

Answer 1

and it appears that the separate parts of the directory name are being individually quoted.

I wouldn't say that. What happens is that when you print e.g. ! -path './My Files*' -prune from a command substitution, it gets word-split into !, -path, './My, Files*', -prune as the word-splitting step doesn't process syntax like quotes, but only looks at whitespace characters (or what ever you've set IFS to). That Files*' would also act as a glob, but you likely don't have filenames that end in a single quote.

The quotes you see in the set -x output are there because Bash makes that output valid as input to the shell for display. The escaped quote there is the one you printed from printf.

Here, you need to produce !, -path, ./My Files*, -prune as distinct arguments to find. One way to do that is to collect the arguments in an array.

#!/bin/bash
args=()
filename='./My Files'
args+=( ! -path "$filename*" -prune )
# etc.
find . -type f "${args[@]}"

But you likely really need that as

find . -type f ! -path './somedir/*' ! -path './otherdir/*'

(without the -prune, walking the whole tree including the listed directories, just ignoring anything in them), or like Stéphane Chazelas shows in their answer,

find . \( -path ./somedir -o -path ./otherdir \) -prune -o -type f -print

(not even descending into the listed directories, you need to escape () for the shell).

And if the excluded names can contain characters that are special for find's pattern matching (*?[), you need to escape those with a backslash, along with any backslashes.

If we use the former pattern, and ignore the escaping:

#!/bin/bash
args=(-type f)
while read -r filename; do
    args+=( ! -path "./$filename/*" )
done < excluded.txt
find . "${args[@]}"

the better way is a bit more involved, something like this:

#!/bin/bash
args=()
first=1
while read -r escaped; do
    if [ "$first" != 1 ]; then
        args+=( -o )
    fi
    args+=( -path "./$escaped" )
    first=0
done < <(sed -e 's/[[?*\]/\\&/g' < excluded.txt)
find . \( "${args[@]}" \) -prune -o -type f -print

(here, also using a process substitution (<(cmd...)) to pipe the list of filenames through sed for escaping)

The key when building the array is to put the arguments in the assignment exactly as you'd put them in a command, and not e.g. add any extra quotes. Then when using the array, you must just the "${args[@]}" syntax, noting the quotes.

Answer 2

If the intention is to find regular files, skipping the ones in directories whose paths followed by a / are listed in .blacklist.txt one per line, then with that file containing:

my dir/
 my [other] dir?/sub dir/

for instance, you'd need to call find with these arguments:

0. find
1. .
2. (
3. -path
4. ./my dir
5. -o
6. -path
7. ./ my \[other\] dir\?/sub dir ([ and ? escaped as otherwise they're treated specially by find's -path; escaping ] shouldn't be necessary in most find implementations but won't harm)
8. )
9. -prune # prune the ones matched above
10. -o # anything else:
11. -type
12. f # files of type "regular"
13. -print

So you need for each line:

• to remove the trailing /
• to escape the characters special to -path: \?*[]
• prepend ./
• prepend a -path line

Then put a -o argument in between each, and collect the resulting lines into a list to pass to find.

To do that splitting, in the bash shell, you can do it with readarray -t and in zsh with the f parameter expansion flag. You could use split+glob like you did by omitting the quotes around $(...), but you'd need first to set $IFS to newline only and disable the glob part which we don't want here (not done in zsh) with set -o noglob

In bash:

readarray -t args < <(
  sed 's|/$||
       s|[][\\?*]|\\&|g
       s|^|./|
       1!i\
-o
       i\
-path' .blacklist.txt
)
find . '(' "${args[@]}" ')' -prune -o -type f -print

In zsh:

find . '(' ${(f)"$(
  sed 's|/$||
       s/[][\\?*]/\\&/g
       s|^|./|
       1!i\
-o
       i\
-path' .blacklist.txt)"} ')' -prune -o -type f -print

zsh has the b parameter expansion flag to escape glob operators, and in that shell, you could use globbing with the Nullglob qualifier to reduce the blacklist to directories that actually exist and Prepend the -o and -path arguments.

() {
  find . '(' $@[2,-1] ')' -prune -o -type f -print
} ./${(f)^"$(<.blacklist.txt)"}(Ne['REPLY=${(b)REPLY%/}']P[-o]P[-path])

Here, we're P prepending both -o and -path to every file using the P glob qualifier, passing the list to an anonymous function and and passing the second to last argument to find ($@[2,-1]). Given that you likely want to omit .blacklist.txt itself from the result, you could also do:

find . '(' -path ./.blacklist.txt ./${(f)^"$(<.blacklist.txt)"}(Ne['REPLY=${(b)REPLY%/}']P[-o]P[-path]) ')' -prune -o -type f -print

If you have GNU find or compatible, an alternative to -path that doesn't require having to do that escaping, removing the trailing / or prepending ./ is to use -samefile:

find . '(' -samefile .blacklist.txt ${(f)^"$(<.blacklist.txt)"}(NP[-o]P[-path]) ')' -prune -o -type f -print

Answer 3

I'd do this with perl's File::Find module because constructing long find commands with lots of predicates is a PITA compared to just writing a simple procedural script.

#!/usr/bin/perl

use strict;
use File::Find;
use autodie qw(open);

# load the blacklist into an array
my @blacklist;
open(my $BL,"<","blacklist.txt");
while(<$BL>) {
  # Assume one directory per line. Paths are treated
  # as regular expressions, not as literal strings.
  chomp;
  push @blacklist, $_;
};
close($BL);

# generate a regexp to match all the entries in the array
our $blacklist_re = "^(?:" . join("|",@blacklist) . ")";
#print "$blacklist_re\n";

find { wanted => \&wanted, preprocess => \&prune }, '.';

sub wanted { -f && print "$File::Find::name\n" };
sub prune  { return grep { ! -d || ! m/$blacklist_re/ } @_ };

Updated 2023-04-24 - this version actually prunes the unwanted directories by using a preprocess subroutine, so that File::Find doesn't descend into them.

The preprocess subroutine prune is called every time find enters into a directory (including any directories given as args to the find function, . in this case).

The filenames (including dirs, sockets, device nodes etc) in a dir are passed as an array to prune, which returns an array of filenames which either aren't directories OR don't match the blacklist regexp. This uses perl's built-in grep function - this is not the external /bin/grep command, it operates on lists/arrays instead of files. See perldoc -f grep.

Any filenames NOT in the array returned by prune are excluded from further processing by find, so the wanted subroutine has been simplified to just a -f test - wanted never even sees pruned directories or files.

Sample Run:

With a blacklist.txt file containing the following:

foo bar
bar baz

and creating those directories and some dummy files:

mkdir 'foo bar' 'bar baz' 
touch 'bar baz/a.txt' 'foo bar/b.txt' 'foo bar/c.txt'
touch d.txt

so that the directory structure looks like this:

.
├── bar baz
│   └── a.txt
├── blacklist.txt
├── d.txt
├── ff.pl
├── foo bar
    ├── b.txt
    └── c.txt

Running the script (after saving it as, e.g., ff.pl, and making it executable with chmod +x ff.pl) produces the following output:

$ ./ff.pl 
./ff.pl
./blacklist.txt
./d.txt

i.e. the directores in blacklist.txt are excluded from the output.

Answer 4

What I'd probably do (assuming bash and GNU find):

find . -exec fgrep -qx \{} $(sed 's#/$##' .blacklist.txt) \; -prune -o -type f -print

It is not very efficient if the blacklist is long, but it works with most filenames. (Most: e.g. filenames with newlines in them are still an issue.)