How can I find files in directories listed in a file?

Question

In a directory there are several subdirectories, something like:

TopLevelDir
+-- SubDir1
+-- SubDir2
+-- SubDir3
+-- SubDir4
+-- SubDir5
+-- SubDir6

I need to find files containing some specific text in a specific subset of those subdirectories. What is needed can be accomplished with:

find SubDir1 SubDir2 SubDir4 -type f -exec grep -H "desired text" {} \;

A similar search will be needed frequently. To save time and typing, I would like to store the names of the subdirectories to be searched in a file and use that when I run the find command next time, something like:

find subdirs2search.txt -type f  -name="*.txt" -exec grep -H "desired text" {} \;

Searching the web, checking man pages, and even checking my *nix book hasn't turned up anything on how to do this.

Is this possible? If so, how can it be done?

Answer 1

If the directory names are one-per-line, then you could avoid issues with directories that have spaces or tabs or wildcard characters in their names by using readarray (bash v4+):

readarray -t dirs < subdirs2search.txt
find "${dirs[@]}" ...

That would still not help if some directory names start with -, but with GNU find, there's no way around that.

Answer 2

I found that it didn't limit the search to only text files

ack is often a handy tool for recursive-grep type stuff. It does by default limit searching to text files (determined using heuristics based on file name and content) and will by default skip directories like .git/.svn which if you're a developer is likely what you want. https://beyondgrep.com/.

It's packed by most GNU/Linux distros so it's easy to install. It's written in perl (so its regexps are perl regexps, similar to those of GNU grep -P).

ack -- "desired text"  $(<subdirs.txt)

should probably do what you want and is very easy to type. Plus it does nice color output for interactive use.

(Different ways of word-splitting subdirs.txt onto a command line are covered in other answers. You might want to just let the shell's standard word-splitting do it, or readarray to split only on lines and also block glob expansion.)

Answer 3

Naturally, posting the question helped snap me out of my fixation on doing this strictly with find and made me think of expanding the file via Bash. I'm posting an answer hoping it will help someone else (and also to document this for my own future use).

The incantation for having Bash expand the file contents is $(<subdirs2search.txt). So, if subdirs2search.txt contains:

SubDir1 SubDir2 SubDir4

A command like the following will accomplish the desired search:

find $(<subdirs2search.txt) -type f -name="*.txt" -exec grep -H "desired text" {} \;

Answer 4

Another tool similar to ack (Peter Cordes's answer) is ripgrep or rg. It supports the same usage:

rg -- "desired text" $(<subdirs2search.txt)

It also skips binary files by default.

Answer 5

#!/usr/bin/perl -w

use strict;
use File::Find ();

sub wanted;
sub process_file ($@);

my $dirfile = shift;    # First argument is the filename containing the list
                        # of directories.

my $pattern = shift;    # Second arg is a perl RE containing the pattern to search
                        # for. Remember to single-quote it on the command line.

# Read in the @dirs array from $dirfile
#
# A NUL-separated file is best, just in case any of the directory names
# contained line-feeds.  If you're certain that could never happen, a
# plain-text LF-separated file would do.
#
# BTW, you can easily generate a NUL-separated file from the shell with:
#    printf "%s\0" dir1 dir2 dir3 dir4 $'dir\nwith\n3\nLFs' > dirs.txt

my @dirs=();

{
  local $/="\0";    # delete this line if you want to use a LF-separated file.
                    # In that case, the { ... } block around the code from open to
                    # close is no longer needed.  It's only there so it's possible
                    # to make a local change to the $/ aka $INPUT_RECORD_SEPARATOR
                    # variable.

  open(DIRFILE,"<",$dirfile);
  while(<DIRFILE>) {
    chomp;
    push @dirs, $_;
  };
  close(DIRFILE);
};

File::Find::find({wanted => \&wanted}, @dirs);
exit;

sub wanted {
    my ($dev,$ino,$mode,$nlink,$uid,$gid);

    (($dev,$ino,$mode,$nlink,$uid,$gid) = lstat($_)) && -f _ && process_file($_);
}

sub process_file ($@) {

    # This function currently just greps for pattern in the filename passed to
    # it. As the function name implies, it could be used to process the file
    # in any way, not just grep it.

    my $filename = shift;

    # uncomment the return statement below to skip "binary" files.
    # (note this is a workable but fairly crude test.  Perl's File::MMagic
    # module can be used to more accurately identify file types, using the
    # same "magic" file databases as the /usr/bin/file command)

    # return if -B $filename;

    open(FILE,"<",$filename);
    while(<FILE>) {
      print "$filename:$_" if (m/$pattern/o) ;
    };

    close(FILE);
}

This uses perl and perl's File::Find module to do the same thing as your find ... -exec grep.

There's nothing particularly interesting or special about this script except that the process_file function can be very easily modified to do anything you want with or to the file - e.g. change owner or perms, delete it, rename it, insert or delete lines or whatever else you might want.

e.g. if you wanted to delete files which contain text matching the pattern, you could replace the process_file function with something like this:

sub process_file ($@) {

    my $filename = shift;
    my $found = 0;

    # uncomment to skip "binary" files:
    return if -B $filename;

    open(FILE,"<",$filename);
    while(<FILE>) {
      if (m/$pattern/o) {
        $found = 1;
        last;
      };
    };

    close(FILE);
    unlink $filename if ($found);
}

It's also worth mentioning that the wanted function in this script is currently only looking for regular files (the -f test). Perl's stat and lstat functions provide access to all of the file metadata that find can use to match files (uid, gid, perms, size, atime, mtime, etc) so the wanted function can replicate ANY and all find predicates. see perldoc -f stat and perldoc -f lstat for details.

BTW, the script was generated initially by find2perl, and then modified substantially to a) read in the list of directories from a file, and b) to do the grep in perl code rather than by forking grep and c) add a lot of comments. Performance should be nearly identical to find ... -exec grep because grep can't open files or do a regexp pattern match significantly faster than perl can. It may even be faster.

Also BTW, find2perl used to be included with perl, but since perl 5.22 it was removed and can now be found on CPAN at find2perl