3

In a directory there are several subdirectories, something like:

TopLevelDir
+-- SubDir1
+-- SubDir2
+-- SubDir3
+-- SubDir4
+-- SubDir5
+-- SubDir6

I need to find files containing some specific text in a specific subset of those subdirectories. What is needed can be accomplished with:

find SubDir1 SubDir2 SubDir4 -type f -exec grep -H "desired text" {} \;

A similar search will be needed frequently. To save time and typing, I would like to store the names of the subdirectories to be searched in a file and use that when I run the find command next time, something like:

find subdirs2search.txt -type f  -name="*.txt" -exec grep -H "desired text" {} \;

Searching the web, checking man pages, and even checking my *nix book hasn't turned up anything on how to do this.

Is this possible? If so, how can it be done?

  • 2
    If you used + to end the -exec instead of \;, you wouldn't need the -H option on the grep command (it's the default if multiple file args are used). More importantly, it would also run noticeably faster because it wouldn't need to fork one grep per file (instead it would run grep with as many filename args as will fit on the command line). find SubDir1 SubDir2 SubDir4 -type f -exec grep "desired text" {} + – cas Aug 15 at 3:32
  • 1
    @cas: it's possible that find only finds 1 regular file total, so -H is still a good idea. But yes, -exec ... {} + is much faster than \; especially for lots of small files where startup overhead matters more. – Peter Cordes Aug 15 at 5:31
  • @PeterCordes yes, that's true. i didn't think of that :) – cas Aug 15 at 5:43
8

If the directory names are one-per-line, then you could avoid issues with directories that have spaces or tabs or wildcard characters in their names by using readarray (bash v4+):

readarray -t dirs < subdirs2search.txt
find "${dirs[@]}" ...

That would still not help if some directory names start with -, but with GNU find, there's no way around that.

  • 2
    Prepending ./ to relative paths would ensure they're not parsed as options. – John Kugelman Aug 15 at 13:23
  • 1
    @John, unless they are full paths already (starting with /) – Jeff Schaller Aug 15 at 13:25
5

I found that it didn't limit the search to only text files

ack is often a handy tool for recursive-grep type stuff. It does by default limit searching to text files (determined using heuristics based on file name and content) and will by default skip directories like .git/.svn which if you're a developer is likely what you want. https://beyondgrep.com/.

It's packed by most GNU/Linux distros so it's easy to install. It's written in perl (so its regexps are perl regexps, similar to those of GNU grep -P).

ack -- "desired text"  $(<subdirs.txt)

should probably do what you want and is very easy to type. Plus it does nice color output for interactive use.

(Different ways of word-splitting subdirs.txt onto a command line are covered in other answers. You might want to just let the shell's standard word-splitting do it, or readarray to split only on lines and also block glob expansion.)

  • Not the it only properly works with ASCII/ISO8859-1 data (not non-ASCII UTF-8 characters, even with PERL_UNICODE=SLAD which seems to only work for its --filter) – Stéphane Chazelas Aug 15 at 6:50
  • 1
    PERLIO=':utf8' PERL_UNICODE=A ack 'utf8 text' helps (though makes it significantly slower and reports decoding errors for non-UTF-8 input). – Stéphane Chazelas Aug 15 at 7:09
5

Naturally, posting the question helped snap me out of my fixation on doing this strictly with find and made me think of expanding the file via Bash. I'm posting an answer hoping it will help someone else (and also to document this for my own future use).

The incantation for having Bash expand the file contents is $(<subdirs2search.txt). So, if subdirs2search.txt contains:

SubDir1 SubDir2 SubDir4

A command like the following will accomplish the desired search:

find $(<subdirs2search.txt) -type f -name="*.txt" -exec grep -H "desired text" {} \;
  • Note that it's not specific to bash. The $(<file) operator comes from ksh and is also supported by zsh. In other POSIX-like shells, you can use $(cat file) instead. – Stéphane Chazelas Aug 14 at 21:02
  • @StéphaneChazelas: Thanks for the input. That said, I've rolled back to my original answer. Testing your solution, I found that it didn't limit the search to only text files, and it ran significantly slower. Also, it is different enough that - IMO - you should submit it as your own answer. – GreenMatt Aug 14 at 21:20
  • 1
    "I found that it didn't limit the search to only text files", to be fair, neither does your find command. – muru Aug 15 at 5:30
  • 2
    @muru and Matt: ack is often a handy tool for recursive-grep type stuff. It does by default limit searching to text files. beyondgrep.com. It's packed by most GNU/Linux distros. e.g. ack "desired text" $(<subdirs.txt) should probably do what you want and is very easy to type. Plus it does nice color output for interactive use. – Peter Cordes Aug 15 at 5:36
  • 1
    With grep, that's --include='*.txt' – muru Aug 15 at 15:46
3

Another tool similar to ack (Peter Cordes's answer) is ripgrep or rg. It supports the same usage:

rg -- "desired text" $(<subdirs2search.txt)

It also skips binary files by default.

3
#!/usr/bin/perl -w

use strict;
use File::Find ();

sub wanted;
sub process_file ($@);

my $dirfile = shift;    # First argument is the filename containing the list
                        # of directories.

my $pattern = shift;    # Second arg is a perl RE containing the pattern to search
                        # for. Remember to single-quote it on the command line.

# Read in the @dirs array from $dirfile
#
# A NUL-separated file is best, just in case any of the directory names
# contained line-feeds.  If you're certain that could never happen, a
# plain-text LF-separated file would do.
#
# BTW, you can easily generate a NUL-separated file from the shell with:
#    printf "%s\0" dir1 dir2 dir3 dir4 $'dir\nwith\n3\nLFs' > dirs.txt

my @dirs=();

{
  local $/="\0";    # delete this line if you want to use a LF-separated file.
                    # In that case, the { ... } block around the code from open to
                    # close is no longer needed.  It's only there so it's possible
                    # to make a local change to the $/ aka $INPUT_RECORD_SEPARATOR
                    # variable.

  open(DIRFILE,"<",$dirfile);
  while(<DIRFILE>) {
    chomp;
    push @dirs, $_;
  };
  close(DIRFILE);
};

File::Find::find({wanted => \&wanted}, @dirs);
exit;

sub wanted {
    my ($dev,$ino,$mode,$nlink,$uid,$gid);

    (($dev,$ino,$mode,$nlink,$uid,$gid) = lstat($_)) && -f _ && process_file($_);
}

sub process_file ($@) {

    # This function currently just greps for pattern in the filename passed to
    # it. As the function name implies, it could be used to process the file
    # in any way, not just grep it.

    my $filename = shift;

    # uncomment the return statement below to skip "binary" files.
    # (note this is a workable but fairly crude test.  Perl's File::MMagic
    # module can be used to more accurately identify file types, using the
    # same "magic" file databases as the /usr/bin/file command)

    # return if -B $filename;

    open(FILE,"<",$filename);
    while(<FILE>) {
      print "$filename:$_" if (m/$pattern/o) ;
    };

    close(FILE);
}

This uses perl and perl's File::Find module to do the same thing as your find ... -exec grep.

There's nothing particularly interesting or special about this script except that the process_file function can be very easily modified to do anything you want with or to the file - e.g. change owner or perms, delete it, rename it, insert or delete lines or whatever else you might want.

e.g. if you wanted to delete files which contain text matching the pattern, you could replace the process_file function with something like this:

sub process_file ($@) {

    my $filename = shift;
    my $found = 0;

    # uncomment to skip "binary" files:
    return if -B $filename;

    open(FILE,"<",$filename);
    while(<FILE>) {
      if (m/$pattern/o) {
        $found = 1;
        last;
      };
    };

    close(FILE);
    unlink $filename if ($found);
}

It's also worth mentioning that the wanted function in this script is currently only looking for regular files (the -f test). Perl's stat and lstat functions provide access to all of the file metadata that find can use to match files (uid, gid, perms, size, atime, mtime, etc) so the wanted function can replicate ANY and all find predicates. see perldoc -f stat and perldoc -f lstat for details.

BTW, the script was generated initially by find2perl, and then modified substantially to a) read in the list of directories from a file, and b) to do the grep in perl code rather than by forking grep and c) add a lot of comments. Performance should be nearly identical to find ... -exec grep because grep can't open files or do a regexp pattern match significantly faster than perl can. It may even be faster.

Also BTW, find2perl used to be included with perl, but since perl 5.22 it was removed and can now be found on CPAN at find2perl

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.