11

I am trying to run grep against a list of a few hundred files:

$ head -n 3 <(cat files.txt)
admin.php
ajax/accept.php
ajax/add_note.php

However, even though I am grepping for a string that I know is found in the files, the following does not search the files:

$ grep -i 'foo' <(cat files.txt)

$ grep -i 'foo' admin.php
The foo was found

I am familiar with the -f flag which will read the patterns from a file. But how to read the input files?

I had considered the horrible workaround of copying the files to a temporary directory as cp seems to support the <(cat files.txt) format, and from there grepping the files. Shirley there is a better way.

21

You seem to be grepping the list of filenames, not the files themselves. <(cat files.txt) just lists the files. Try <(cat $(cat files.txt)) to actually concatenate them and search them as a single stream, or

grep -i 'foo' $(cat files.txt)

to give grep all the files.

However, if there are too many files on the list, you may have problems with number of arguments. In that case I'd just write

while read filename; do grep -Hi 'foo' "$filename"; done < files.txt
  • Thank you! I did not realize that while could receive the lines of file.txt as such. – dotancohen Jan 13 '15 at 9:25
  • You'll want to disable the glob part of that split+glob operator here (unless the shell is zsh). – Stéphane Chazelas Jan 13 '15 at 10:01
  • 1
    while isn't exactly receiving the lines from the file, read is doing that;while just lets us do that in a loop. The loop ends when read fails (i.e. returns a non-zero return code), normally due to the End Of File being reached. – PM 2Ring Jan 13 '15 at 10:01
  • 1
    To read a (text) line, the syntax is IFS= read -r filename, read filename is something else. – Stéphane Chazelas Jan 13 '15 at 10:02
  • 1
    Note that -H is a GNU extension. You're missing some --. – Stéphane Chazelas Jan 13 '15 at 10:03
8
xargs grep -i -- foo /dev/null < files.txt

assuming files are blank or newline delimited (where quotes or backslashes can be used to escape those separators). With GNU xargs you can specify the delimiter with -d (which then disables the quoting handling though).

(unset -v IFS; set -f; grep -i -- foo $(cat files.txt))

assuming files are space, tab or newline separated (no way to escape those though you can choose a different separator by assigning it to IFS). That one will fail if the file list is too big on most systems.

Those also assume that none of the files are called -.

  • It is better/faster to use $(< file) instead of $(cat file), at least in bash and zsh. – jimmij Jan 13 '15 at 12:49
7

To read a list of file names from stdin you can use xargs. E.g.,

cat files.txt | xargs -d'\n' grep -i -- 'foo'

By default, xargs reads items from the standard input, delimited by blanks. The -d'\n' tells it to use newline as the argument delimiter, so it can handle file names containing blanks. (As Stéphane Chazelas points out, that's a GNU extension). However, it won't cope with file names containing newlines; we'd need a slightly more complicated approach to handle those.

FWIW, this approach is somewhat faster than a while read loop, as bash's read command is very slow - it reads its data character by character, whereas xargs reads its input more efficiently. Also, xargs only invokes the grep command as many times as it needs to, with each invocation receiving multiple file names, and that's more efficient than invoking grep individually for each file name.

See the xargs man page and the xargs info page for further details.

3

xargs can read items from a file (like your files.txt list) with it's option:

   --arg-file=file
   -a file
          Read items from file instead of standard input.  If you use this
          option, stdin remains unchanged when commands are  run.   Other‐
          wise, stdin is redirected from /dev/null.

So this should work too:

xargs -a files.txt grep -i 'foo'

or for spaces in filenames

xargs -d'\n' -a files.txt grep -i 'foo'
xargs -I{} -a files.txt grep -i 'foo' {}
1

You can also do a for but Orion's example is the simplest:

for i in $(cat files.txt); do grep -i 'foo' $i ; done

(For each file listed in the files.txt execute the grep command on it.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.