Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a plain text file, in every line is a path to a file. Now I have to grep for a string in these files.

Is there any way to use this file as "search source" for grep? Or do I have to copy and paste every path to the bash?

2nd: Is there a way to provide grep with different files as search source in one line? Like

egrep --color -i "test" /tmp/1.txt, /tmp/2.txt ... ?

share|improve this question

4 Answers 4

up vote 3 down vote accepted

I've rewritten this answer, as it brought up some questions, the answers to which have been a bit foggy. I hope this answer clears some of the fog...

Note: Using xargs is suited to passing positional parameters (args) to a program when there are so many args that they would exceed the memory available to the command line...

The notes are in the script.

#!/bin/bash

  rm -f "/tmp/file   "*

# Create some dummy test files and write their names to /tmp/list
  for x in {A..D} ;do 
      echo "text-$x" >"/tmp/file   $x"
      echo "/tmp/file   $x"
  done >"/tmp/list"

# Set up Quirk 1... with an escaped char \A in the file-name.
        # Replace one of the file-names in the list with a quirky but valid one.
          echo 'quirk 1. \A in filename' >'/tmp/file   \A'
          sed -i 's/A/\\A/'  "/tmp/list"

        # The next two lines show  that 
        #            'file   \A' is in the list      and DOES exist.
        #            'file   A'  is NOT in the list, but DOES exist.
        # Therefore, 'file   A'  should NOT produce a 'grep' match
          echo "Quirk 1. backslash in file-name"  
          echo "   ls:"; ls -1 '/tmp/file   '*A    |nl  
          echo " list:"; sed -n '/A/p' "/tmp/list" |nl
          echo "===================="

# Set up Quirk 2... with $D in the file name
        # Replace one of the file-names in the list with a quirky but valid one.
          echo 'quirk 2. $D in filename' >'/tmp/file   $D'
          sed -i 's/D/\$D/'  "/tmp/list"
          D='D' 
        # The next two lines show  that 
        #            'file   $D' is in the list      and DOES exist.
        #            'file   D'  is NOT in the list, but DOES exist.
          echo "Quirk 2. var \$D=$D in file-name"  
          echo "   ls:"; ls -1 '/tmp/file   '*D    |nl  
          echo " list:"; sed -n '/D/p' "/tmp/list" |nl
          echo "===================="

# The regex search pattern
  regex='(A|C|D)'

# Read lines of a file, and use them as positional parameters.
#  Note: 'protection' means protected from bash pre-processing. (eg path expansion) 
# ============================================================
  ###  
  echo 
  echo "========================================"
  echo "Passing parameters to 'grep' via 'xargs'"    
  echo "========================================"
  echo 
  ###
    echo "# Use 'xargs' with the assumption that every file name contains no meta characters."
    echo "# The result is that file names which contain meta characters, FAILS."   
    echo "# So it interprets '\A' as 'A' and whitespace as a delimiter!"
      <"/tmp/list" xargs  grep -E -H "$regex" 
      echo =====; echo "ERROR: All files in the sample list FAIL!" 
      echo =====; echo
  ###  
    echo "# Use xargs -I{} to avoid problems of whitespace in filenames"
    echo "# But the args are further interpreted by bash, as in escape '\' expansion."
    echo "# Bash still interprets xarg's '\A' as 'A' and so 'grep' processes the wrong file"
    echo "# However the -I{} does protect the $D from var expansion"
      <"/tmp/list" xargs -I{} grep -E -H "$regex" {}
      echo =====; echo "ERROR: The 1st line refers to 'file   A' which is NOT in the list!" 
      echo =====; echo
  ###  
    echo "# Use xargs -0 to avoid problems of whitespace in filenames"
    echo "# 'xargs -0' goes further with parameter protection than -I." 
    echo "# Quotes and backslash are not special (every character is taken literally)" 
      <"/tmp/list" tr '\n' '\0' |xargs -0 grep -E -H "$regex"
      echo  ==; echo "OK" 
      echo  ==; echo
  ###
  echo "====================================="
  echo "Passing parameters directly to 'grep'"    
  echo "====================================="
  echo 
  ###
    echo "# Use 'grep' with the assumption that every file name contains no meta characters."    
    echo "# The result is that file names which contain meta characters, FAILS."   
      grep -E -H "$regex" $(cat "/tmp/list") 
      echo =====; echo "ERROR: All files in the sample list FAIL!" 
      echo =====; echo
  ###  
    echo '# Set bash positional parameters "$1" "$2" ... "$n"'  
    echo "# Note: destructive... original parameters are overwritten"
    echo '#   and, you may need to reset $IFS to its original value'
    IFS=$'\n'
    set $(cat "/tmp/list") 
    grep -E "$regex" "$@"
      echo  ==; echo "OK" 
      echo  ==; echo
  ###
    echo '# Set bash positional parameters "$1" "$2" ... "$n"'  
    echo '# Note: non-destructive... original parameters are not overwritten' 
    echo '# Variable set in the sub-shell are NOT accessible on return.'
    echo '# There is no need to reset $IFS'
    ( IFS=$'\n'
      set $(cat "/tmp/list") 
      grep -E "$regex" "$@" )
      echo  ==; echo "OK" 
      echo  ==; echo
  ### 
    echo '# Using bash array elements "${list[0]}" "${list[1]}" ... "${list[n-1]}"'
    echo '# Note: you may need to reset $IFS to its original value'
    IFS=$'\n'
    list=($(cat "/tmp/list")) 
    grep -E "$regex" "${list[@]}"
      echo  ==; echo "OK" 
      echo  ==; echo
  ### 

Here is the output

Quirk 1. backslash in file-name
   ls:
     1  /tmp/file   A
     2  /tmp/file   \A
 list:
     1  /tmp/file   \A
====================
Quirk 2. var $D=D in file-name
   ls:
     1  /tmp/file   D
     2  /tmp/file   $D
 list:
     1  /tmp/file   $D
====================

========================================
Passing parameters to 'grep' via 'xargs'
========================================

# Use 'xargs' with the assumption that every file name contains no meta characters.
# The result is that file names which contain meta characters, FAILS.
# So it interprets '\A' as 'A' and whitespace as a delimiter!
grep: /tmp/file: No such file or directory
grep: A: No such file or directory
grep: /tmp/file: No such file or directory
grep: B: No such file or directory
grep: /tmp/file: No such file or directory
grep: C: No such file or directory
grep: /tmp/file: No such file or directory
grep: $D: No such file or directory
=====
ERROR: All files in the sample list FAIL!
=====

# Use xargs -I{} to avoid problems of whitespace in filenames
# But the args are further interpreted by bash, as in escape '\' expansion.
# Bash still interprets xarg's '\A' as 'A' and so 'grep' processes the wrong file
# However the -I{} does protect the D from var expansion
/tmp/file   A:text-A
/tmp/file   C:text-C
/tmp/file   $D:quirk 2. $D in filename
=====
ERROR: The 1st line refers to 'file   A' which is NOT in the list!
=====

# Use xargs -0 to avoid problems of whitespace in filenames
# 'xargs -0' goes further with parameter protection than -I.
# Quotes and backslash are not special (every character is taken literally)
/tmp/file   \A:quirk 1. \A in filename
/tmp/file   C:text-C
/tmp/file   $D:quirk 2. $D in filename
==
OK
==

=====================================
Passing parameters directly to 'grep'
=====================================

# Use 'grep' with the assumption that every file name contains no meta characters.
# The result is that file names which contain meta characters, FAILS.
grep: /tmp/file: No such file or directory
grep: \A: No such file or directory
grep: /tmp/file: No such file or directory
grep: B: No such file or directory
grep: /tmp/file: No such file or directory
grep: C: No such file or directory
grep: /tmp/file: No such file or directory
grep: $D: No such file or directory
=====
ERROR: All files in the sample list FAIL!
=====

# Set bash positional parameters "$1" "$2" ... "$n"
# Note: destructive... original parameters are overwritten
#   and, you may need to reset $IFS to its original value
/tmp/file   \A:quirk 1. \A in filename
/tmp/file   C:text-C
/tmp/file   $D:quirk 2. $D in filename
==
OK
==

# Set bash positional parameters "$1" "$2" ... "$n"
# Note: non-destructive... original parameters are not overwritten
# Variable set in the sub-shell are NOT accessible on return.
# There is no need to reset $IFS
/tmp/file   \A:quirk 1. \A in filename
/tmp/file   C:text-C
/tmp/file   $D:quirk 2. $D in filename
==
OK
==

# Using bash array elements "${list[0]}" "${list[1]}" ... "${list[n-1]}"
# Note: you may need to reset $IFS to its original value
/tmp/file   \A:quirk 1. \A in filename
/tmp/file   C:text-C
/tmp/file   $D:quirk 2. $D in filename
==
OK
==
share|improve this answer
    
sorry iam an linux newb, could you explain in more detail please. Thank you –  ArneRie Feb 22 '12 at 9:43
    
And of course, Gilles' set -f version reduces the command to a minimilist syntax, by disabling all globbing with a single option switch. –  Peter.O Feb 23 '12 at 10:36

what about

fgrep <pattern> `cat file_list.txt`

pay attention to put the correct quotes ` and not ' -- if I have understood what you want to do

share|improve this answer
5  
The use of backquotes is strongly discouraged in favor of $() nowadays: fgrep <pattern> $(cat file_list.txt) –  ℝaphink Feb 22 '12 at 10:19
2  
This will fail for files whose names contain whitespace. –  Peter.O Feb 22 '12 at 12:02
1  
And will possibly search in files not included in the list if any of the names include meta characters and the expansion matches existing files. –  manatwork Feb 22 '12 at 16:05
  1. cat filenames.txt | xargs grep <pattern>

  2. grep <pattern> filename1 filename2 filename*

share|improve this answer
1  
Beware that xargs requires input quoted in a peculiar way, so the first command won't work if one of the file names contains whitespace or \'". –  Gilles Feb 22 '12 at 23:53
    
one more quirk is that if the filename or any line in the "filenames.txt" starts with a '-', it is interpreted as a command option. considering this case, and the situation that you've mentioned above, is there a protection to this? –  Sriharsha Mar 16 '12 at 4:47
1  
Put -- before the first file name (see my answer). –  Gilles Mar 16 '12 at 8:35

If you want to search in several files at once, you can type them all at the end of the command line, separated by spaces.

grep -i test /path/to/file /some/other/file

You can use wildcard patterns.

grep -i test README ChangeLog *.txt

If you have a list of files, with one file name per line, then you have several possibilities. If there aren't any exotic characters in your file names, then either of these will work:

grep -i test -- $(cat list_of_file_names.txt)
<list_of_file_names.txt xargs grep -i test -H --

The first command substitutes the output of the command cat list_of_file_names.txt into the command line. It fails if any of the file names contains whitespace or shell wildcards (\[?*). It also fails if the list is so large as to go over the command line length limit (more than about 128kB, on many systems). The second command fails if any of the file names contains whitespace of \"'. It takes care of running egrep multiple times if the command line length limit demands it. The -H option ensures that grep will always print the name of the matching file, even if it happens to have been called with a single file. The -- ensures that if the first file name begins with a -, it will be treated as a file name and not as an option.

A safe way to handle file names that may contain any character other than newlines is to turn off splitting on whitespace other than newlines and turn off globbing (wildcard expansion).

set -f; IFS='
'
grep -i test -- $(cat list_of_file_names.txt)
set +f; unset IFS
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.