Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
I have a plain text file, in every line is a path to a file. Now I have to grep for a string in these files.
Is there any way to use this file as "search source" for grep? Or do I have to copy and paste every path to the bash?
2nd: Is there a way to provide grep with different files as search source in one line? Like
egrep --color -i "test" /tmp/1.txt, /tmp/2.txt ... ?
what about
fgrep <pattern> `cat file_list.txt`
pay attention to put the correct quotes ` and not ' -- if I have understood what you want to do
$() nowadays: fgrep <pattern> $(cat file_list.txt)
I've rewritten this answer, as it brought up some questions, the answers to which have been a bit foggy. I hope this answer clears some of the fog...
Note: Using xargs is suited to passing positional parameters (args) to a program when there are so many args that they would exceed the memory available to the command line...
The notes are in the script.
#!/bin/bash
rm -f "/tmp/file "*
# Create some dummy test files and write their names to /tmp/list
for x in {A..D} ;do
echo "text-$x" >"/tmp/file $x"
echo "/tmp/file $x"
done >"/tmp/list"
# Set up Quirk 1... with an escaped char \A in the file-name.
# Replace one of the file-names in the list with a quirky but valid one.
echo 'quirk 1. \A in filename' >'/tmp/file \A'
sed -i 's/A/\\A/' "/tmp/list"
# The next two lines show that
# 'file \A' is in the list and DOES exist.
# 'file A' is NOT in the list, but DOES exist.
# Therefore, 'file A' should NOT produce a 'grep' match
echo "Quirk 1. backslash in file-name"
echo " ls:"; ls -1 '/tmp/file '*A |nl
echo " list:"; sed -n '/A/p' "/tmp/list" |nl
echo "===================="
# Set up Quirk 2... with $D in the file name
# Replace one of the file-names in the list with a quirky but valid one.
echo 'quirk 2. $D in filename' >'/tmp/file $D'
sed -i 's/D/\$D/' "/tmp/list"
D='D'
# The next two lines show that
# 'file $D' is in the list and DOES exist.
# 'file D' is NOT in the list, but DOES exist.
echo "Quirk 2. var \$D=$D in file-name"
echo " ls:"; ls -1 '/tmp/file '*D |nl
echo " list:"; sed -n '/D/p' "/tmp/list" |nl
echo "===================="
# The regex search pattern
regex='(A|C|D)'
# Read lines of a file, and use them as positional parameters.
# Note: 'protection' means protected from bash pre-processing. (eg path expansion)
# ============================================================
###
echo
echo "========================================"
echo "Passing parameters to 'grep' via 'xargs'"
echo "========================================"
echo
###
echo "# Use 'xargs' with the assumption that every file name contains no meta characters."
echo "# The result is that file names which contain meta characters, FAILS."
echo "# So it interprets '\A' as 'A' and whitespace as a delimiter!"
<"/tmp/list" xargs grep -E -H "$regex"
echo =====; echo "ERROR: All files in the sample list FAIL!"
echo =====; echo
###
echo "# Use xargs -I{} to avoid problems of whitespace in filenames"
echo "# But the args are further interpreted by bash, as in escape '\' expansion."
echo "# Bash still interprets xarg's '\A' as 'A' and so 'grep' processes the wrong file"
echo "# However the -I{} does protect the $D from var expansion"
<"/tmp/list" xargs -I{} grep -E -H "$regex" {}
echo =====; echo "ERROR: The 1st line refers to 'file A' which is NOT in the list!"
echo =====; echo
###
echo "# Use xargs -0 to avoid problems of whitespace in filenames"
echo "# 'xargs -0' goes further with parameter protection than -I."
echo "# Quotes and backslash are not special (every character is taken literally)"
<"/tmp/list" tr '\n' '\0' |xargs -0 grep -E -H "$regex"
echo ==; echo "OK"
echo ==; echo
###
echo "====================================="
echo "Passing parameters directly to 'grep'"
echo "====================================="
echo
###
echo "# Use 'grep' with the assumption that every file name contains no meta characters."
echo "# The result is that file names which contain meta characters, FAILS."
grep -E -H "$regex" $(cat "/tmp/list")
echo =====; echo "ERROR: All files in the sample list FAIL!"
echo =====; echo
###
echo '# Set bash positional parameters "$1" "$2" ... "$n"'
echo "# Note: destructive... original parameters are overwritten"
echo '# and, you may need to reset $IFS to its original value'
IFS=$'\n'
set $(cat "/tmp/list")
grep -E "$regex" "$@"
echo ==; echo "OK"
echo ==; echo
###
echo '# Set bash positional parameters "$1" "$2" ... "$n"'
echo '# Note: non-destructive... original parameters are not overwritten'
echo '# Variable set in the sub-shell are NOT accessible on return.'
echo '# There is no need to reset $IFS'
( IFS=$'\n'
set $(cat "/tmp/list")
grep -E "$regex" "$@" )
echo ==; echo "OK"
echo ==; echo
###
echo '# Using bash array elements "${list[0]}" "${list[1]}" ... "${list[n-1]}"'
echo '# Note: you may need to reset $IFS to its original value'
IFS=$'\n'
list=($(cat "/tmp/list"))
grep -E "$regex" "${list[@]}"
echo ==; echo "OK"
echo ==; echo
###
Here is the output
Quirk 1. backslash in file-name
ls:
1 /tmp/file A
2 /tmp/file \A
list:
1 /tmp/file \A
====================
Quirk 2. var $D=D in file-name
ls:
1 /tmp/file D
2 /tmp/file $D
list:
1 /tmp/file $D
====================
========================================
Passing parameters to 'grep' via 'xargs'
========================================
# Use 'xargs' with the assumption that every file name contains no meta characters.
# The result is that file names which contain meta characters, FAILS.
# So it interprets '\A' as 'A' and whitespace as a delimiter!
grep: /tmp/file: No such file or directory
grep: A: No such file or directory
grep: /tmp/file: No such file or directory
grep: B: No such file or directory
grep: /tmp/file: No such file or directory
grep: C: No such file or directory
grep: /tmp/file: No such file or directory
grep: $D: No such file or directory
=====
ERROR: All files in the sample list FAIL!
=====
# Use xargs -I{} to avoid problems of whitespace in filenames
# But the args are further interpreted by bash, as in escape '\' expansion.
# Bash still interprets xarg's '\A' as 'A' and so 'grep' processes the wrong file
# However the -I{} does protect the D from var expansion
/tmp/file A:text-A
/tmp/file C:text-C
/tmp/file $D:quirk 2. $D in filename
=====
ERROR: The 1st line refers to 'file A' which is NOT in the list!
=====
# Use xargs -0 to avoid problems of whitespace in filenames
# 'xargs -0' goes further with parameter protection than -I.
# Quotes and backslash are not special (every character is taken literally)
/tmp/file \A:quirk 1. \A in filename
/tmp/file C:text-C
/tmp/file $D:quirk 2. $D in filename
==
OK
==
=====================================
Passing parameters directly to 'grep'
=====================================
# Use 'grep' with the assumption that every file name contains no meta characters.
# The result is that file names which contain meta characters, FAILS.
grep: /tmp/file: No such file or directory
grep: \A: No such file or directory
grep: /tmp/file: No such file or directory
grep: B: No such file or directory
grep: /tmp/file: No such file or directory
grep: C: No such file or directory
grep: /tmp/file: No such file or directory
grep: $D: No such file or directory
=====
ERROR: All files in the sample list FAIL!
=====
# Set bash positional parameters "$1" "$2" ... "$n"
# Note: destructive... original parameters are overwritten
# and, you may need to reset $IFS to its original value
/tmp/file \A:quirk 1. \A in filename
/tmp/file C:text-C
/tmp/file $D:quirk 2. $D in filename
==
OK
==
# Set bash positional parameters "$1" "$2" ... "$n"
# Note: non-destructive... original parameters are not overwritten
# Variable set in the sub-shell are NOT accessible on return.
# There is no need to reset $IFS
/tmp/file \A:quirk 1. \A in filename
/tmp/file C:text-C
/tmp/file $D:quirk 2. $D in filename
==
OK
==
# Using bash array elements "${list[0]}" "${list[1]}" ... "${list[n-1]}"
# Note: you may need to reset $IFS to its original value
/tmp/file \A:quirk 1. \A in filename
/tmp/file C:text-C
/tmp/file $D:quirk 2. $D in filename
==
OK
==
set -f version reduces the command to a minimilist syntax, by disabling all globbing with a single option switch.
If you want to search in several files at once, you can type them all at the end of the command line, separated by spaces.
grep -i test /path/to/file /some/other/file
You can use wildcard patterns.
grep -i test README ChangeLog *.txt
If you have a list of files, with one file name per line, then you have several possibilities. If there aren't any exotic characters in your file names, then either of these will work:
grep -i test -- $(cat list_of_file_names.txt)
<list_of_file_names.txt xargs grep -i test -H --
The first command substitutes the output of the command cat list_of_file_names.txt into the command line. It fails if any of the file names contains whitespace or shell wildcards (\[?*). It also fails if the list is so large as to go over the command line length limit (more than about 128kB, on many systems). The second command fails if any of the file names contains whitespace of \"'. It takes care of running egrep multiple times if the command line length limit demands it. The -H option ensures that grep will always print the name of the matching file, even if it happens to have been called with a single file. The -- ensures that if the first file name begins with a -, it will be treated as a file name and not as an option.
A safe way to handle file names that may contain any character other than newlines is to turn off splitting on whitespace other than newlines and turn off globbing (wildcard expansion).
set -f; IFS='
'
grep -i test -- $(cat list_of_file_names.txt)
set +f; unset IFS
cat filenames.txt | xargs grep <pattern>
grep <pattern> filename1 filename2 filename*
xargs requires input quoted in a peculiar way, so the first command won't work if one of the file names contains whitespace or \'".
Feb 22, 2012 at 23:53
-- before the first file name (see my answer).
Mar 16, 2012 at 8:35