5

I'm trying to do the following:

cat file1.txt | xargs -I{} "cat file2.txt | grep {}"

I'm expecting each line from file1 to be the value for grep at the end of the third pipe. It's not working as expected.

Is this because -I{} stops looking for things to replace once it hits the pipe? Is there a way around this?

13

It's because you need a shell to create a pipe or perform redirection. Note that cat is the command to concatenate, it makes little sense to use it just for one file.

cat file1.txt | xargs -I{} sh -c 'cat file2.txt | grep -e "$1"' sh {}

Do not do:

cat file1.txt | xargs -I{} sh -c 'cat file2.txt | grep -e {}'

as that would amount to a command injection vulnerability. The {} would be expanded in the code argument to sh so interpreted as shell code. For instance, if one the line of file1.txt was $(reboot) that would call reboot.

The -e (or you could also use --) is also important. Without it, you'd have problems with regexps starting with -.

You can simplify the above using redirections instead of cat:

< file1.txt xargs -I{} sh -c '< file2.txt grep -e "$1"' sh {}

Or simply pass the file names as argument to grep instead of using redirections in which case you can even drop the sh:

< file1.txt xargs -I{} grep -e {} file2.txt

You could also tell grep to look for all the regexps at once in a single invocation:

grep -f file1.txt file2.txt

Note however, that in that case, that's just one regexp for each line of file1.txt, there's none of the special quote processing done by xargs.

xargs by default considers its input as a list of blank (with some implementations only space and tab, on others any in the [:blank:] character class of the current locale) or newline separated words for which backslash and single and double quotes can be used to escape the separators (newline can only be escaped by backslash though) or each other.

For instance, on an input like:

 'a "b'\" "bar baz" x\
y

xargs without -I{} would pass a "b", bar baz and x<newline>y to the command.

With -I{}, xargs gets one word per line but still does some extra processing. It ignores leading (but not trailing) blanks. Blanks are no longer considered as separators, but quote processing is still being done.

On the input above xargs -I{} would pass one a "b" foo bar x<newline>y argument to the command. Also note that one many systems, as required by POSIX, that won't work if words are more than 255 characters long. All in all, xargs -I{} is pretty useless.

If you want each line to be passed verbatim as argument to the command you could use GNU xargs -d '\n' extension:

< file1.txt xargs -d '\n' -n 1 grep file2.txt -e

(here relying on another extension of GNU grep that allows passing options after arguments (provided POSIXly correct is not in the environment) or portably:

sed "s/'/'\\\\\\''/g;s/.*/'&'/" file1.txt | xargs -n1 sh -c '
  for line do
    grep -e "$line" file2.txt
  done' sh

If you wanted each word in file1.txt (quotes still recognised) as opposed to each line to be looked for (which would also work around your trailing space issue if you have one word per line anyway), you can use xargs -n1 alone instead of using -I:

< file1.txt xargs -n1 sh -c '
  for word do
    grep -e "$word" file2.txt
  done' sh

To strip leading and trailing blanks (but without the quote processing that xargs does), you could also do:

unset IFS # restore word splitting to its default
while read -r regexp; do
  grep -e "$regexp" file2.txt
done < file1.txt
  • When I used your commands they didn't work for me but the following modification did: cat file1.txt | xargs -I{} sh -c 'cat file2.txt | grep {}' – Philip Kirkbride Apr 22 '17 at 16:03
  • @Philip, see edit. – Stéphane Chazelas Apr 22 '17 at 16:07
  • I appreciate knowing the vulnerability but I don't understand why the vulnerable command works but none of the other ones suggested do. – Philip Kirkbride Apr 22 '17 at 16:09
  • @Philip, in which way does it not work for you? – Stéphane Chazelas Apr 22 '17 at 16:10
  • 1
    @PhilipKirkbride No extra spaces/quotes/metachars wouldn't break it if you use the -0 option: < file1.txt tr '\n' '\0' | xargs -0 -I{} grep -e {} file2.txt – user218374 Apr 22 '17 at 19:41
8

Depending on what you're trying to do, you may be better off skipping xargs entirely, and going with this solution instead:

grep -f file1.txt file2.txt

This differs from your original command (once we fix it as in Stéphane Chazelas's answer) as follows:

  • Lines are printed in the order they appear in file2.txt regardless of which patterns they match. In your command, all of the lines which match the first pattern are printed, then all of the lines which match the second, and so on.
  • Lines which match more than one pattern are printed exactly once. In your command, they are printed once for each pattern they match.
  • Several flags can be used more easily, including both -v and -c.

The -f flag is specified by POSIX and therefore reasonably portable.

  • Thanks @Stephane posted this option too but I appreciate the expanded explanation of how it differs. – Philip Kirkbride Apr 22 '17 at 19:48
  • @PhilipKirkbride: Good point. Stéphane's answer is quite comprehensive, and I actually didn't spot that in my first read of it. – Kevin Apr 22 '17 at 19:49
  • A great example of KISS and "less is more" – Criggie Apr 22 '17 at 22:19
  • "Lines are printed in the order they appear in file2.txt regardless of which patterns they match" - this also means if you want to implement such custom sorting that can expressed by patterns it can be done very simply and fast using grep. Using grep to sort is not always intuitive. – chx Apr 23 '17 at 6:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.