14

I'm looking for a command or script to do the following - given:

file1.txt:

abcd
efgh 
ijkl
mnop

file2.txt:

123abcd123
123efgh123
123mnop123

I want a command that does something like this:

ungrep file1.txt file2.txt

and returns the following:

ijkl

In other words it is giving me the lines in file1.txt that will not return any results on a grep of file2.txt. I know that I can do this by iterating through file1.txt, grepping file2.txt for each line and storing the result, and outputting any lines where the result is empty, but I was hoping for a more efficient way to do this.

0

2 Answers 2

18

With GNU grep the following should work. Using the -f option, pass file1.txt as a "pattern file" - but also pass it in a second time as a data file. Use -o to report only the matching parts. Finally extracts those words that match only once - these correspond to the lines from file1.txt that do not find a match in file2.txt.

grep -h -o -f  file1.txt file2.txt file1.txt | sort | uniq -u
ijkl
4
  • Very good description. Thanks and +1.
    – unxnut
    Jun 20, 2013 at 17:57
  • 4
    You could achieve the same effect without the grep trickiness: sort file1.txt <(grep -of file1.txt file2.txt) | uniq -u, but, like your solution, this only works when the pattern file doesn't actually contain any regex metacharacters.
    – rici
    Jun 20, 2013 at 18:12
  • @rici, that is a very good point
    – iruvar
    Jun 20, 2013 at 18:16
  • 2
    Improvement: grep -oFf file1.txt file2.txt | sort file1.txt - | uniq -u Jun 21, 2013 at 9:42
10

You could do it with awk like:

awk '
  NR == FNR {w[$0]; next}
  {for (i in w) if (index($0,i)) delete w[i]}
  END {for (i in w) print i}' file1.txt file2.txt

By using index, we're looking for substrings rather than matching regular expressions.

Because we delete the word from the array as soon as we find a match, we avoid unnecessary searches.

6
  • 1
    I would only accept this one. It does not invoke any O(n log n) sorting, and does not fail strangely when the patterns contain regex meta-characters, and could be extended to support regexes.
    – Kaz
    Jun 20, 2013 at 23:23
  • I can't believe that simply evaluating w[$0] has the side effect of adding the key to the array.
    – Kaz
    Jun 20, 2013 at 23:26
  • 1
    @Kaz, yes that can be confusing, and you find many scripts not knowingly allocating array elements unintentionally by doing if (a[$1]) instead of if ($1 in a) for instance. It's the case of every awk including the original awk and nawk, but looking at the standard yesterday, I couldn't find it specified. Jun 21, 2013 at 7:19
  • 1
    @Kaz Here's the POSIX quote: "The application shall ensure that a multi-dimensioned index used with the in operator is parenthesized. The in operator, which tests for the existence of a particular array element, shall not cause that element to exist. Any other reference to a nonexistent array element shall automatically create it." It can be found by scrolling a paragraph or two up from here.
    – jw013
    Jun 25, 2013 at 19:50
  • 1
    As long as file1 is not huge (for some value of huge), I'd prefer this solution as it doesn't require any sorting of file2 and would be expected to be much more efficient.
    – jw013
    Jun 25, 2013 at 19:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .