Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm looking for a command or script to do the following - given:

file1.txt:

abcd
efgh 
ijkl
mnop

file2.txt:

123abcd123
123efgh123
123mnop123

I want a command that does something like this:

ungrep file1.txt file2.txt

and returns the following:

ijkl

In other words it is giving me the lines in file1.txt that will not return any results on a grep of file2.txt. I know that I can do this by iterating through file1.txt, grepping file2.txt for each line and storing the result, and outputting any lines where the result is empty, but I was hoping for a more efficient way to do this.

share|improve this question

2 Answers 2

up vote 17 down vote accepted

With GNU grep the following should work. Using the -f option, pass file1.txt as a "pattern file" - but also pass it in a second time as a data file. Use -o to report only the matching parts. Finally extracts those words that match only once - these correspond to the lines from file1.txt that do not find a match in file2.txt.

grep -h -o -f  file1.txt file2.txt file1.txt | sort | uniq -u
ijkl
share|improve this answer
    
Very good description. Thanks and +1. –  unxnut Jun 20 '13 at 17:57
4  
You could achieve the same effect without the grep trickiness: sort file1.txt <(grep -of file1.txt file2.txt) | uniq -u, but, like your solution, this only works when the pattern file doesn't actually contain any regex metacharacters. –  rici Jun 20 '13 at 18:12
    
@rici, that is a very good point –  1_CR Jun 20 '13 at 18:16
    
Thanks, and +1 - accepting this as it's shorter and was answered first –  Edward Shtern Jun 20 '13 at 19:28
2  
Improvement: grep -oFf file1.txt file2.txt | sort file1.txt - | uniq -u –  Stéphane Chazelas Jun 21 '13 at 9:42

You could do it with awk like:

awk '
  NR == FNR {w[$0]; next}
  {for (i in w) if (index($0,i)) delete w[i]}
  END {for (i in w) print i}' file1.txt file2.txt

By using index, we're looking for substrings rather than matching regular expressions.

Because we delete the word from the array as soon as we find a match, we avoid unnecessary searches.

share|improve this answer
    
Thanks, and +1 - wish I could accept both answers. –  Edward Shtern Jun 20 '13 at 19:28
1  
I would only accept this one. It does not invoke any O(n log n) sorting, and does not fail strangely when the patterns contain regex meta-characters, and could be extended to support regexes. –  Kaz Jun 20 '13 at 23:23
1  
@Kaz, yes that can be confusing, and you find many scripts not knowingly allocating array elements unintentionally by doing if (a[$1]) instead of if ($1 in a) for instance. It's the case of every awk including the original awk and nawk, but looking at the standard yesterday, I couldn't find it specified. –  Stéphane Chazelas Jun 21 '13 at 7:19
1  
@Kaz Here's the POSIX quote: "The application shall ensure that a multi-dimensioned index used with the in operator is parenthesized. The in operator, which tests for the existence of a particular array element, shall not cause that element to exist. Any other reference to a nonexistent array element shall automatically create it." It can be found by scrolling a paragraph or two up from here. –  jw013 Jun 25 '13 at 19:50
1  
As long as file1 is not huge (for some value of huge), I'd prefer this solution as it doesn't require any sorting of file2 and would be expected to be much more efficient. –  jw013 Jun 25 '13 at 19:52

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.