finding all the words in a text file appearing in another text file

Question

Suppose I have two files a.txt and b.txt. I want to find all the words in a.txt which appear in b.txt.

Is there a specific command to do that?

Answer 1

With bash, zsh and some implementations of ksh:

comm -12 <(tr -s '[:space:]' '[\n*]' < a.txt | sort -u) \
         <(tr -s '[:space:]' '[\n*]' < b.txt | sort -u)

There, word is a sequence of non-spacing character (beware that with GNU tr, that doesn't work with multi-byte spacing characters).

comm finds the common lines between two sorted files. Without options, it prints 3 columns: the lines only in file1, the lines only in file2, and the lines common to both. You add -1, -2, -3 to remove the corresponding columns from the output. So comm -12 only leaves the third column (the common lines).

tr -s '[:space:]' '[\n*]' transliterate any sequence of characters of class space into newlines, to put every word on its own line.

sort -u sorts and removes duplicates from tr's output.

Process substitution <(...) pipes the outputs of the tr|sort commands to comm.

---

With zsh:

w1=($(<a.txt)) w2=($(<b.txt))
print -rl -- ${(u)${w1:*w2}}

There, word is a sequence of characters other than space, tab, nul and newline (with the default value of $IFS).

$(<a.txt) is an optimised version of $(cat a.txt) where zsh reads the content of the file by itself without invoking cat, since it's not quoted, it undergoes word splitting (but not globbing contrary to other shells).

So w1 and w2 are arrays containing all the words in a.txt and b.txt.

${w1:*w2} is a zsh operator that gives the intersection of two arrays (the elements common to both). (u) is a parameter expansion flag that retains unique elements (removes duplicates).

print -rl prints each argument one per line.

Answer 2

# Create dummy text file containing two words
$ echo -e "overflow\ngrep" > b
# Search in file for lines containing one word from file b
$ grep --color --fixed-strings --file b /usr/share/dict/words

Result on my system:

overflow
overflow's
overflowed
overflowing
overflows

Add the --only-matching (-o) parameter to only get the words and not the whole line they appear in.

Answer 3

Assuming the words in the files are separated by LF and the words consist only of "nice" characters and there is no stray last LF in b.txt, then

egrep `tr '\n' '|' < b.txt` a.txt

might do the trick.

Answer 4

While not working on a by word level, more in towards working on lines this could be of use to you or someone else looking for an answer.

diff --left-column --from-file=a.txt --to-file=b.txt

Compares from file a.txt to file b.txt outputting only common lines.