5

Suppose I have two files a.txt and b.txt. I want to find all the words in a.txt which appear in b.txt.

Is there a specific command to do that?

4 Answers 4

8

With bash, zsh and some implementations of ksh:

comm -12 <(tr -s '[:space:]' '[\n*]' < a.txt | sort -u) \
         <(tr -s '[:space:]' '[\n*]' < b.txt | sort -u)

There, word is a sequence of non-spacing character (beware that with GNU tr, that doesn't work with multi-byte spacing characters).

comm finds the common lines between two sorted files. Without options, it prints 3 columns: the lines only in file1, the lines only in file2, and the lines common to both. You add -1, -2, -3 to remove the corresponding columns from the output. So comm -12 only leaves the third column (the common lines).

tr -s '[:space:]' '[\n*]' transliterate any sequence of characters of class space into newlines, to put every word on its own line.

sort -u sorts and removes duplicates from tr's output.

Process substitution <(...) pipes the outputs of the tr|sort commands to comm.


With zsh:

w1=($(<a.txt)) w2=($(<b.txt))
print -rl -- ${(u)${w1:*w2}}

There, word is a sequence of characters other than space, tab, nul and newline (with the default value of $IFS).

$(<a.txt) is an optimised version of $(cat a.txt) where zsh reads the content of the file by itself without invoking cat, since it's not quoted, it undergoes word splitting (but not globbing contrary to other shells).

So w1 and w2 are arrays containing all the words in a.txt and b.txt.

${w1:*w2} is a zsh operator that gives the intersection of two arrays (the elements common to both). (u) is a parameter expansion flag that retains unique elements (removes duplicates).

print -rl prints each argument one per line.

1
  • sorry, my fault now it is ok
    – Reyx_0
    Dec 15, 2015 at 3:39
2
# Create dummy text file containing two words
$ echo -e "overflow\ngrep" > b
# Search in file for lines containing one word from file b
$ grep --color --fixed-strings --file b /usr/share/dict/words

Result on my system:

overflow
overflow's
overflowed
overflowing
overflows

Add the --only-matching (-o) parameter to only get the words and not the whole line they appear in.

2
  • I just noticed: this answer has one of the same problems that I pointed out in another answer: it will display every word in a.txt that contains a word in b.txt.  For example, if b.txt contains the words a and the, you will get every word in a.txt that contains an a, and words that contain the, like there, other, and lathe. Dec 15, 2015 at 14:38
  • If you want the's without lathe's, use grep -w. Aug 9, 2017 at 16:29
-1

Assuming the words in the files are separated by LF and the words consist only of "nice" characters and there is no stray last LF in b.txt, then

egrep `tr '\n' '|' < b.txt` a.txt

might do the trick.

4
  • mmm not really, I get a confused output including all the words in b.txt followed by "no such file or directory", I would like to have the correct output instead i.e. just the words appearing in both. Maybe it is possible to first modify the file splitting the words in different lines and then comparing the lines
    – Reyx_0
    Dec 12, 2015 at 21:36
  • (0) Reyx_0: I don't understand how you got that output.  However, I would expect this command to display all the words in a.txt.  (1) RadovanGarabík: "Assuming ... there is no stray last LF in b.txt"?  I.e., you're assuming (and depending on the "fact") that the file is malformed.  That a hard assumption to justify.  (2) Even if you get past that problem, your command will display every word in a.txt that contains a word in b.txt.  For example, if b.txt contains the cat caught a rat, you will get every word in a.txt that contains an a, and words like there and other. Dec 13, 2015 at 3:39
  • @G-Man Well, the original question had nothing about the presumed format the files are in, so I went for the simplest command, upon which more complicated scripts can be build. The assumption was that words are on separate lines, which is quite common in text corpora processing. Dec 13, 2015 at 10:09
  • @RadovanGarabík: It seems like you don't get what I'm saying.  (1) The question refers to the files as a.txt and b.txt.  That implies that they are text files.  In fact, the question title says "text file" (twice).  And this site is unix.se.  Text files on Unix end with LF.  (2) Your command will report words that are not in b.txt.  An even simpler answer would be echo foo, but that's wrong.  Everything should be made as simple as possible, but not simpler (attributed to Albert Einstein).  That fact that it's simple is not a justification for posting a wrong answer. Dec 13, 2015 at 19:29
-1

While not working on a by word level, more in towards working on lines this could be of use to you or someone else looking for an answer.

diff --left-column --from-file=a.txt --to-file=b.txt

Compares from file a.txt to file b.txt outputting only common lines.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .