compare two files get identical list

Question

file1.txt (50 lines)

TERYUFV00000010753
TERYUFV00000009526 

file2.txt (500 lines)

TERYUFV00000009526 refids_739_known_8/10_target
TERYUFV00000018907 refids_12023_known_21/22_target
TERYUFV00000010753 refids_11775_known_1/1_target

Output.txt

TERYUFV00000010753 refids_11775_known_1/1_target
TERYUFV00000009526 refids_739_known_8/10_target

Compare file1.txt (has 50 lines) with file2.txt (has 500 lines), get the list from file2.txt which are identical to file1.txt.

I tried both join & fgrep command and it outputs empty file

Answer 1

When you use join the entries on each line are like "cells" in a db, but they should be sorted, so you can try,

sort file1.txt > file1_t.txt
sort file2.txt > file2_t.txt

And then do the join

$ join file1_t.txt file2_t.txt

which will give you an external join, i.e. a list of all the occurrences of the cells in both files. To reduce this list to only the entries in both files, pipe the output of the above command into uniq

$ join file1_t.txt file2_t.txt | uniq

Answer 2

fgrep -f file1.txt file2.txt

Here we are obtaining search pattern from file1.txt and searching it in file2.txt. As the text is fixed we are using fgrep for faster search operation.

Answer 3

You need to sort before you join.

$ cat a.in
TERYUFV00000010753
TERYUFV00000009526
$ cat b.in
TERYUFV00000009526 refids_739_known_8/10_target
TERYUFV00000018907 refids_12023_known_21/22_target
TERYUFV00000010753 refids_11775_known_1/1_target
$ join a.in b.in
$ join <(sort a.in) <(sort b.in)
TERYUFV00000009526 refids_739_known_8/10_target
TERYUFV00000010753 refids_11775_known_1/1_target

Answer 4

Assuming that your files are sorted:

comm -12 file1 file2

Answer 5

Does the following line work?

grep -iw -f file1.txt file2.txt

If the files were uploaded to the server from a Windows client, maybe you should run the dos2unix first.

dos2unix file1.txt file2.txt

If the above commands do not work, you can try the following lines to see whether there are extra nonprinting characters at the beginning or end of the lines in file1.txt. The extra nonprinting characters in items of file1.txt may lead to the failure of grep from file2.txt.

cat -v file1.txt
sed -n -l file1.txt

Answer 6

You can also solve this problem using AWK:

NR == FNR {
    line[$1];
    next;
}
$1 in line {
    print $0;
}

As an one liner:

awk 'NR == FNR {line[$1]; next;} $1 in line' file1.txt file2.txt

Be sure to store in memory the smaller file, that is, put it as the first argument of the one liner.