Given two ordinary Unix text files, your shell loop prints
since this is the line that occurs in both files. If it does not, then one of your files may be a DOS text file. You can convert DOS text files into Unix text files with the
There is nothing major wrong with your loop given the type of data that you have, apart from the fact that it calls
grep once for every line in
file1. It also would match substrings, for example
1001, and it would, if any line in
file1 contained spaces or tabs, split these lines into multiple words (due to the
for i in $(cat ...) where the
$(cat ...) is unquoted).
If you want to solve your issue this way (with a loop), you would better do
while IFS= read -r word; do
grep -xF -e "$word" file2
-F are explained later in my answer, and
-e signifies that the next argument is the pattern to match with (otherwise, it may be taken as a command line option if it starts with a dash (
This would still execute
grep once for each line in
file1, but it would do it correctly.
To extract lines in
file2 that exactly correspond to line in
file1, without using a shell loop, you would use
$ grep -xF -f file1 file2
This is assuming that
file1 contains a reasonable number of lines, but not too many ("too many" will depend on the amount of memory that you have).
The command uses
-x, which forces matches across full lines only (no substring matches), and with
-F which changes
grep to do string comparisons rather than regular expression matches.
-f file1 instructs
grep to read the patterns (the strings to match with) from
For really massive amounts of data, it would be hugely inefficient to use
grep though. Instead, for this task and with this type of data (single words on individual lines), it would be better to do a relational join operation between the files:
$ join file1 file2
This would, assuming that both files are lexicographically sorted, return the numbers that are the same between both files.
$ comm -1 -2 file1 file2
comm also compares sorted files and can easily handle very large datasets. It prints three columns by default:
- lines that occur in the first file only
- lines that occur in the second file only
- lines that occurs in both files
-1 we turn off the output of the first column, and with
-2 we disable the second column, leaving
comm to only output the lines that are the same in both files.