Given two ordinary Unix text files, your shell loop prints
1235
since this is the line that occurs in both files. If it does not, then one of your files may be a DOS text file. You can convert DOS text files into Unix text files with the dos2unix
utility.
There is nothing major wrong with your loop given the type of data that you have, apart from the fact that it calls grep
once for every line in file1
. It also would match substrings, for example 100
in 1001
, and it would, if any line in file1
contained spaces or tabs, split these lines into multiple words (due to the for i in $(cat ...)
where the $(cat ...)
is unquoted).
If you want to solve your issue this way (with a loop), you would better do
while IFS= read -r word; do
grep -xF -e "$word" file2
done <file1
The -x
and -F
are explained later in my answer, and -e
signifies that the next argument is the pattern to match with (otherwise, it may be taken as a command line option if it starts with a dash (-
).
This would still execute grep
once for each line in file1
, but it would do it correctly.
To extract lines in file2
that exactly correspond to line in file1
, without using a shell loop, you would use
$ grep -xF -f file1 file2
1235
This is assuming that file1
contains a reasonable number of lines, but not too many ("too many" will depend on the amount of memory that you have).
The command uses grep
with -x
, which forces matches across full lines only (no substring matches), and with -F
which changes grep
to do string comparisons rather than regular expression matches.
The -f file1
instructs grep
to read the patterns (the strings to match with) from file1
.
For really massive amounts of data, it would be hugely inefficient to use grep
though. Instead, for this task and with this type of data (single words on individual lines), it would be better to do a relational join operation between the files:
$ join file1 file2
1235
This would, assuming that both files are lexicographically sorted, return the numbers that are the same between both files.
Using comm
:
$ comm -1 -2 file1 file2
1235
comm
also compares sorted files and can easily handle very large datasets. It prints three columns by default:
- lines that occur in the first file only
- lines that occur in the second file only
- lines that occurs in both files
With -1
we turn off the output of the first column, and with -2
we disable the second column, leaving comm
to only output the lines that are the same in both files.
grep -Fwf file1 file2
insteadfor i in $(cat file1); do grep ${i} file1; done
it still doesn't work. I'll try your advicesort
,uniq
anddiff
(orkompare
ork3diff
or any other file comparison tool)1235
as output, it seems to be the lone duplicate. In other words, I can't see an issue with the result here. Of course if the data is broken, like CRLF line endings infile1
but not infile2
, then you'd have problems.