0

I've got 2 files, both with numerous lines that only contain one number. I'm trying to see if any number from file1 matches a number in file2. This is what I tried, and for some reason it doesn't work:

for i in $(cat file1); do grep ${i} file2; done

Fore reference here is data from file1 and file2

file1   file2
2134    1251
2135    5626
5342    4327
6456    8453
3413    4537
4525    3533
2347    5738
1235    1235
7453    3462

So shouldn't this command take each line from file 1 and grep it against the whole of file2? In that case, shouldn't a match be printed on screen?

  • It should - but you would probably be advised to use something like grep -Fwf file1 file2 instead – steeldriver Mar 10 at 17:51
  • It should but it consistently doesn't, even if I do it on the same file like for i in $(cat file1); do grep ${i} file1; done it still doesn't work. I'll try your advice – user323587 Mar 10 at 17:54
  • Notice: I just tryed your code and works for me. Is there any chance that file1 contains hidden characters? ... may be \0 or \r or tabs? – Juan Mar 10 at 17:55
  • if what you really wants is to compare the files, you might want to use sort, uniq and diff (or kompare or k3diff or any other file comparison tool) – Juan Mar 10 at 17:58
  • With those columns of numbers in two files, and that command, I get the 1235 as output, it seems to be the lone duplicate. In other words, I can't see an issue with the result here. Of course if the data is broken, like CRLF line endings in file1 but not in file2, then you'd have problems. – ilkkachu Mar 10 at 20:15
0

Given two ordinary Unix text files, your shell loop prints

1235

since this is the line that occurs in both files. If it does not, then one of your files may be a DOS text file. You can convert DOS text files into Unix text files with the dos2unix utility.

There is nothing major wrong with your loop given the type of data that you have, apart from the fact that it calls grep once for every line in file1. It also would match substrings, for example 100 in 1001, and it would, if any line in file1 contained spaces or tabs, split these lines into multiple words (due to the for i in $(cat ...) where the $(cat ...) is unquoted).

If you want to solve your issue this way (with a loop), you would better do

while IFS= read -r word; do
    grep -xF -e "$word" file2
done <file1

The -x and -F are explained later in my answer, and -e signifies that the next argument is the pattern to match with (otherwise, it may be taken as a command line option if it starts with a dash (-).

This would still execute grep once for each line in file1, but it would do it correctly.


To extract lines in file2 that exactly correspond to line in file1, without using a shell loop, you would use

$ grep -xF -f file1 file2
1235

This is assuming that file1 contains a reasonable number of lines, but not too many ("too many" will depend on the amount of memory that you have).

The command uses grep with -x, which forces matches across full lines only (no substring matches), and with -F which changes grep to do string comparisons rather than regular expression matches.

The -f file1 instructs grep to read the patterns (the strings to match with) from file1.


For really massive amounts of data, it would be hugely inefficient to use grep though. Instead, for this task and with this type of data (single words on individual lines), it would be better to do a relational join operation between the files:

$ join file1 file2
1235

This would, assuming that both files are lexicographically sorted, return the numbers that are the same between both files.


Using comm:

$ comm -1 -2 file1 file2
1235

comm also compares sorted files and can easily handle very large datasets. It prints three columns by default:

  1. lines that occur in the first file only
  2. lines that occur in the second file only
  3. lines that occurs in both files

With -1 we turn off the output of the first column, and with -2 we disable the second column, leaving comm to only output the lines that are the same in both files.

1

You simply need to use grep -f file1 file2 OR you may also use cat file1 | grep -f /dev/stdin file2

  • 1
    Thank you for contributing. Please note that a) a good answer explains what you do, so others can not just use it, but learn from it and b) without specifying -x or -w to grep you can get unwanted results if not all numbers are four-digit numbers like in the example (like 234 in file1 ould match 1234 in file2). That was probably the reason for someone to downvote your answer (sadly, without leaving a comment) – Philippos Mar 10 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.