How can grep return a different number of matches?

Question

I need to grep a file (24 lines) into two files (which are kind of big).

The original files contains 24 lines that resemble this string:

AATGGACGCTTAAC[A|C|T][A|C|G]CGGC[A|T]TCGGAT

I do the following:

grep -f aList hugeFile_N*.csv | wc -l
4396868

Then, to double check that everything went fine, I split the file in two files and I use both of them through another grep:

cat aList | head -n 12 > firstHalf
cat aList | tail -n +13 > secondHalf
grep -f firstHalf hugeFile_N*.csv | wc -l
2169008
grep -f secondHalf hugeFile_N*.csv | wc -l
2228046

Now, I was expecting that the total number of matches from the two separately grepped files would be equal to the number of matches I found for the original file. But, as you can see:

2169008 + 2228046 = 4397054
4397054 != 4396868

This is not the case. We have 186 matches that are missing. What is going on here?

I also did some investigations into the (actually very simple) files. Here I grep the results from the two separate halves:

grep -f <(cat firstHalf secondHalf) hugeFile_N*.csv > together

Then I grep the two halves separately:

grep -f firstHalf hugeFile_N*.csv > separately
grep -f secondHalf hugeFile_N*.csv >> separately

And, as already shown, the number of matches is different:

wc -l together
4396868 together
wc -l separately
4397054 separately

However, the number of unique matches is the same:

sort -u together | wc -l
3735836
sort -u separately | wc -l
3735836

Interestingly, if I grep the two halves in the original file, I have no matches at all. I wonder why this is happening:

grep -f ../code/firstHalf ../code/aList | wc -l
0
grep -f ../code/secondHalf ../code/aList | wc -l
0

I am 100% sure that both halves are present in aList since I can see it with my editor (they are 12 lines each, so it's not even that difficult to see by naked eyes).

I feel I am doing something wrong with the grep but...what?

All the lines in aList are unique.

Answer 1

grep only looks for lines that match at least one of the patterns given. With 24 patterns in aList, it's possible that one from the first half and one from the second half match. That would mean you'd get matches on the same line for both grep -f firstHalf and grep -f secondHalf. Running the two halves of the pattern list separately would double count those lines.

E.g.

$ cat test.txt 
abc 
foo
bar
foobar
$ cat patterns 
foo
bar
$ grep -c -f patterns  test.txt 
3

but of course also:

$ grep -c -e foo test.txt
2
$ grep -c -e bar test.txt
2

And 2+2 > 3.

If all the lines are distinct, counting unique matching lines would of course be a way to remove that effect. You could use grep -n to add line numbers to the output, making each output line unique. Of course remember that by default grep looks for matches anywhere on the line, and if that's not what you want, you need to use grep -x.

Note also that [A|C|T] means to match any of the characters A, C, T or |. If you don't want or need to match the pipe character, use just [ACT]. Or, if you need alternation, you have to use extended regexes (grep -E), and then (this|that) (with parenthesis, not brackets). But that's not necessary as long as all the alternatives are single characters.