2

I need to grep a file (24 lines) into two files (which are kind of big).

The original files contains 24 lines that resemble this string:

AATGGACGCTTAAC[A|C|T][A|C|G]CGGC[A|T]TCGGAT

I do the following:

grep -f aList hugeFile_N*.csv | wc -l
4396868

Then, to double check that everything went fine, I split the file in two files and I use both of them through another grep:

cat aList | head -n 12 > firstHalf
cat aList | tail -n +13 > secondHalf
grep -f firstHalf hugeFile_N*.csv | wc -l
2169008
grep -f secondHalf hugeFile_N*.csv | wc -l
2228046

Now, I was expecting that the total number of matches from the two separately grepped files would be equal to the number of matches I found for the original file. But, as you can see:

2169008 + 2228046 = 4397054
4397054 != 4396868

This is not the case. We have 186 matches that are missing. What is going on here?

I also did some investigations into the (actually very simple) files. Here I grep the results from the two separate halves:

grep -f <(cat firstHalf secondHalf) hugeFile_N*.csv > together

Then I grep the two halves separately:

grep -f firstHalf hugeFile_N*.csv > separately
grep -f secondHalf hugeFile_N*.csv >> separately

And, as already shown, the number of matches is different:

wc -l together
4396868 together
wc -l separately
4397054 separately

However, the number of unique matches is the same:

sort -u together | wc -l
3735836
sort -u separately | wc -l
3735836

Interestingly, if I grep the two halves in the original file, I have no matches at all. I wonder why this is happening:

grep -f ../code/firstHalf ../code/aList | wc -l
0
grep -f ../code/secondHalf ../code/aList | wc -l
0

I am 100% sure that both halves are present in aList since I can see it with my editor (they are 12 lines each, so it's not even that difficult to see by naked eyes).

I feel I am doing something wrong with the grep but...what?

All the lines in aList are unique.

1 Answer 1

13

grep only looks for lines that match at least one of the patterns given. With 24 patterns in aList, it's possible that one from the first half and one from the second half match. That would mean you'd get matches on the same line for both grep -f firstHalf and grep -f secondHalf. Running the two halves of the pattern list separately would double count those lines.

E.g.

$ cat test.txt 
abc 
foo
bar
foobar
$ cat patterns 
foo
bar
$ grep -c -f patterns  test.txt 
3

but of course also:

$ grep -c -e foo test.txt
2
$ grep -c -e bar test.txt
2

And 2+2 > 3.

If all the lines are distinct, counting unique matching lines would of course be a way to remove that effect. You could use grep -n to add line numbers to the output, making each output line unique. Of course remember that by default grep looks for matches anywhere on the line, and if that's not what you want, you need to use grep -x.

Note also that [A|C|T] means to match any of the characters A, C, T or |. If you don't want or need to match the pipe character, use just [ACT]. Or, if you need alternation, you have to use extended regexes (grep -E), and then (this|that) (with parenthesis, not brackets). But that's not necessary as long as all the alternatives are single characters.

6
  • sorry, I didn't say that all the lines in aList are unique! I am editing the question accordingly. Also, I don't understand what do you mean when you mention using grep -n.
    – gabt
    Commented Sep 2, 2021 at 11:27
  • 1
    @gabt, all patterns in the file might be unique, but they might still match the same lines in data. Like in my example, foo and bar are distinct, but both still match the line foobar.
    – ilkkachu
    Commented Sep 2, 2021 at 11:35
  • ok, I need to do some experiments in order to properly understand what is going on here. baseline is that I will not be able to compare the results from grepping the whole list with grepping the two halves. I have to consider them as two separate, different things and I have to expect different outcomes. Is this correct?
    – gabt
    Commented Sep 2, 2021 at 11:43
  • 3
    @gabt, well, in a way, yes. The lines in the file given by -f are patterns, not fixed strings. The patterns could match multiple lines, and multiple patterns could match the same line. So there's no 1:1 relationship. As for splitting the list, it's the same as asking a bunch of students how many have taken course A and how many have taken course B. The total of those numbers could be different from the number of students who have taken either A or B (i.e. have taken at least one of A and B). Indeed the total could be higher than the total number of students.
    – ilkkachu
    Commented Sep 2, 2021 at 11:51
  • 1
    E.g. the pattern GGC[A|T] (to take a short example) would match both GGCA and GGCT, but also ABCGGCAXYZ (since it's not locked to start/end of the line). And that could also be matched by AB[DEF]GG.
    – ilkkachu
    Commented Sep 2, 2021 at 11:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .