I need to grep a file (24 lines) into two files (which are kind of big).
The original files contains 24 lines that resemble this string:
I do the following:
grep -f aList hugeFile_N*.csv | wc -l 4396868
Then, to double check that everything went fine, I split the file in two files and I use both of them through another grep:
cat aList | head -n 12 > firstHalf cat aList | tail -n +13 > secondHalf grep -f firstHalf hugeFile_N*.csv | wc -l 2169008 grep -f secondHalf hugeFile_N*.csv | wc -l 2228046
Now, I was expecting that the total number of matches from the two separately grepped files would be equal to the number of matches I found for the original file. But, as you can see:
2169008 + 2228046 = 4397054 4397054 != 4396868
This is not the case. We have 186 matches that are missing. What is going on here?
I also did some investigations into the (actually very simple) files. Here I grep the results from the two separate halves:
grep -f <(cat firstHalf secondHalf) hugeFile_N*.csv > together
Then I grep the two halves separately:
grep -f firstHalf hugeFile_N*.csv > separately grep -f secondHalf hugeFile_N*.csv >> separately
And, as already shown, the number of matches is different:
wc -l together 4396868 together wc -l separately 4397054 separately
However, the number of unique matches is the same:
sort -u together | wc -l 3735836 sort -u separately | wc -l 3735836
Interestingly, if I grep the two halves in the original file, I have no matches at all. I wonder why this is happening:
grep -f ../code/firstHalf ../code/aList | wc -l 0 grep -f ../code/secondHalf ../code/aList | wc -l 0
I am 100% sure that both halves are present in
aList since I can see it with my editor (they are 12 lines each, so it's not even that difficult to see by naked eyes).
I feel I am doing something wrong with the
All the lines in
aList are unique.