0

I have a list of about 23 million hashes in a text file, one hash per line. Call it hashes.txt I have a list of nuisance hash values (app 40) in another text file, again one hash per line. Call it culls.txt

I'm attempting to use the patterns in culls.txt to do a reverse grep, which is to say I want to output any line that does not match any of the patterns in the cull list.

grep -v -F -f culls.txt hashes.txt > Output.txt

What is happening as that only the very last hash in the list of 40 or so it removed. All hashes on the preceding lines remain. I'm stymied. Suggestions? And, yes, the values are in the hash file ;->

3

The fact that only the last hash is recognised could point to a line ending issue on all but the last line.

I've tested this on my system (Ubuntu 14.04; grep (GNU grep) 2.16) with the following files:

$ cat culls.txt 
h00
h10
h20
h30
h40
$ cat culls2.txt 
h00
h10
h20
h30
h40
$ cat hashes.txt 
h04
h11
h13
h30
h61
h40
h41
h39
h42
$ file culls.txt hashes.txt  culls2.txt
culls.txt:  ASCII text
hashes.txt: ASCII text
culls2.txt: ASCII text, with CRLF line terminators

The output I get using culls.txt is as expected (h30 and h40 removed from the output):

$ grep -v -F -f culls.txt hashes.txt
h04
h11
h13
h61
h41
h39
h42

The output I get using culls2.txt is similar to what you see (only h40 removed from the output; h30 still there):

$ grep -v -F -f culls2.txt hashes.txt
h04
h11
h13
h30
h61
h41
h39
h42

There are no line endings on the last line of culls2.txt.

When I also added a CRLF line terminator to the last line I got:

$ grep -v -F -f culls3.txt hashes.txt
h04
h11
h13
h30
h61
h40
h41
h39
h42

Now no of the hashes are stripped from the output.

That proves that the CRLF line terminators are the problem.

  • Thank you... it was indeed a CTLF issue. The culls.txt file came over from a Windoze box.. Saving the file to a Unix LF did the trick... The hashes.txt file had 23,592,956 lines of hashes. After culling out the 34 hashes in the culls.txt file, 23,592,922 remained. Perfect... Thanks for the 'fix'.... – Steve B Nov 16 '15 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.