sed or other equivalent for grep with vfF

Question

RHEL6.10 , gnu grep 2.2

grep -vfF stem.txt source.txt > filteredoutput.txt

stem.txt has unique 13 digit numbers.

source.txt is fixed length (400) data file where the 13 digit field value starts at position 225. The field values are unique in the file. Newline character is the line separator in both cases.

source.txt has about 80k lines, stem.txt about 4.5k

HOWEVER, my output file has around 75k lines, meaning it has filtered more than in the stem file.

I could not figure out why this happens.

While troubleshooting, I replaced everything else in each line in source.txt except the number field with spaces and then it gives the correct result. Very confusing behavior.

1. Not sure how I can figure it out?
2. What are my other options -sed /awk equivalents.

EDIT: Hail grep! Issue was with the user/me. Due to a oversight I did not account for the possibility of a special sequence of 13 digits that might appear elsewhere in the source file. The grep command with its arguments worked as expected.

Answer 1

With awk, making sure you only look for those strings wherever they're meant to be.

awk '!source {stem[$0]; next}
     ! (substr($0, 225, 13) in stem)
     ' stem.txt source=1 source.txt > filteredoutput.txt

(substr() start position is 1-based).

That will also make it a lot more efficient as it only needs to do one hash lookup per line of source.txt rather than 4.5k substring lookups.

Note that

grep -vfF stem.txt source.txt

is

grep -v -f F stem.txt source.txt

(looks for lines of stem.txt and source.txt not matching any of the regexps in the F file).

You probably meant:

grep -vFf stem.txt source.txt

(look for lines of source.txt not containing any of the strings in stem.txt).

Answer 2

Another way is to create dynamically the EREs, to start matching at the 225th position, with some help from sed, and grep for these EREs:

grep -vEf <(sed 's/.*/^.{224}&/' stem.txt) source.txt

So the matching is required to start at the specific position, preventing any other possible matches in random positions from impacting the output.

You could also test if there are matches in other positions by running your grep without -v and comparing the result to stem.txt.