0

RHEL6.10 , gnu grep 2.2

grep -vfF stem.txt source.txt > filteredoutput.txt

stem.txt has unique 13 digit numbers.

source.txt is fixed length (400) data file where the 13 digit field value starts at position 225. The field values are unique in the file. Newline character is the line separator in both cases.

source.txt has about 80k lines, stem.txt about 4.5k

HOWEVER, my output file has around 75k lines, meaning it has filtered more than in the stem file.

I could not figure out why this happens.

While troubleshooting, I replaced everything else in each line in source.txt except the number field with spaces and then it gives the correct result. Very confusing behavior.

  1. Not sure how I can figure it out?
  2. What are my other options -sed /awk equivalents.

EDIT: Hail grep! Issue was with the user/me. Due to a oversight I did not account for the possibility of a special sequence of 13 digits that might appear elsewhere in the source file. The grep command with its arguments worked as expected.

5
  • It is possible there are more occurences starting from other positions than 225, so a few more lines are excluded. You could test running your grep without -v and see if it's output is more lines than stem.txt.
    – thanasisp
    May 4 at 14:29
  • 2
    can you edit the question to show some representative sample of what the files have? or shortened but complete samples and the associated expected output?
    – ilkkachu
    May 4 at 14:41
  • 1
    To add a couple of links to what @ikkachu said - please edit your question to include a Minimal, Reproducible Example with concise, testable sample input, expected output and your attempt to solve the problem yourself so we can help you further. See How to Ask and look at existing questions that have been upvoted and answered for examples.
    – Ed Morton
    May 4 at 18:55
  • @thanasisp I lied. Due to my oversight there is a possibility that the sequence of digits in one of the special IDs can repeat elsewhere. That was the issue.
    – Brian
    May 5 at 7:56
  • @Brian good news, it's solved then. This was my first thought, it should be a data issue, see my comment above. You just have to anchor your matching at the specific character position. There are many ways to do it, I would use awk for this also. Good day.
    – thanasisp
    May 5 at 8:09

2 Answers 2

4

With awk, making sure you only look for those strings wherever they're meant to be.

awk '!source {stem[$0]; next}
     ! (substr($0, 225, 13) in stem)
     ' stem.txt source=1 source.txt > filteredoutput.txt

(substr() start position is 1-based).

That will also make it a lot more efficient as it only needs to do one hash lookup per line of source.txt rather than 4.5k substring lookups.

Note that

grep -vfF stem.txt source.txt

is

grep -v -f F stem.txt source.txt

(looks for lines of stem.txt and source.txt not matching any of the regexps in the F file).

You probably meant:

grep -vFf stem.txt source.txt

(look for lines of source.txt not containing any of the strings in stem.txt).

1
  • 1
    Well spotted on the grep arguments order! Hopefully the OP will understand that the awk solution is much more robust/
    – Ed Morton
    May 4 at 18:58
2

Another way is to create dynamically the EREs, to start matching at the 225th position, with some help from sed, and grep for these EREs:

grep -vEf <(sed 's/.*/^.{224}&/' stem.txt) source.txt

So the matching is required to start at the specific position, preventing any other possible matches in random positions from impacting the output.

You could also test if there are matches in other positions by running your grep without -v and comparing the result to stem.txt.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.