Delete lines from one file if they contain a regex of content in another file

Question

I have a large file (150K+ lines) containing DNS records, I'll call that FileA. I have a smaller file containing certain names, FileB. I want to delete all lines in FileA that end with what's in FileB. But I do not want to delete the line if the name appears at the beginning of the record.

I know how to use grep -v name$ to manually remove the names that appear at the end of the record, but I need a loop to go through the whole FileB. So far my attempts have failed.

Here's an example I hope illustrates what I'm after:

FileA:
hosta IN A 10.20.30.40
hostb IN A 20.30.40.50
myurl IN CNAME hostb
yours IN CNAME hostb

If FileB contains hostb, only the last two lines are deleted; the first two lines are left as is.

Answer 1

You can use the -f flag to grep for all lines in FileB:

grep -v -f FileB FileA

This is almost what you want. But this removes also lines where the patterns from FileB are not at the end, and you explicitly stated that it should only match when it is at the end. So we need to modify FileB accordingly. We can use sed, to add the regex for the end of line, which is the $ sign:

sed 's/$/$/' FileB

This looks like it's substituting nothing, but it is in fact adding a $ at every end of line.

Now we can use process substitution to put all of this together:

grep -v -f <(sed 's/$/$/' FileB) FileA

Answer 2

To loop entire FileB you can use approach like:

for i in `cat FileB`
do
<do your work>
done

But maybe will be better to filter also by CNAME

Answer 3

With sed you can do it like this:

sed -n '/^[^ ]*$/{H;d;};G;/ \(.*\)\n.*\n\1/d;P' fileB fileA

This is: First process fileB. If the lines contain no spaces, they are supposed to be hostnames in fileB and we add them to the hold space and go on ({H;d;})

All other lines are supposed to be of fileA. By appending the hold space with the host name list (G), all lines with the last word of the line that get repeated in host name list can be deleted. The remaining lines are printed without the trailing list (P). Option -n supresses default output.

Edit: Actually one should do

sed -n '/^[^ ]*$/{H;d;};G;/ \(.*\)\n.*\n\1\n/d;/ \(.*\)\n.*\n\1$/d;P' fileB fileA

to avoid special cases.