I have a large file (150K+ lines) containing DNS records, I'll call that FileA. I have a smaller file containing certain names, FileB. I want to delete all lines in FileA that end with what's in FileB. But I do not want to delete the line if the name appears at the beginning of the record.

I know how to use grep -v name$ to manually remove the names that appear at the end of the record, but I need a loop to go through the whole FileB. So far my attempts have failed.

Here's an example I hope illustrates what I'm after:

hosta IN A
hostb IN A
myurl IN CNAME hostb
yours IN CNAME hostb

If FileB contains hostb, only the last two lines are deleted; the first two lines are left as is.


You can use the -f flag to grep for all lines in FileB:

grep -v -f FileB FileA

This is almost what you want. But this removes also lines where the patterns from FileB are not at the end, and you explicitly stated that it should only match when it is at the end. So we need to modify FileB accordingly. We can use sed, to add the regex for the end of line, which is the $ sign:

sed 's/$/$/' FileB

This looks like it's substituting nothing, but it is in fact adding a $ at every end of line.

Now we can use process substitution to put all of this together:

grep -v -f <(sed 's/$/$/' FileB) FileA
  • Better also add a whitespace at the beginning of the line or abar line in FileB will also remove a FileA line ending in foobar – Philippos Jun 29 '17 at 8:05
  • 1
    Should add: it also completed instantly. – Piotr Jun 29 '17 at 19:41

To loop entire FileB you can use approach like:

for i in `cat FileB`
<do your work>

But maybe will be better to filter also by CNAME


With sed you can do it like this:

sed -n '/^[^ ]*$/{H;d;};G;/ \(.*\)\n.*\n\1/d;P' fileB fileA

This is: First process fileB. If the lines contain no spaces, they are supposed to be hostnames in fileB and we add them to the hold space and go on ({H;d;})

All other lines are supposed to be of fileA. By appending the hold space with the host name list (G), all lines with the last word of the line that get repeated in host name list can be deleted. The remaining lines are printed without the trailing list (P). Option -n supresses default output.

Edit: Actually one should do

sed -n '/^[^ ]*$/{H;d;};G;/ \(.*\)\n.*\n\1\n/d;/ \(.*\)\n.*\n\1$/d;P' fileB fileA

to avoid special cases.

  • Seems overly complex, but I did try it. After more than 30 minutes of pegging the CPU, it was still churning before I killed it. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5593 piotr 20 0 1019m 961m 656 R 100.0 12.0 13:34.29 sed – Piotr Jun 29 '17 at 19:18
  • So I suppose that fileB is not that small? Execution time will grow linear with fileA, but Regular expressions with back references can slow down exponentially with length of fileB. Using \([^ \n]*\) instead of \(.*\) should drastically reduce execution time. If you are still interested, I can explain. – Philippos Jun 29 '17 at 22:04
  • No, I did test run with FileB only 76 records. Thanks for the help anyway, some aspects of sed I've never played with. – Piotr Jun 29 '17 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.