-1

I have a case similar to this, but it has some differences.

Content listA:

.co
.best.co
.com
.test.server.cloud.us-east.amazonaws.com
.com.co
.abc.com.co
.jp
.def.museum.hiroshima.jp
.net
.xyz.xxx.yyy.net
.exe
.xyz.exe
# and anything else i want to add

Content listB:

.bar
.co
.com
.server.cloud.us-east.amazonaws.com
.com.co
.jp
.museum.hiroshima.jp
.net
.xxx.yyy.net
# and anything else i want to add

What I want is to remove from the "listA", those lines that do not end in what appears in the "listB", and duplicates too

Desired output:

.best.co
.test.server.cloud.us-east.amazonaws.com
.abc.com.co
.def.museum.hiroshima.jp
.xyz.xxx.yyy.net

I tried some commands and it didn't work for me:

grep -vi -f <(sed 's:^\(.*\)$:\\\1\$:' listB ) listA > out
grep -v -f <(sed 's/$/$/' listB ) listA > out

thanks in advance

0
1
$ grep -x -f <(sed 's/\./\\./g;s/^/.*/' listB) <(grep -v -F -x -f listB listA)
.best.co
.test.server.cloud.us-east.amazonaws.com
.abc.com.co
.def.museum.hiroshima.jp
.xyz.xxx.yyy.net

Explanation as two commands:

1.) grep -v -F -x -f listB listA

Remove literal duplicates from listA and use this output as input of the second grep. This leaves these entries of listA:

.best.co
.test.server.cloud.us-east.amazonaws.com
.abc.com.co
.def.museum.hiroshima.jp
.xyz.xxx.yyy.net
.exe
.xyz.exe

(Remaining lines to remove: .exe and .xyz.exe)

2.) grep -x -f <(sed 's/\./\\./g;s/^/.*/' listB) <(...)

Escape the dots . in listB, add .* to the start and grep again to match those lines ending with a line in listB. Input is the result of the first grep.

4
  • Very good command. Congratulations. But it doesn't work for my purposes. Although I appreciate your participation and effort to help me. I do not need the command to verify the complete lines. Only the end of the lines of listA, should be in listB and if it is not, then delete them. The rest does not interest me. I can not put more examples because they can be many (no point at the beginning, or with host 000 or with comments, etc, etc). Your command removes any line that does not comply with the published format. And that does not relate to the question
    – ajcg
    Aug 23 '19 at 13:52
  • 1
    Updated the sed command to only match the end of the lines (removed the mandatory starting .), I hope that's what you meant. Maybe you shouldn't give an example with 12 lines all starting with a . if the line can start with any character.
    – Freddy
    Aug 23 '19 at 14:41
  • That is exactly what I meant and it works. Now i am testing with a larger list to verify if the published script or your command offers better results. I thank you very much for your enormous effort. Priceless
    – ajcg
    Aug 23 '19 at 21:40
  • works with a big list... and faster. THK
    – ajcg
    Aug 23 '19 at 22:17
0
# save valid extension from listB
vexts=($(sed "s/^.*\.//g" listB))
# loop over listA and filter desired output
while read line; do 
    if [[ " ${vexts[@]} " == *" ${line##*.} "* ]] ; then
        echo "${line}";
    fi
done < listA
3
  • need to add: done < listA | grep -Fvxf <(cat listB.txt) to delete duplicates in listA.
    – ajcg
    Aug 23 '19 at 13:32
  • 1
    you can also add : ... done < listA | sort | uniq
    – Yunus
    Aug 23 '19 at 18:49
  • Not Work: "done < listA | sort | uniq". Work: done < listA | grep -Fvxf <(cat listB.txt). Anyway i really appreciate your help. A bit sophisticated script (and very slow), but it works. Thanks for everything
    – ajcg
    Aug 23 '19 at 22:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.