Making an ugly script that compiles a bunch of hosts files into one big list.
Most lists will have two entries for a URL, one with
www. and one without like this:
127.0.0.1 facebook.com 127.0.0.1 www.facebook.com
However some of my lists don't double up like that, and I want them to. I have a lot of straggling URLs like this:
127.0.0.1 twitter.com 127.0.0.1 www.facebook.com 127.0.0.1 subdomain.facebook.com 127.0.0.1 tumblr.com 127.0.0.1 www.reddit.com
What I'd like to do is scan the file for URLs that are listed to verify all lines without
www. have a matching line with
www., and that all lines with a
www. have a matching like without the
www.. Then I want to append any missing lines to the list so my list would look like this:
127.0.0.1 twitter.com 127.0.0.1 www.twitter.com 127.0.0.1 facebook.com 127.0.0.1 subdomain.facebook.com 127.0.0.1 www.facebook.com 127.0.0.1 www.subdomain.facebook.com 127.0.0.1 tumblr.com 127.0.0.1 www.tumblr.com 127.0.0.1 reddit.com 127.0.0.1 www.reddit.com
It doesn't need to be in order, I can just append all the missing lines on the end of the file because I run
sort -u as a final step.