1

Making an ugly script that compiles a bunch of hosts files into one big list.

Most lists will have two entries for a URL, one with www. and one without like this:

127.0.0.1 facebook.com
127.0.0.1 www.facebook.com

However some of my lists don't double up like that, and I want them to. I have a lot of straggling URLs like this:

127.0.0.1 twitter.com
127.0.0.1 www.facebook.com
127.0.0.1 subdomain.facebook.com
127.0.0.1 tumblr.com
127.0.0.1 www.reddit.com

What I'd like to do is scan the file for URLs that are listed to verify all lines without www. have a matching line with www., and that all lines with a www. have a matching like without the www.. Then I want to append any missing lines to the list so my list would look like this:

127.0.0.1 twitter.com
127.0.0.1 www.twitter.com
127.0.0.1 facebook.com
127.0.0.1 subdomain.facebook.com
127.0.0.1 www.facebook.com
127.0.0.1 www.subdomain.facebook.com
127.0.0.1 tumblr.com
127.0.0.1 www.tumblr.com
127.0.0.1 reddit.com
127.0.0.1 www.reddit.com

It doesn't need to be in order, I can just append all the missing lines on the end of the file because I run sort -u as a final step.

1

You could just skip the checking and add the pair line anyway then pipe the output to sort -u:

sed '/ www./{                  # if line matches  www.
h                              # copy it over the hold buffer
s// /                          # remove the www. part
G                              # append the original line to the modified one
}
//!{                           # if line doesn't match  www.
h                              # copy it over the hold buffer
s/ / www./                     # add the www. part
G                              # append the original line to the modified one
}
' infile | sort -u

This assumes the two columns (IP and URL) are delimited by a space. Adjust the regex if needed.


Or, with zsh, you could read the lines in an array, remove the www. part where present and store the result in an array of unique elements then print each element twice, once as is and once with the www. part added:

mylist=(${(f)"$(<infile)"})
for i (${(u)mylist[@]/ www./ }) printf '%s\n' ${i} ${i/ / www.}
  • 1
    that's almost exactly the thought i had as well before reading this. vaguely I was considering: sed -Ee's/[[:blank:]]+(www\.)?/ www./p;s// /' | sort -u – mikeserv Jan 21 '16 at 22:00
  • @mikeserv - yep; I like yours better. – don_crissti Jan 21 '16 at 23:00
0

You can use the following awk script:

script.awk:

{
    gsub("www.", "", $2)
    address[$2]
}
END{
    for (item in address){
        print("127.0.0.1", item)
        print("127.0.0.1", "www." item)
    }
}

Invoke it with

awk -f script.awk input

It removes www. at the beginning from all entries and then fills an array (with no value). If the value is already there, it will simply overwrite it, so you don't have to care about duplicates. In the end we print the content of the array, once as it is, and once with the www. prefix.

  • Am I correct in assuming I could change the two print commands to include "127.0.0.1 " and pipe them into a new file with >>? It looks like it's just printing it to the console without the localhost address as is. – deskjockey Jan 21 '16 at 21:38
  • Sorry, forgot about the 127.0.0.1. I edited my post. Yes, you can pipe the output to a file. – pfnuesel Jan 21 '16 at 21:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.