Making an ugly script that compiles a bunch of hosts files into one big list.

Most lists will have two entries for a URL, one with www. and one without like this: facebook.com www.facebook.com

However some of my lists don't double up like that, and I want them to. I have a lot of straggling URLs like this: twitter.com www.facebook.com subdomain.facebook.com tumblr.com www.reddit.com

What I'd like to do is scan the file for URLs that are listed to verify all lines without www. have a matching line with www., and that all lines with a www. have a matching like without the www.. Then I want to append any missing lines to the list so my list would look like this: twitter.com www.twitter.com facebook.com subdomain.facebook.com www.facebook.com www.subdomain.facebook.com tumblr.com www.tumblr.com reddit.com www.reddit.com

It doesn't need to be in order, I can just append all the missing lines on the end of the file because I run sort -u as a final step.


2 Answers 2


You could just skip the checking and add the pair line anyway then pipe the output to sort -u:

sed '/ www./{                  # if line matches  www.
h                              # copy it over the hold buffer
s// /                          # remove the www. part
G                              # append the original line to the modified one
//!{                           # if line doesn't match  www.
h                              # copy it over the hold buffer
s/ / www./                     # add the www. part
G                              # append the original line to the modified one
' infile | sort -u

This assumes the two columns (IP and URL) are delimited by a space. Adjust the regex if needed.

Or, with zsh, you could read the lines in an array, remove the www. part where present and store the result in an array of unique elements then print each element twice, once as is and once with the www. part added:

for i (${(u)mylist[@]/ www./ }) printf '%s\n' ${i} ${i/ / www.}
  • 1
    that's almost exactly the thought i had as well before reading this. vaguely I was considering: sed -Ee's/[[:blank:]]+(www\.)?/ www./p;s// /' | sort -u
    – mikeserv
    Jan 21, 2016 at 22:00
  • @mikeserv - yep; I like yours better. Jan 21, 2016 at 23:00

You can use the following awk script:


    gsub("www.", "", $2)
    for (item in address){
        print("", item)
        print("", "www." item)

Invoke it with

awk -f script.awk input

It removes www. at the beginning from all entries and then fills an array (with no value). If the value is already there, it will simply overwrite it, so you don't have to care about duplicates. In the end we print the content of the array, once as it is, and once with the www. prefix.

  • Am I correct in assuming I could change the two print commands to include " " and pipe them into a new file with >>? It looks like it's just printing it to the console without the localhost address as is.
    – deskjockey
    Jan 21, 2016 at 21:38
  • Sorry, forgot about the I edited my post. Yes, you can pipe the output to a file.
    – pfnuesel
    Jan 21, 2016 at 21:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .