I have a text file with lines that are a mixture like this:

###  Comments

# Comments banana.domain.net          # comment orange.domain.net monkey.anotherdomain.net puffin.domainz.com
# monkey.anotherdomain.net   # comment orange.domain.net

How do I find the host.domain duplicates?

In this case, there are two: monkey.anotherdomain.net and orange.domain.net

Taking into account that..

  • Trailing comments after the entry need to be ignored, as they may not be on the duplicate.
  • If the line is commented out, the duplicate should still be found.
  • Differences in IP address should be ignored.
  • 2
    It seems to be a logic conflict to say that a commented-out line should be ignored, yet still included. – Jeff Schaller Aug 8 '15 at 1:52
  • @JeffSchaller I see what you mean, but I meant the hash at the front commenting the line out should be ignored. I've edited the question. – paradroid Aug 8 '15 at 2:02
  • 1
    omg, please tell me you're not distributing a hosts file that's so complex you have to write scripts to manage it. That's why we have DNS. – Wouter Verhelst Aug 8 '15 at 7:50
  • @WouterVerhelst It's a hosts file that is not to be used as a hosts file. It's a record set. – paradroid Aug 8 '15 at 12:39

This was a fun one.

First, we need to eliminate trailing comments, as in: banana.domain.net          # comment

We can do that with the following (assuming just spaces, no tabs):

sed 's/  *#.*//'

If you have tabs in your hosts file, maybe run this first:

tr '\t' ' '

Then we need to eliminate "comment out this line" comments, which I'm going to define as a single hash character preceding an ip address. We can remove those like this:

sed '/^#[0-9]/ s/^#//'

Putting the above together gets us:

###  Comments

# Comments banana.domain.net orange.domain.net monkey.anotherdomain.net puffin.domainz.com monkey.anotherdomain.net orange.domain.net

If we sort this on the second column (sort -k2), we get a list sorted by name: banana.domain.net
# Comments
###  Comments monkey.anotherdomain.net monkey.anotherdomain.net orange.domain.net orange.domain.net puffin.domainz.com

And now we can apply uniq to find duplicates, if we tell uniq to ignore the first field:

uniq -c -f 1

Which gives us:

  1 banana.domain.net
  1 # Comments
  1 ###  Comments
  2 monkey.anotherdomain.net
  2 orange.domain.net
  1 puffin.domainz.com

So if we look for lines with a count of 2 or higher, we have found our duplicates. Putting this all together we get:


tr '\t' ' ' |
sed '
    /^#[0-9]/ s/^#//
    s/  *#.*//
    /^ *$/ d
' |
sort -k2 |
uniq -f 1 -c |
awk '$1 > 1 {print}'

The final awk statement in the above script looks for lines where the count from uniq (field1 ) is > 1.

Running the above script looks like this.

  • why not just a quick and dirty: grep -i "[a-z]\\.[a-z]" the-file | awk '{print $2}'|sort | uniq -c|grep -v "1 " – Jeff Schaller Aug 8 '15 at 2:08
  • 1
    I started out on the supposition that he was going to want the ip addresses, and then went from there because I was having so much fun. And ended up discarding the ip addresses. You should post the simpler solution. – larsks Aug 8 '15 at 2:08

If actual comment lines like the ones you've depicted at the beginning of your data exist and are to be ignored, I don't see how point 2 can be satisfied without some kind of assumption around how commented lines that are material can be told apart from commented lines that are to be ignored. I've made the assumption that commented lines that are material contain a period in the second field.

awk 'NF && $2 ~ /[.]/{++a[$2]}; 
 END{for (k in a) if(a[k] > 1) print k}' file
sed 's/\(.\)#.*/\1/' file | cut -f 2 -d\ | sort | uniq -d
  • Remove comments that are preceded by anything sed 's/\(.\)#.*/\1/'
  • Filter just the second column cut -f 2 -d\
  • Sort the lines for the following comparison sort
  • And only print duplicates uniq -d

Note that unless it is a possibility to have a line like banana.domain.net#comment

You can simplify the above to:

cut -f 2 -d\  file | sort | uniq -d

as the comment would be considered a third field.


How about the short, easy and straight way?

awk '/#*\d/{print $2}' file | sort | uniq -d

This allows to report host.domain duplicates even if they are commented out by a hash sign (#).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.