-1

I have a large file that contains domain names in the form of:

domain.com
sub.domain.com
sub.domain.co.uk
domain.co.uk

I want to extract main domain names (no sub domains) with top level domain name (e.g. .com) or with country code top level domain name.

The top level domain name is always between 2-3 letters (e.g. .com, .net, .gov)

The country code top level domain name is always 2 letters (e.g. .uk, .us) and comes at the end of line.

So if the above list in an input, the output should extract:

domain.com
domain.co.uk

I tried this expression:

grep -P '^[^\.]+\.[a-zA-Z]{2,3}\.[a-zA-Z]{2}$

This is my interpretation. -P: perl regex ^: beginning of line ^\.: exclude dot +: one or more times \.: dot [a-zA-Z]{2,3}: two or three alphabetical characters (e.g., .com, .co) [a-zA-Z]{2}$: two alphabetical characters at the end of the line

My questions: The output I get always extracts:

domain.co.uk

But not domain.com

How to make my regex extracts domain names with or without country code top level domain names like domain.com and domain.co.uk BUT without subdomains like sub.domain.co.uk or sub.domain.com

closed as unclear what you're asking by αғsнιη, Kusalananda, Thomas, Jesse_b, G-Man Jul 30 '18 at 20:26

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 3
    How are you going to distinguish "domain.ltd.uk" (first-level) and "subdomain.bbc.uk" (second-level)? – Michael Homer Jul 29 '18 at 5:23
2

your question is ambiguous. if your definition of domain only item like you mentioned, to find them you could use:

grep -P "^.[^.]+\.[a-zA-Z]{3}$|^.[^.]+\.[a-zA-Z]{2}\.[a-zA-Z]{2}$" FileName
  • grep -P use Perl regex
  • ^.[^.]+ start with every char except and not contain . as much would
  • \.[a-zA-Z]{3}$ occures . follow with 3 chars in the end
  • | OR
  • ^.[^.]+ like above
  • \.[a-zA-Z]{2} occures 2 chars two times twise in the end
1

Given the way TLDs & FLDs get dolled out by registrars this is a non-trivial problem that I don't think you'll be able to tackle with simple regexes and CLI tools.

I'd lean on something like this Python module, tld. This module has both a get_tld and get_fld function. The 2nd one will print first level domains, which is what you're looking for.

Example

$ cat fld.py
#!/bin/python

from tld import get_fld

fldList = []
domList = open("domlist.txt").read().splitlines()
for dom in domList:
  fldList.append(get_fld(dom, fix_protocol=True))

print("\n".join(sorted(set(fldList))))

Sample run:

$ ./gtld.py
domain.co.uk
domain.com

NOTE: The list of domains is in a file called domlist.txt.

References

Not the answer you're looking for? Browse other questions tagged or ask your own question.