0

I have a large file that contains domain names in the form of:

domain.com
sub.domain.com
sub.domain.co.uk
domain.co.uk

I want to extract main domain names (no sub domains) with top level domain name (e.g. .com) or with country code top level domain name.

The top level domain name is always between 2-3 letters (e.g. .com, .net, .gov)

The country code top level domain name is always 2 letters (e.g. .uk, .us) and comes at the end of line.

So if the above list in an input, the output should extract:

domain.com
domain.co.uk

I tried this expression:

grep -P '^[^\.]+\.[a-zA-Z]{2,3}\.[a-zA-Z]{2}$

This is my interpretation. -P: perl regex ^: beginning of line [^\.]: exclude dot +: one or more times \.: dot [a-zA-Z]{2,3}: two or three alphabetical characters (e.g., .com, .co) [a-zA-Z]{2}$: two alphabetical characters at the end of the line

My questions: The output I get always extracts:

domain.co.uk

But not domain.com

How to make my regex extracts domain names with or without country code top level domain names like domain.com and domain.co.uk BUT without subdomains like sub.domain.co.uk or sub.domain.com

3
  • Given that domain.co.uk and domain.uk (and domain.co for that matter, but there a different TLCD) are equally valid and do not have to represent the same entity, how would you want this TLCD's domains identified? May 9, 2023 at 22:13
  • How about domain.us and sub.domain.us (or sub.d2.us) or domain.gov.uk? May 10, 2023 at 5:46
  • 1
    the direct answer to why it doesn't match: \.[a-zA-Z]{2}$ matches a literal dot and exactly two letters at the end, while domain.com has a dot and three letters at the end.
    – ilkkachu
    May 10, 2023 at 8:33

3 Answers 3

1

If you consider that top-level suffixes can be .us, .gov.uk, .tas.gov.au, .uk, the only valid approach is to hardcode the full list.

You can use the Domain::PublicSuffix perl module for that:

$ perl -MDomain::PublicSuffix -lne '
  BEGIN{$s = Domain::PublicSuffix->new}
  print if $_ eq $s->get_root_domain($_)' < your-file
domain.com
domain.co.uk

Here using the default list as shipped with the module, but you can also give it a more up-to-date list if need be as per its documentation.

On Debian, that module can be found in the libdomain-publicsuffix-perl package.

1

Try making the last country code TLD optional by using ? Here are some examples to do so.

Using extended-regex

grep -E '^[^.]+\.[a-zA-Z]{2,3}(\.[a-zA-Z]{2})?$'

or perl regex

grep -P '^[^.]+\.[a-zA-Z]{2,3}(?:\.[a-zA-Z]{2})?$'
0
0
grep -E "^[[:alnum:]-]+(([.][[:alpha:]]{2}){2}|[.][[:alpha:]]{3})$"

^[[:alnum:]-]+ one or more domain valid chars anchored at start

( start overarching group, first term follows

([.][[:alpha:]]{2}) sub group which begins with dot, followed by two alpha chars

{2} previous sub group repeats exactly twice

| delimit next term in over group

[.][[:alpha:]]{3} group term which begins with dot, followed by three alpha chars

)$ close over group, anchor to end of string

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .