Remove hostnames from URL with sed/awk

Question

I got lists of urls with different domains and I want the hostname removed with sed, awk or something similar and only keep the path. There are no urls with port or username@password in it.

input:

http://www.example.com/
https://www.example.com/
http://example.com/blog/
https://example.com/blog/
https://www.example.co.uk/blog/
https://example.co.uk/blog/
https://sub.example.co.uk/blog/
https://www.example.com/blog/
https://www.example.com/cases/page/4/
https://www.example.com/cdn-cgi/challenge-platform/h/g/cv/result/7c9123dc38da6841
https://www.example.com/cdn-cgi/challenge-platform/h/g/scripts/jsd/7fe83wdcs/invisible.js
https://www.example.co.uk/cdn-cgi/challenge-platform/h/g/scripts/jsd/7fe83wdcs/invisible.js
https://sub.example.co.uk/cdn-cgi/challenge-platform/h/g/scripts/jsd/7fe83wdcs/invisible.js

output should be:

/
/
/blog/
/blog/
/blog/
/blog/
/blog/
/blog/
/cases/page/4/
/cdn-cgi/challenge-platform/h/g/cv/result/7c9123dc38da6841
/cdn-cgi/challenge-platform/h/g/scripts/jsd/7fe83wdcs/invisible.js
/cdn-cgi/challenge-platform/h/g/scripts/jsd/7fe83wdcs/invisible.js
/cdn-cgi/challenge-platform/h/g/scripts/jsd/7fe83wdcs/invisible.js

I hope someone can help me because I can only find regex commands. I don't know how to convert these correctly to a sed or awk command

Answer 1

With perl:

perl -pe 's|^([^/:]+:)?//[^/]*||' < your-file

Would remove an optional scheme (to take care of both http://host/path and //host/path) followed by // followed by all the characters other than / that follow (would remove the host and also the user:password@host:8080 in ftp://user:password@host:8080/pub for instance).

A sed equivalent could be:

LC_ALL=C sed 's|^\([^/:]\{1,\}:\)\{0,1\}//[^/]*||' < your-file

In any case, the s/pattern/replacement/ operator of both sed and perl take regular expressions for the pattern, so called basic regular expressions for sed, perl regular expressions for perl (which improve and extend the extended regular expressions that many sed implementations also support with the -E option these days).

There's also a URI module for perl which you can use to parse a URI into a structured object.

perl -MURI -lpe '$_ = URI->new($_)->path' < your-file

Note that it discards the query string (as in http://host/path?query) and fragment (as in http://host/file.html#anchor) if any. Replace ->path with ->path_query if you want the query if any to be included.

Answer 2

This is fairly easy to do with linux coreutils:

cut -d '/' -f 3- somefilewithyoururls.txt | sed 's/^/\//'

Cut everything after the third /, then replace the start of the line with a /. No need for complicated regexes.

Answer 3

Using any sed:

$ sed 's:[^/]*//[^/]*::' file
/
/
/blog/
/blog/
/blog/
/blog/
/blog/
/blog/
/cases/page/4/
/cdn-cgi/challenge-platform/h/g/cv/result/7c9123dc38da6841
/cdn-cgi/challenge-platform/h/g/scripts/jsd/7fe83wdcs/invisible.js
/cdn-cgi/challenge-platform/h/g/scripts/jsd/7fe83wdcs/invisible.js
/cdn-cgi/challenge-platform/h/g/scripts/jsd/7fe83wdcs/invisible.js

Answer 4

Using Raku (formerly known as Perl_6)

~$ raku -MURL -ne 'my $url = URL.new($_); put "/" ~ .path.join("/") for $url;'  file

Sample Output:

/
/
/blog
/blog
/blog
/blog
/blog
/blog
/cases/page/4
/cdn-cgi/challenge-platform/h/g/cv/result/7c9123dc38da6841
/cdn-cgi/challenge-platform/h/g/scripts/jsd/7fe83wdcs/invisible.js
/cdn-cgi/challenge-platform/h/g/scripts/jsd/7fe83wdcs/invisible.js
/cdn-cgi/challenge-platform/h/g/scripts/jsd/7fe83wdcs/invisible.js

For Raku, loading the URL module might be the cleanest answer, as it can handle usernames/passwords in the url. Above, the recognized path element is both preceded-by and joined-on / forward_slash, then output.

Simplifying the code above lets you know what elements are recognized:

~$ raku -MURL -ne 'my $url = URL.new($_); .raku.put for $url;'  file
URL.new(scheme => "http", username => Str, password => Str, hostname => "www.example.com", port => Int, path => [], query => {}, fragment => Str)
URL.new(scheme => "https", username => Str, password => Str, hostname => "www.example.com", port => Int, path => [], query => {}, fragment => Str)
URL.new(scheme => "http", username => Str, password => Str, hostname => "example.com", port => Int, path => ["blog"], query => {}, fragment => Str)
URL.new(scheme => "https", username => Str, password => Str, hostname => "example.com", port => Int, path => ["blog"], query => {}, fragment => Str)
URL.new(scheme => "https", username => Str, password => Str, hostname => "www.example.co.uk", port => Int, path => ["blog"], query => {}, fragment => Str)
URL.new(scheme => "https", username => Str, password => Str, hostname => "example.co.uk", port => Int, path => ["blog"], query => {}, fragment => Str)
URL.new(scheme => "https", username => Str, password => Str, hostname => "sub.example.co.uk", port => Int, path => ["blog"], query => {}, fragment => Str)
URL.new(scheme => "https", username => Str, password => Str, hostname => "www.example.com", port => Int, path => ["blog"], query => {}, fragment => Str)
URL.new(scheme => "https", username => Str, password => Str, hostname => "www.example.com", port => Int, path => ["cases", "page", "4"], query => {}, fragment => Str)
URL.new(scheme => "https", username => Str, password => Str, hostname => "www.example.com", port => Int, path => ["cdn-cgi", "challenge-platform", "h", "g", "cv", "result", "7c9123dc38da6841"], query => {}, fragment => Str)
URL.new(scheme => "https", username => Str, password => Str, hostname => "www.example.com", port => Int, path => ["cdn-cgi", "challenge-platform", "h", "g", "scripts", "jsd", "7fe83wdcs", "invisible.js"], query => {}, fragment => Str)
URL.new(scheme => "https", username => Str, password => Str, hostname => "www.example.co.uk", port => Int, path => ["cdn-cgi", "challenge-platform", "h", "g", "scripts", "jsd", "7fe83wdcs", "invisible.js"], query => {}, fragment => Str)
URL.new(scheme => "https", username => Str, password => Str, hostname => "sub.example.co.uk", port => Int, path => ["cdn-cgi", "challenge-platform", "h", "g", "scripts", "jsd", "7fe83wdcs", "invisible.js"], query => {}, fragment => Str)

If you really dare to parse URLs with a regex (are you certain no data has been maliciously crafted?), then the following is a rather direct translation of the Perl answer posted by @Stéphane_Chazelas:

~$ raku -pe 's|^ ( <-[/:]>+ \: )? \/ \/ <-[/]>* ||;'  < file

https://raku.land/cpan:TYIL/URL
https://raku.org