1

Let's say I have an output file with a large quantity of URLs in it. How would I remove a part of the URL with awk or sed so that the final result would be http://example.com and not http://example.com/folder/file.html.

  • 3
    What does the input look like? one URL per line? delimited by spaces, pipes? – Jeff Schaller Oct 20 '17 at 19:20
1

To remove everything after the first lone (not double) /:

sed -r 's#([^/])/[^/].*#\1#'

The expression s#([^/])/[^/].*#\1# will perform a substitution using the regular expression ([^/])/[^/].*. This will match any character that is not a /, followed by a single / (and other characters to the end of the input line).

The substitution replaces the match with the character in front of the lone / (which otherwise would have been substituted away).

Testing it on the file

http://example.com/folder/file.html
http://example.bar.com/folder/file.html
http://example.com:8080/folder/file.html
$ sed -r 's#([^/])/[^/].*#\1#' file
http://example.com
http://example.bar.com
http://example.com:8080

Using awk:

$ awk -F'/' '{ print $1 "//" $3 }' file
http://example.com
http://example.bar.com
http://example.com:8080

Here, I'm using / as a field delimiter, and then I'm simply picking out the first and thrid field and outputting them with the string // in-between.

|improve this answer|||||
1

You could use sed:

 sed 's#^\(https\?://\)\?\([^/]*\)/.*#\1\2#' filename

With this command and the following file,

http://example.com/foo/bar.html
https://example.com/foo/bar.html
example.com/foo/bar.html

the output will be

http://example.com
https://example.com
example.com

The way it works is as follows

  • \(https\?://\)\? matches at most 1 occurence of either http:// or https://. The parenthesis capture the match in \1

  • \([^/]*\) matches anything after that until the next /, and the match is captured in \2

  • /.* matches the rest of the line.

|improve this answer|||||
0

TO use sed to strip trailing parts of URLs, assuming they are one-per-line, or space-separated:

sed 's!\(http://[^/]*\)[^ ]*!\1!g' < input > output

This uses sed's search & replace functionality; the delimiters between the pattern, replacement, and flags is set to ! here.

The pattern to match is:

  • http:// followed by anything except a forward-slash -- and this section of the pattern is captured in parenthesis as "group 1"
  • (anything except a space), zero or more times -- this is the trailing part of the URL, specifically not captured in the parenthesis

The replacement is:

  • the previously-captured "group 1"

The flags are:

  • do this globally, meaning as many times per line as possible
|improve this answer|||||
  • Why would you need the global flag? – Kusalananda Oct 20 '17 at 19:32
  • 1
    In case there were multiple URLs per line (not yet clarified by OP) – Jeff Schaller Oct 20 '17 at 19:32
0

For large files grep might get the job done pretty fast. Also easy to remember syntax.

grep -Eo '^https?://[^/]+' fname

Using sed doesn't require more than

sed -r 's|(https?://[^/]*).*|\1|' fname

or cut without using difficult regexes

cut -d'/' -f1,2,3 fname
|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.