17

I'd like to have bash parse/extract a full URL (and only the url) from a random short string.

Examples:

bob, the address is http://www.google.com

or

https://foo.com/category/example.html is up

or

Error 123 occurred at http://bit.ly/~1223456677878

or

Stats are up: https://foo1234.net/report.jpg

I tried using cat foo_output | egrep -o "https?://[\w'-\.]*\s" but that didn't seem to work.

1
  • Sounds scary, depending on what you want to do with the extracted URL...
    – vonbrand
    Mar 4, 2014 at 17:28

6 Answers 6

31

Did you try:

egrep -o 'https?://[^ ]+' foo_output

instead?

Note that anything with a character class is taken as literal, so saying [\w] doesn't match a word character. Moreover, you don't need to escape a regex metacharacter within a character class, i.e, saying [\.] isn't quite the same as [.].

3
  • 2
    [^ ] is too wide, you'll want to exclude other blanks, (, ), possibly comas, and all the characters that are not allowed in URLs. Mar 4, 2014 at 10:13
  • @StephaneChazelas You're right. However, I assumed that the URL is preceded and followed by a space unless at the beginning or the end of line.
    – devnull
    Mar 4, 2014 at 10:18
  • need to add " to ignore list.
    – chovy
    Nov 21, 2021 at 10:41
6

URIs aren't well-suited for regular expression matching when embedded in natural language. However, the current state of the art is John Gruber's Improved Liberal, Accurate Regex Pattern for Matching URLs. As currently posted, the one-line version is as follows:

(?i)\b((?:[a-z][\w-]+:(?:/{1,3}|[a-z0-9%])|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'".,<>?«»“”‘’]))

John also appears to maintain a gist here, although his blog entry does a much better job of explaining his test corpus and the limitations of the regular expression pattern.

If you want to implement the expression from the command line, you may find yourself limited by the regular expression engine you're using or by shell quoting issues. I've found a Ruby script to be the best option, but your mileage may vary.

3
  • 2
    Please include the regex in your answer instead of linking to it.
    – terdon
    Mar 4, 2014 at 17:00
  • @terdon, the full regexp is some 60 lines.
    – vonbrand
    Mar 4, 2014 at 17:30
  • 2
    @vonbrand I know, I saw it. We just tend to avoid linking to external resources. The whole point of the SE sites is to be a wiki. What if the blog you linked to goes offline? Your answer will become useless. Anyway, 60 lines is not that much and it is only 60 lines for readability.
    – terdon
    Mar 4, 2014 at 17:32
2

The problem with matching URLs is that just about anything can be in a URL:

https://encrypted.google.com/search?hl=en&q=foo#hl=en&q=foo&tbs=qdr:w,sbd:1

As you can see, the (valid) URL above contains $,?,#,&,,,. and :. Basically, the only thing you can be sure a URL does not contain is a blank space. With that in mind, you could extract your URLs with as simple a pattern as:

$ grep -oP 'http.?://\S+' file 
http://www.google.com
https://foo.com/category/example.html
http://bit.ly/~1223456677878
https://foo1234.net/report.jpg

The \S matches any non-space characters in perl compatible regular expressions (PCREs), the -P activates PCREs for grep and the -o makes it print only the matched segment of the line.

0

Just egrep -o 'https?://[^ ")]+'

which will include url() and "http"

3
  • 3
    How is this different from the answer by devnull? I hope you realise that the use of egrep is deprecated.
    – Anthon
    Apr 26, 2016 at 20:25
  • If you have an improvement over an existing answer, you can refer back to via the "share" link under that answer. See also the help pages
    – Jeff Schaller
    Apr 26, 2016 at 20:48
  • this shows " at end of urls
    – chovy
    Nov 21, 2021 at 10:40
0

I would go for chaining but a bit different. If you have a text snippet like yours in a text file called strings.txt you can do as follows:

grep http ./strings.txt | sed 's/http/\nhttp/g' | grep ^http | sed 's/\(^http[^ <]*\)\(.*\)/\1/g' | grep IWANTthis | sort -u

Explanation:

grep http ./st3.txt      => will catch lines with http from text file
sed 's/http/\nhttp/g'    => will insert newline before each http
grep ^http               => will take only lines starting with http
sed 's/\(^http[^ <]*\)\(.*\)/\1/g'   
                         => will preserve string from ^http until first space or < (the latter in hope if 
grep IWANTthis           => will take only urls containing your text of your interest; you can omit this.
sort -u                  => will sort the list and remove duplicates from it 

As there is a chance the url might not work you could do additional error checking with your URL of interest. e.g. wget -p URL -O /dev/null - it will print quite different error codes in case the URL is not available, so you could set up a loop to process your list of links and output their validity status.

If you are ultimately extracting links from html files then there can be some trouble with sed in special cases. As it has been suggested in a funny (post) that you probably have seen already - it may be best not to use regexps but a html parser engine. One such easily available parser is the text only browser lynx (available on any linux). This allows you to instantly dump list of all links in a file and then you just extract the urls you want with grep.

lynx -dump -listonly myhtmlfile.html | grep IWANTthisString | sort -u

However this will not work on most mangled html files or text snippets with links.

-1
cat text-file.txt | grep -Eo '(https?|ftp|file)://[-A-Za-z0-9\+&@#/%?=~_|!:,.;]*[-A-Za-z0-9\+&@#/%=~_|]'

alternatively append SED command to store it to CSV file:

| sed 's/;/<tab>/g' > file.csv

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .