21

I am trying to use grep and cut to extract URLs from an HTML file. The links look like:

<a href="http://examplewebsite.com/">

Other websites have .net, .gov, but I assume I could make the cut off point right before >. So I know I can use grep and cut somehow to cut off everything before http and after .com, but I have been stuck on it for a while.

  • I edited it. For some ignore the space between < and a, the HTML wouldn't show up without it. Thanks for catching that! – eltigre Jan 27 '15 at 4:46
  • Use code formatting (select the text and press Ctrl-K). Otherwise the <> forces it to be seen as an HTML tag. – muru Jan 27 '15 at 4:49
  • why not match the opening and ending quote of href parameter ? besides , i believe regular expressions are not best fit for html's . – 把友情留在无盐 Jan 27 '15 at 4:50
  • I want to write a command using specifically grep and cut to do it. I realize there are other ways, but I wanted to know about those. – eltigre Jan 27 '15 at 4:54
  • 9
    In general it's not a good idea to parse HTML with Regular Expressions, since HTML is not a regular language. If you can guarantee that the HTML you're parsing is fairly simple, and the stuff you're trying to extract is predictable you may be able to get away with it. But please see stackoverflow.com/a/1732454/4014959 – PM 2Ring Jan 27 '15 at 5:22
25

As I said in my comment, it's generally not a good idea to parse HTML with Regular Expressions, but you can sometimes get away with it if the HTML you're parsing is well-behaved.

In order to only get URLs that are in the href attribute of <a> elements, I find it easiest to do it in multiple stages. From your comments, it looks like you only want the top level domain, not the full URL. In that case you can use something like this:

grep -Eoi '<a [^>]+>' source.html |
grep -Eo 'href="[^\"]+"' | 
grep -Eo '(http|https)://[^/"]+'

where source.html is the file containing the HTML code to parse.

This code will print all top-level URLs that occur as the href attribute of any <a> elements in each line. The -i option to the first grep command is to ensure that it will work on both <a> and <A> elements. I guess you could also give -i to the 2nd grep to capture upper case HREF attributes, OTOH, I'd prefer to ignore such broken HTML. :)

To process the contents of http://google.com/

wget -qO- http://google.com/ |
grep -Eoi '<a [^>]+>' | 
grep -Eo 'href="[^\"]+"' | 
grep -Eo '(http|https)://[^/"]+'

output

http://www.google.com.au
http://maps.google.com.au
https://play.google.com
http://www.youtube.com
http://news.google.com.au
https://mail.google.com
https://drive.google.com
http://www.google.com.au
http://www.google.com.au
https://accounts.google.com
http://www.google.com.au
https://www.google.com
https://plus.google.com
http://www.google.com.au

My output is a little different from the other examples as I get redirected to the Australian Google page.

  • THANK YOU. Now this is exactly what I was looking for. This is the cleanest way to do it. – eltigre Jan 27 '15 at 6:17
  • @eltigre: My pleasure! But please take heed of the warning I linked to in my comment above. :) – PM 2Ring Jan 27 '15 at 6:18
  • I came to this question expecting easy points... and you'd already hit the nail on the head completely – Mark K Cowan Jan 27 '15 at 14:58
  • Thanks, @MarkKCowan. :) FWIW, I originally started writing an answer using awk, but then I decided that a grep-based solution would be easier to understand for those not familiar with awk. And anyway, the code above is shorter than my awk code. – PM 2Ring Jan 28 '15 at 3:28
  • 2
    @mavavilj: Because the OP only wanted the the top level domain, so after the :// we only accept chars before the first / or ". But if you want to see the full URL, change that command to grep -Eo '(http|https)://[^"]+. Another option for that line is grep -Eo '(http|https)://[^?"]+' which cuts off query options. However, that variation will still print URLs that are contained within another URL as a query parameter, but they'll be printed on a separate line. – PM 2Ring Nov 6 '15 at 15:14
25

Not sure if you are limited on tools:

But regex might not be the best way to go as mentioned, but here is an example that I put together:

cat urls.html | grep -Eo "(http|https)://[a-zA-Z0-9./?=_-]*" | sort -u
  • grep -E : is the same as egrep
  • grep -o : only outputs what has been grepped
  • (http|https) : is an either / or
  • a-z : is all lower case
  • A-Z : is all uper case
  • . : is dot
  • \?: is ?
  • *: is repeat the [...] group
  • uniq : will remove any duplicates

Output:

bob@bob-NE722:~s$  wget -qO- https://stackoverflow.com/ | grep -Eo "(http|https)://[a-zA-Z0-9./?=_-]*" | sort -u
https://stackauth.com
https://meta.stackoverflow.com
https://cdn.sstatic.net/Img/svg-icons
https://stackoverflow.com
https://www.stackoverflowbusiness.com/talent
https://www.stackoverflowbusiness.com/advertising
https://stackoverflow.com/users/login?ssrc=head
https://stackoverflow.com/users/signup?ssrc=head
https://stackoverflow.com
https://stackoverflow.com/help
https://chat.stackoverflow.com
https://meta.stackoverflow.com
...

You can also add in \d to catch other numeral types.

  • 2
    IRI regexes! Use one of them and scare the OP! :) – muru Jan 27 '15 at 5:05
  • 2
    @muru ... shivering I ... I don't know what to say. Are those even real!? – jmunsch Jan 27 '15 at 5:08
  • 4
    @jmunsch, uniq just remove adjacent duplicates. sort -u ? – JJoao Jan 27 '15 at 8:54
  • 1
    it works great, best answer!! – Gery Dec 25 '16 at 18:07
  • @JJoao is the source for sort -u faster than piping? Just a thought experiment, id have to look. But you are probably right, about the shell middleware. – jmunsch Oct 30 '17 at 7:18
9

If your grep supports Perl regexes:

grep -Po '(?<=href=")[^"]*(?=")'
  • (?<=href=") and (?=") are lookaround expressions for the href attribute. This needs the -P option.
  • -o prints the matching text.

For example:

$ curl -sL https://www.google.com | grep -Po '(?<=href=")[^"]*(?=")'
/search?
https://www.google.co.in/imghp?hl=en&tab=wi
https://maps.google.co.in/maps?hl=en&tab=wl
https://play.google.com/?hl=en&tab=w8
https://www.youtube.com/?gl=IN&tab=w1
https://news.google.co.in/nwshp?hl=en&tab=wn
...

As usual, there's no guarantee that these are valid URIs, or that the HTML you're parsing will be valid.

8

As a non-regex alternative, use pup:

pup 'a[href] attr{href}' < yourfile.html

Will find all a elements that have a href attribute, then display the value of the href attribute.

To install pup, you need Go (a programming language):

sudo apt-get install golang
sudo go get github.com/ericchiang/pup

The advantage of this solution is that it doesn't rely on the HTML being properly formatted.

  • 1
    +1 for pup, time to install that.... – Mark K Cowan Jan 27 '15 at 15:01
  • You can put them in file as well. pup 'a.classname[href] attr{href}' < tut.html >links.md – Ahmad Awais Aug 5 '16 at 16:53
1

I have found a solution here that is IMHO much simpler and potentially faster than what was proposed here. I have adjusted a little bit to support https files. But the TD;TR version is ...

PS: You can replace the site URL with a path to a file and it will work the same way.

lynx -dump -listonly -nonumbers "http://www.goggle.com" > links.txt

lynx -dump -listonly -nonumbers "some-file.html" > links.txt

If you just want to see the links instead of placing them in a file, then try this instead ...

lynx -dump -listonly -nonumbers "http://www.google.com"

lynx -dump -listonly -nonumbers "some-file.html"

The result will look similar to the following ...

http://www.google.ca/imghp?hl=en&tab=wi
http://maps.google.ca/maps?hl=en&tab=wl
https://play.google.com/?hl=en&tab=w8
http://www.youtube.com/?gl=CA&tab=w1
http://news.google.ca/nwshp?hl=en&tab=wn
https://mail.google.com/mail/?tab=wm
https://drive.google.com/?tab=wo
https://www.google.ca/intl/en/options/
http://www.google.ca/history/optout?hl=en
...
etc.

For my use case, this worked just fine. But beware of the fact that nowadays, people add links like src="//blah.tld" for CDN URI of libraries. I didn't want to see those in the retrieved links.

No need to try to check for href or other sources for links because "lynx -dump" will by default extract all the clickable links from a given page. So the only think you need to do after that is to parse the result of "lynx -dump" using grep to get a cleaner raw version of the same result.

  • But the question says “extract URLs from an HTML file [that looks] like” (example), NOT “extract URLs from a web page”.   If your answer can be used against a file that’s on the local machine, explain how.    Please do not respond in comments; edit your answer to make it clearer and more complete. – G-Man Says 'Reinstate Monica' Dec 7 '17 at 1:39
  • 1
    You can replace the URL by a file name. – asiby Dec 7 '17 at 1:43
  • @G-Man, why the -1? You need to try the code yourself and see that it works for local files also. I have added that clarification in case it was not obvious. – asiby Dec 7 '17 at 2:42
  • This is really handy .. if you're using with xargs it's worth adding | sort | uniq to trim the duplicate links. – Stuart Axon Sep 24 '18 at 20:23
0
wget -qO- google.com |
tr \" \\n | grep https\*://

...would probably do pretty well. As written, it prints:

http://schema.org/WebPage
http://www.google.com/imghp?hl=en&tab=wi
http://maps.google.com/maps?hl=en&tab=wl
https://play.google.com/?hl=en&tab=w8
http://www.youtube.com/?tab=w1
http://news.google.com/nwshp?hl=en&tab=wn
https://mail.google.com/mail/?tab=wm
https://drive.google.com/?tab=wo
http://www.google.com/intl/en/options/
http://www.google.com/history/optout?hl=en
https://accounts.google.com/ServiceLogin?hl=en&continue=http://www.google.com/
https://www.google.com/culturalinstitute/project/the-holocaust?utm_source=google&amp;utm_medium=hppromo&amp;utm_campaign=auschwitz_q1&amp;utm_content=desktop
https://plus.google.com/116899029375914044550

If it is important that you only match links and from among those top-level domains, you can do:

wget -qO- google.com |
sed '/\n/P;//!s|<a[^>]*\(https*://[^/"]*\)|\n\1\n|;D'

...or something like it - though for some seds you may need to substitute a literal \newline character for each of the last two ns.

As written, the above command prints:

http://www.google.com
http://maps.google.com
https://play.google.com
http://www.youtube.com
http://news.google.com
https://mail.google.com
https://drive.google.com
http://www.google.com
http://www.google.com
http://www.google.com
https://www.google.com
https://plus.google.com

...and for either case (but probably most usefully with the latter) you can tack on a |sort -u filter to the end to get the list sorted and to drop duplicates.

0

Shortest

grep -r http . --color
-1
echo "<a href="http://examplewebsite.com/">"|sed -r 's:<.*"::g'|sed 's:/">$::g'
  • I'm not sure that those “smart“ quotes are what you intended there -- perhaps regular "double" quotes? – Jeff Schaller Dec 10 '17 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.