I am try to fetch all the links from materialdesignicons.com! I do the following:

curl -X GET https://materialdesignicons.com | grep -i "<link href=" | grep -v "<link href="

But it doesn't output anything. Everything works until grep -v!


Let's break this down:

  1. Grab the web page

    curl -X GET https://materialdesignicons.com
  2. Feed the results through a search (without regard to case) for the pattern <link href=

    grep -i "<link href="
  3. Feed the next stage of results through a search for lines that do not match the pattern <link href=

    grep -v "<link href="

The result is that you'll only get matches for text matching <link href= where one or more of the letters are capitalised. For example

<link href=        # Will not match
<link HREF=        # Will match
<LinK HrEf=        # Will match

If you want to list all the link href values you could try this, which will match links that are not split across multiple lines:

curl -X GET https://materialdesignicons.com | grep -Po "(?<=link href=([\"'])).*?(?=\g1)"

It's not a particular pleasant regular expression so I'll try to break it down for you:

(?<=link href=([\"']))    # Look for "link href=" followed by either single or double quote
.*?                       # Match and output the shortest possible string until...
(?=\g1)                   # We have found a repeat of the quote we found earlier

An alternative grep solution is

grep -iPo "(<link href=\")[^\"]*"

The []defines a character class which includes only a single character, a ", escaped (\")for reasons I hope you appreciate, and then the class is negated ^ meaning "anything which isn't a ""

This translates as "find a case insensitive occurrence of <link href=" then match all of the following characters until you find a " and then stop, but don't include the "".

Anyway, since @roaima beat me to a regex solution and I am repeatedly having it beaten into me that it is (apparently) not recommended to parse html with grep, I thought I'd also play with xmllint and the xpath (//link/@href) may be easier to grasp if you are unfamiliar with regex.

xmllint --html --xpath "//link/@href" <( curl -X GET https://materialdesignicons.com )

This tells xmllint to expect an --html document and uses --xpath expression //link/@href to select the href attributes within all the link tags.

However xmllint complains about the html downloaded from your url being not well formed, so we pipe it through tidy to correct it, suppressing any verbosity (-q) and discarding any error messages (2>/dev/null) before redirecting the well formed output to xmllint

xmllint --html --xpath "//link/@href" <( curl -X GET https://materialdesignicons.com | tidy -q 2>/dev/null)

This yields the href attributes still lovingly(?) wrapped in quotes

href="/favicon.png" href="//fonts.googleapis.com/css?family=Roboto:400,300" href="styles/css/bootstrap.css" href="styles/app.css"

A bit of post-processing needed to give the bare links, but now you have a choice of tools.

  • +1 for using xmllint; this is by far the cleaner solution – roaima Jan 22 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.