I am try to fetch all the links from
materialdesignicons.com! I do the following:
curl -X GET https://materialdesignicons.com | grep -i "<link href=" | grep -v "<link href="
But it doesn't output anything. Everything works until
Let's break this down:
Grab the web page
curl -X GET https://materialdesignicons.com
Feed the results through a search (without regard to case) for the pattern
grep -i "<link href="
Feed the next stage of results through a search for lines that do not match the pattern
grep -v "<link href="
The result is that you'll only get matches for text matching
<link href= where one or more of the letters are capitalised. For example
<link href= # Will not match <link HREF= # Will match <LinK HrEf= # Will match
If you want to list all the
link href values you could try this, which will match links that are not split across multiple lines:
curl -X GET https://materialdesignicons.com | grep -Po "(?<=link href=([\"'])).*?(?=\g1)"
It's not a particular pleasant regular expression so I'll try to break it down for you:
(?<=link href=([\"'])) # Look for "link href=" followed by either single or double quote .*? # Match and output the shortest possible string until... (?=\g1) # We have found a repeat of the quote we found earlier
An alternative grep solution is
grep -iPo "(<link href=\")[^\"]*"
defines a character class which includes only a single character, a
", escaped (
\")for reasons I hope you appreciate, and then the class is negated
^ meaning "anything which isn't a
This translates as "find a case insensitive occurrence of
<link href=" then match all of the following characters until you find a
" and then stop, but don't include the
Anyway, since @roaima beat me to a regex solution and I am repeatedly having it beaten into me that it is (apparently) not recommended to parse html with
grep, I thought I'd also play with
xmllint and the xpath (
//link/@href) may be easier to grasp if you are unfamiliar with regex.
xmllint --html --xpath "//link/@href" <( curl -X GET https://materialdesignicons.com )
xmllint to expect an
--html document and uses
//link/@href to select the
href attributes within all the
xmllint complains about the html downloaded from your url being not well formed, so we
pipe it through
tidy to correct it, suppressing any verbosity (
-q) and discarding any error messages (
2>/dev/null) before redirecting the well formed output to
xmllint --html --xpath "//link/@href" <( curl -X GET https://materialdesignicons.com | tidy -q 2>/dev/null)
This yields the
href attributes still lovingly(?) wrapped in quotes
href="/favicon.png" href="//fonts.googleapis.com/css?family=Roboto:400,300" href="styles/css/bootstrap.css" href="styles/app.css"
A bit of post-processing needed to give the bare links, but now you have a choice of tools.