curl get all links of a web-page

Question

I used to utilize following command to get all links of a web-page and then grep what I want:

curl $URL 2>&1 | grep -o -E 'href="([^"#]+)"' | cut -d'"' -f2 | egrep $CMP-[0-9].[0-9].[0-9]$ | cut -d'-' -f3

It was doing great till yesterday. I tried to run curl itself and I saw it returns:

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

Was there any possible updates which causes the command not working or what?

EDIT 1:

I changed my approach to wget regarding this answer:

wget -q $URL -O - | grep -o -E 'href="([^"#]+)"' | cut -d'"' -f2 | egrep $CMP-[0-9].[0-9].[0-9]$ | cut -d'-' -f3

But still doesn't know why curl approach suddenly stopped working.

Answer 1

Warning: Using regex for parsing HTML in most cases (if not all) is bad, so proceed at your own discretion.

---

This should do it:

curl -f -L URL | grep -Eo "https?://\S+?\""

or

curl -f -L URL | grep -Eo '"(http|https)://[a-zA-Z0-9#~.*,/!?=+&_%:-]*"'

Note:

• This does not take into account links that aren't "full" or basically are what I call "half a link", where only a part of the full link is shown. I don't recall where I saw this, but it should appear on certain sites under certain/particular HTML tags. EDIT: Gilles Quenot kindly provided a solution for what I wrongly described as "half-link" (the correct term being relative link):

curl -Ls URL |  grep -oP 'href="\K[^"]+'

• This also doesn't "clean" whatever won't be part of the link (eg: a "&" character, etc). If you want to remove that, make/use sed or something else like so:

curl -f -L URL | grep -Eo "https?://\S+?\"" | sed 's/&.*//'

• Lastly, this does not take into account every possible way a link is displayed. Thus certain knowledge of the webpage structure or HTML is required. Given you can't/don't show an example of said structure or the webpage itself, it is difficult to make an answer that works on it, unless more HTML knowledge was involved.

• P.S.: This may or may not be obvious, but this also doesn't take into account links/URLs that are generated dynamically (eg: PHP, JS, etc) since curl mostly works on static links.

• P.S.(2): You should, if you want to make use of a better way to parse HTML, instead of using my solution above (which doesn't handle every corner case, given the lack of an HTML example/sample) use the better answer from Gilles Quenot that is more fit for general (eg: complete) and more optimized support of HTML syntaxes.

I am in no way recommending regex for parsing HTML, unless you know what you're doing or have very limited needs (eg: only want links), like in this case.

Answer 2

Parsing HTML with regex is a regular discussion: this is a bad idea. Instead, use a proper parser:

mech-dump

mech-dump --links --absolute --agent-alias='Linux Mozilla' <URL>

This comes with the package www-mechanize-perl (Debian based distro).

_{^{(Written by Andy Lester, the author of ack and more...)}}

mech-dump doc

xidel or saxon-lint

Or a xpath & network aware tool like xidel or saxon-lint:

xidel -se '//a/@href' <URL>
saxon-lint --html --xpath 'string-join(//a/@href, "^M")' <URL>

^M is Control+v Enter

xmlstarlet:

curl -Ls <URL> |
    xmlstarlet format -H - 2>/dev/null |  # convert broken HTML to HTML 
    xmlstarlet sel -t -v '//a/@href' -    # parse the stream with XPath expression

javascript generated web page

You even can use XPath in a puppeteer javascript script

const puppeteer = require('puppeteer');

var base_url = 'https://stackoverflow.com';

(async () => {
    const browser = await puppeteer.launch({
        headless: true,
    });
    
    // viewportSize
    await page.setViewport({'width': 1440, 'height': 900});

    // UA
    await page.setUserAgent('Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0')

    // open main URL
    await page.goto(base_url, { waitUntil: 'networkidle2' }); 

    const xpath_expression = '//a[@href]';
    await page.waitForXPath(xpath_expression);
    const links = await page.$x(xpath_expression);
    const link_urls = await page.evaluate((...links) => {
        return links.map(e => e.href);
    }, ...links);

    await browser.close();

    link_urls.forEach((elt) => console.log(elt));

})();

Usage:

nodejs retrieve_all_links.js

Answer 3

You can use argument -s for curl, it is for the quiet mode. It will not show progress meter or error message.

Answer 4

I realize it's not what was asked by OP, but lynx, the text browser is an easier option, eg:

pages=($(lynx -dump -hiddenlinks=listonly "$1" | awk "/EpisodeDownload/{print $2}"))

above sample for scraping podcasts URLs from a particular site.

Answer 5

The problem is because curl sends its output to STDERR, while | passes on STDOUT. See examples here.

Two possible solutions are:

1. Pipe STDERR to STDOUT and then pipe that to grep. curl -v http://vimcasts.org/episodes/archive/ 2>&1 | grep archive
2. Use the --stderr flag and give it a hyphen as argument. This will tell curl to use STDOUT instead. curl -v --stderr - http://vimcasts.org/episodes/archive/ | grep archive