0

I have an index.html file containing href link to PDF files.

When I do : grep -i 'href=' index.html, I get for example :

<p>Télécharger : <a href="https://ecole-euclid.cnrs.fr/wp-content/uploads/EDE2019_Henrot-Versillé-C1_L1.pdf"><span style="color: #0000ff;">Cours n°1</span></a> (S. Henrot-Versillé), <span style="color: #0000ff;"><a href="https://ecole-euclid.cnrs.fr/wp-content/uploads/EDE2019_Henrot-Versillé_C1_L2.pdf">Cours n°2</a></span> (S. Henrot-Versillé), <a href="https://ecole-euclid.cnrs.fr/wp-content/uploads/EDE2018_Henrot-Versillé_C3.pdf"><span style="color: #0000ff;">Cours n°3</span></a> (S. Henrot-Versillé)</p>
<p>Télécharger le cours sur <a href="https://ecole-euclid.cnrs.fr/wp-content/uploads/EDE2018_Martinelli_C2_L1_Bayesian.pdf">la méthode bayésienne</a> (M. Martinelli) et <a href="https://ecole-euclid.cnrs.fr/wp-content/uploads/EDE2018_Martinelli_C2_TD_Bayesian.pdf">son TD</a> (M. Martinelli).</p></div>
<p><a href="https://github.com/mhuertascompany/EDE19" title="GitHub Deep Learning 2019 EDE">https://github.com/mhuertascompany/EDE19</a></p>
<p><a href="https://colab.research.google.com/drive" title="TDs Deep Learning 2019">https://colab.research.google.com/drive</a></p></div>
        <a href="https://www.facebook.com/euclid.france" class="icon">
        <a href="https://twitter.com/Euclid_FR" class="icon">
        <a href="#" class="icon">
        <a href="https://ecole-euclid.cnrs.fr/feed/" class="icon">

Now, I would like to pipe this output of grep with gsed (on MacOS Catalina) in order to extract all the full href of PDF files, even when there are multiple PDF links on a same line.

I tried firstly :

grep -i 'href=' index.html | gsed 's/href="\(.*pdf\)"/\1/g'

But this doesn't work and as you can see, I would only print the first PDF link, not all the PDF link (on a same link), so moreover, how print all pattern matching ?

The goal is to donwload after this all the PDF files present in index.html file

Any help would be great.

4
  • 1
    Try grep -Poi 'href="\K[^"]*\.pdf'
    – user414777
    Sep 4 '20 at 20:26
  • 1
    Or just grep -oi '[^"]*\.pdf'
    – user414777
    Sep 4 '20 at 20:29
  • 1
    @user414777 those would extract parts of text from strings like href="ftp://foo.pdfdir/bar.gif" etc.
    – Ed Morton
    Sep 6 '20 at 13:12
  • 1
    It's easy to workaround for such cases. grep -Poi 'href="\K[^"]*\.pdf(?=")'. or grep -oi '[^"]*\.pdf\>'. Using regular expressions or tools like awk, sed, grep to parse (and interpret) html is on a best effort basis, anyways. For a change, just consider the case of relative urls, non quoted attribute values, entities, etc.
    – user414777
    Sep 7 '20 at 0:07
1

Since you have GNU sed you can install GNU awk. With GNU awk for multi-char RS and RT:

$ awk -v RS='href="http[^"]+.pdf"' -F'"' 'RT{$0=RT; print $2}' file
https://ecole-euclid.cnrs.fr/wp-content/uploads/EDE2019_Henrot-Versillé-C1_L1.pdf
https://ecole-euclid.cnrs.fr/wp-content/uploads/EDE2019_Henrot-Versillé_C1_L2.pdf
https://ecole-euclid.cnrs.fr/wp-content/uploads/EDE2018_Henrot-Versillé_C3.pdf
https://ecole-euclid.cnrs.fr/wp-content/uploads/EDE2018_Martinelli_C2_L1_Bayesian.pdf
https://ecole-euclid.cnrs.fr/wp-content/uploads/EDE2018_Martinelli_C2_TD_Bayesian.pdf

otherwise using any awk in any shell on every UNIX box:

$ awk '{
    while ( match($0,/href="http[^"]+.pdf"/) ) {
        split(substr($0,RSTART,RLENGTH),f,/"/)
        print f[2]
        $0 = substr($0,RSTART+RLENGTH)
    }
}' file
https://ecole-euclid.cnrs.fr/wp-content/uploads/EDE2019_Henrot-Versillé-C1_L1.pdf
https://ecole-euclid.cnrs.fr/wp-content/uploads/EDE2019_Henrot-Versillé_C1_L2.pdf
https://ecole-euclid.cnrs.fr/wp-content/uploads/EDE2018_Henrot-Versillé_C3.pdf
https://ecole-euclid.cnrs.fr/wp-content/uploads/EDE2018_Martinelli_C2_L1_Bayesian.pdf
https://ecole-euclid.cnrs.fr/wp-content/uploads/EDE2018_Martinelli_C2_TD_Bayesian.pdf

Just pipe that output to xargs -n 1 curl -O, to download the PDFs (assuming no spaces in the URL).

1
  • 1
    Thanks for your quick answer, this works perfectly !
    – youpilat13
    Sep 4 '20 at 21:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.