1

This website contains pdf and zip files for download.

enter image description here

The problem is that I should download these files one by one by clicking on a button. I tried using wget to download the entire website without successful. What I can do?

1

The curl request is as follows (you can get it from the network section of developer tools of browsers):

curl 'https://www.concours-maths-cpge.fr/' -H 'Cookie: startBAK=xxxxxxxxxxxxxxxx; start=xxxxxxxxxxxx' -H 'Origin: https://www.concours-maths-cpge.fr' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.8' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.86 Safari/537.36' -H 'Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryBj169hPosWDbuaq7' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'Cache-Control: max-age=0' -H 'Referer: https://www.concours-maths-cpge.fr/' -H 'Connection: keep-alive' --data-binary $'------WebKitFormBoundaryBj169hPosWDbuaq7\r\nContent-Disposition: form-data; name="cmd"\r\n\r\n\r\n------WebKitFormBoundaryBj169hPosWDbuaq7\r\nContent-Disposition: form-data; name="mode"\r\n\r\nsimple\r\n------WebKitFormBoundaryBj169hPosWDbuaq7\r\nContent-Disposition: form-data; name="annee"\r\n\r\n2017\r\n------WebKitFormBoundaryBj169hPosWDbuaq7\r\nContent-Disposition: form-data; name="concours"\r\n\r\n0\r\n------WebKitFormBoundaryBj169hPosWDbuaq7\r\nContent-Disposition: form-data; name="filiere"\r\n\r\n0\r\n------WebKitFormBoundaryBj169hPosWDbuaq7\r\nContent-Disposition: form-data; name="matiere"\r\n\r\n0\r\n------WebKitFormBoundaryBj169hPosWDbuaq7\r\nContent-Disposition: form-data; name="epreuve"\r\n\r\n0\r\n------WebKitFormBoundaryBj169hPosWDbuaq7\r\nContent-Disposition: form-data; name="commande"\r\n\r\nrechercher\r\n------WebKitFormBoundaryBj169hPosWDbuaq7--\r\n' --compressed

And the curled data is:

<head><title>Sujets de concours</title>
<link rel='stylesheet' href='javascript/table.css' type='text/css' />
<script language='javascript' src='javascript/table.js'></script>
</head>
<body background='images/julia.gif'>
<div align=left><table style='background:none'><tr>
  <td><img src='images/ups.gif'>
  <td><h1>Union des Professeurs de classes préparatoires Scientifiques<br>Sujets de concours
</tr></table></div>
<div align='center'><form enctype='multipart/form-data' method='post'>
<input type='hidden' name='cmd' value=''/>
<input type='hidden' name='mode' value='simple'/>
<input type='hidden' name='annee'       value='2017' />
<input type='hidden' name='ordre'       value='-annee:+nom:+filiere:+matiere:+epreuve' />
<table border=2 rules='none' style='table-layout:fixed' width='90%'><tr><td><button title='tri croissant' style='width:100%' onclick='mysubmit(this,"commande=rechercher|ord=+annee")'>année</button><td><button title='placer à gauche' style='width:100%' onclick='mysubmit(this,"commande=rechercher|ord=+nom")'>concours</button><td><button title='placer à gauche' style='width:100%' onclick='mysubmit(this,"commande=rechercher|ord=+filiere")'>filière</button><td><button title='placer à gauche' style='width:100%' onclick='mysubmit(this,"commande=rechercher|ord=+matiere")'>matière</button><td><button title='placer à gauche' style='width:100%' onclick='mysubmit(this,"commande=rechercher|ord=+epreuve")'>épreuve</button><td style='width:5em; padding:0px'></tr>
  <tr class='even'><td class='rowspan' rowspan=10><a title='2017'>2017</a><td class='rowspan' rowspan=2><a title='Agrégation externe'>Agr-ext</a><td class='rowspan' rowspan=2><a title='M1'>M1</a><td class='rowspan' rowspan=2><a title='maths'>maths</a><td><a title='algèbre'>algèbre</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17a4a")'>fichiers</button></tr>
  <tr class='odd'><td><a title='analyse'>analyse</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17a4y")'>fichiers</button></tr>
  <tr class='even'><td><a title='Agrégation concours spécial'>Agr-special</a><td><a title='autre'>autre</a><td><a title='maths'>maths</a><td><a title='unique'>unique</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17azu")'>fichiers</button></tr>
  <tr class='odd'><td class='rowspan' rowspan=2><a title='Agrégation interne'>Agr-int</a><td class='rowspan' rowspan=2><a title='autre'>autre</a><td class='rowspan' rowspan=2><a title='maths'>maths</a><td><a title='1'>1</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17bz1")'>fichiers</button></tr>
  <tr class='even'><td><a title='2'>2</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17bz2")'>fichiers</button></tr>
  <tr class='odd'><td class='rowspan' rowspan=2><a title='Banque Agro-Véto'>Agro</a><td class='rowspan' rowspan=2><a title='BCPST'>BCPST</a><td class='rowspan' rowspan=2><a title='maths'>maths</a><td><a title='A'>A</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17ab1")'>fichiers</button></tr>
  <tr class='even'><td><a title='B'>B</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17ab2")'>fichiers</button></tr>
  <tr class='odd'><td class='rowspan' rowspan=3><a title='Concours Commun Polytechniques'>CCP</a><td class='rowspan' rowspan=3><a title='MP'>MP</a><td><a title='info'>info</a><td><a title='option'>option</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=i17pmo")'>fichiers</button></tr>
  <tr class='even'><td class='rowspan' rowspan=2><a title='maths'>maths</a><td><a title='1'>1</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17pm1")'>fichiers</button></tr>
  <tr class='odd'><td><a title='2'>2</a><td><button title='fichiers disponibles' style='width:100%' onclick='mysubmit(this,"commande=rechercher|init=0|numero=m17pm2")'>fichiers</button></tr>
<tr><td colspan=5><table width='100%'><tr>
  <td>sujets 1 à 10 (total = 63)
  <td align='right'><button title='page suivante'   onclick='mysubmit(this,"commande=rechercher|init=10")'>&gt;</button></tr></table>
  <td><button title='modifier la recherche' style='width:100%' onclick='mysubmit(this,"commande=connexion")'>retour</button>
</tr></table>
</form></div></body>

and there are no pdf links to follow with wget. So with ajax, the pdf links are generated with javascripts on click.

The only solution is Selenium Webdriver which basically automates the interactive moves in a browser. You can use chrome/chormium or firefox with selenium (When your code with Selenium runs, an open browser window automates the clicks) but the best way to do is to use a headless browser like PhantomJS with Selenium.

Here are some links to examples and some issues about scraping with Selenium (and PhantomJS):

Scraping with Python Selenium and PhantomJS

Headless Selenium Testing With Python and PhantomJS

WebDriver click() vs JavaScript click()

Using Selenium with Python and PhantomJS to download file to filesystem

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.