0

I am attempting to use the CURL command to download the latest zip file (as of today it is FuelWatchRetail-03-2020.csv.zip) from this site.

The command I currently have is as follows:

curl https://www.fuelwatch.wa.gov.au/fuelwatch/pages/public/historicalFileDownloadRetail.jspx#/FuelWatchRetail-03-2020.csv.zip -o new.zip

The file that is output appears to be the html from the initial webpage. I can not seem to find a link to the file location I am trying to download. I have googled extensively to find a solution, however I am obviously not searching for the correct thing.

1

If you try to retrieve this file using a regular browser with Developer console launched, you'll notice that, when you click on the "link", it will trigger a HTTP POST request to https://www.fuelwatch.wa.gov.au/fuelwatch/pages/public/historicalFileDownloadRetail.jspx with the following parameters:

{
   "j_idt72":"j_idt72",
   "j_idt72:resultsTbl_rppDD":"20",
   "javax.faces.ViewState":"-4860850130551349268:6438609436745021067",
   "j_idt72:resultsTbl:0:j_idt75":"j_idt72:resultsTbl:0:j_idt75"
}

In return, the requested file is given as attachment:

{"headers":
   [
       {
          "name":"Content-Disposition",
          "value":"attachment;filename=\"FuelWatchRetail-03-2020.csv.zip\"; filename*=UTF-8''FuelWatchRetail-03-2020.csv.zip"
       },
       {
          "name":"Content-Type","value":"application/pdf"},
       [remaining data are useless for this explanation]
}

So it's quite impossible to retrieve the file you want without implementing a bit of "logic" around curl. It'll probably be less painfull to implement this using a scripting language with HTTP/HTML abilities such as Perl, Python, etc...

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.