0

I am having roughly the same problem as in this question, where the problem is, there is a certain URL that, when called in the browser, will download a file, but when called from the command line with wget, will result in a file called API.php?bla (where bla is a string). In that earlier question, some answers are given that suggest using the --referer option of wget. This doesn't work for me and besides seems overly complicated.

I am wondering if there is an easier solution: Is it somehow possible to call a URL from the terminal AS IF it were called from the browser, so that it does anything the browser normally does to ensure that the correct file is downloaded? We could use xdg-open, but this would also unnecessarily open a browser window, which is ugly.

1 Answer 1

0

There are many reasons why a URL might return different content when accessed by wget and when accessed with an interactive browser. The server may serve different content depending on the user agent, on the referrer, on cookies, on the availability of JavaScript, on the many properties that can be queried with JavaScript…

The only general solution to emulating the behavior of a “heavy” browser (with JS support, etc.) is to run the browser. More lightweight solutions may or may not work depending on what browser features the server relies on. You can run the browser headlessly. See How to use wget to download from hosting sites?, Automating web requests using curl?, How can I run Firefox on Linux headlessly (i.e. without requiring libgtk-x11-2.0.so.0)?, …

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .