1

I try to download the 1198 pages of signatures on this petition page http://www.musikskatt.se/ (at the bottom) but both curl and wget leaves the div with the signatures empty in the resulting file.

How can I fix this?

(Want to download it all to search for a specific name with grep or so)

3
  • It's a javascript triggered next page function, 1200 pages or whatever. You can't request the data with curl or wget. In my test, their server is now rejecting wget/curl requests, something seems to have activated mod security, a little late considering their site is possibly hacked (unless they put in the seo spam to generate income), lol. But javascript is loading the content you see, it's an ajax type request, not a get html request, so curl/wget can't grab that data.
    – Lizardx
    Commented Dec 26, 2019 at 3:50
  • @Lizardx It never accepted wget/curl, I had to add -A "Mozilla/4.0" to make it work.
    – d-b
    Commented Dec 26, 2019 at 8:16
  • Oh, that explains it, that would have been my next test, if I thought the problem could be solved, but it can't be using wget/curl. Sometimes if you root through the js you can find a way to get the data directly from an ajax request, which will be in the form of actual html, but that takes forever to figure out so rarely worth it.
    – Lizardx
    Commented Dec 26, 2019 at 23:15

3 Answers 3

2

You can check in your browser's development tools how that data is received. Namely, through XHR POST to http://www.musikskatt.se/Home/LoadData with parameter { 'pageNumber': '1'}. And that's the way you should get them.

See also AJAX.

3
  • Hmm, should something like musikskatt.se/Home/LoadData?pageNumber=2 work? Because it doesn't...
    – d-b
    Commented Dec 25, 2019 at 21:10
  • @d-b This is GET not POST. Take a short lesson on the fundamentals of HTTP.
    – user147505
    Commented Dec 25, 2019 at 21:12
  • There are very few cases when you can't switch between GET and POST. I use the Wed Developer extension for this purpose almost daily.
    – d-b
    Commented Dec 25, 2019 at 22:42
2

If you disable in your browser, you will see that if you refresh the page, the names are not rendered.

This is not a task for wget or curl but for a programing language with libs that are javascript capable.

Or you can eventually try to figute out what request is called by , and do it with curl by digging around in the browser dev tools, network tab

-1

For downloading files from a directory listing, use -r (recursive)

wget -r www.example.com 

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .