1

I want to download the full HTML source code of a X web page, however curling the X link will return partial HTML source code as that web page requires scrolling to see further content. It seems that curl doesn't go past "scroll down".

So far, I can only do this manually:
1) Go to the desired website
2) Execute the following command in browser's console to auto-scroll (load every object):

var scroll = setInterval(function(){ window.scrollBy(0,1000); }, 2000);

3) Copy the full HTML source code from inspect element

So the question is, how can I run curl command so it scrapes full web page content (scrolls until it loads all objects) before outputing at terminal to achieve same result as the abovementioned steps? If not with curl, maybe wget?

2

curl isn't a full-fledged browser and to the best of my knowledge does not support executing JavaScript. It uses HTTP/FTP to fetch files; that is all. If you want to do testing of functionality which depends upon scripting or other tooling which a bare HTTP request is incapable of touching upon, you will need to look into a more in-depth test suite such as Selenium.

  • Thanks for the input, but I don't actually want to run javascript with curl, I just mentioned it to be clear what my goal is - to automate scripts rather than do stuff manually. I'm just wondering if there is a workaround to do the same thing via linux terminal. – bashbin Apr 30 '18 at 18:23
  • Your step 2 is a JavaScript thing. More broadly, "infinite scrolling" is entirely implemented in scripting which is not something which can be performed by bare HTTP request. – DopeGhoti Apr 30 '18 at 18:44
  • 1
    I actually found a workaround. I am using Firebug extension to see the request behaviour on the web page after I scroll down (to load more content). I'm working to find out the logic behind the HTTP requests so I can curl the releavant links instead. – bashbin Apr 30 '18 at 20:10
1

The only way to do this is to emulate the webbrowser, because you need to run the Javascript. You should be able to do this with Selenium, but it definitely requires some programming, probably in Python.

Selenium link: https://www.seleniumhq.org/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.