Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

One major shortcoming of curl is that more and more wepages are having their main piece of content painted by a JavaScript AJAX response that occurs after the initial HTTP response. curl never picks up on this post-painted content.

So to fetch these types of webpages from the command line, I've been reduced to writing scripts in Ruby that drive the SeleniumRC to fire up a Firefox instance and then return the source HTML after these AJAX calls have completed.

It would be much better to have a leaner command line solution for this type of problem. Does anyone know of any?

share|improve this question
No one's suggested anything else on Does anybody here have experience in automating some tasks in web applications using curl?, but that question wasn't specifically asking about scraping Javascript. – Gilles Apr 28 '11 at 21:54
up vote 2 down vote accepted

I just recently started using the WebDriver from Selenium 2 in Java. There is a driver called HtmlUnitDriver that fully supports JavaScript but does not fire up an actual browser.

It is not a light solution but it does get the job done.

I've designed the code to run from the command line and save the web data to files.

share|improve this answer

Have you considered Watir?


When you've added the package, you can run it as a standalone file or from irb, line-by-line after include 'watir-webdriver'. I've found it to be more responsive than selenium-webdriver, but without the test recording GUI to help work out complex test conditions.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.