13

I have some "corporative" systems I have to use at work. Some of them are terrible, with frames, pop-ups and such.

I was thinking about automating some stuff using curl or something like that. I need login, click on hyperlinks, set some data, and them do some post.

Is it possible to use curl? Or am I'm going to be better using something like Selenium?

13

Yes, it's fully possible with curl. Most importantly will be to save and reload cookies between uses of curl with --cookie-jar. You can also post form data as needed. I usually use a Firefox add-on called Live HTTP Headers to capture what happens when I navigate a website. It will record any headers, but also any form posts which is very helpful when trying to figure out what to do with curl. I've written bash scripts automating various curl invocations and use pipes and temporary files (courtesy of mktemp) to do some limited processing on the webpage, but if have to do a lot of webpage processing I usually switch to Perl with LibWWW.

  • In addition to Firefox Live HTTP Headers, both Safari and Chrome can display the headers broken down by object in the Web Inspector. – bahamat Jul 20 '12 at 0:17
15

For simple cases of downloading the contents of a page, use curl or wget. Both are command line tools designed to download files over HTTP and have many options. In your case, you'll likely need to make these tools look more like a browser; lutzky's answer and penguin359's answer mention some curl and wget options that are useful in that respect.

Sometimes, when you need to log in, it's a lot easier to first log in manually in a web browser, then export the web browser's cookies (extensions like allcookies or Export Cookies for Firefox can help).

If you need to parse the contents of some pages or post forms, you may need fancier tools than curl and wget. Some good tools there are Perl with LWP (libwww) and HTML::TreeBuilder (HTML-Tree) or Python with standard libraries (especially httplib and htmllib).

For more complex interactions with a website, the reference is Perl's WWW::Mechanize. This Perl library defines high-level functions to interact with a website like a web browser does, including POSTing, forms, cookies, but not Javascript. If Perl isn't your cup of tea, this library has imitations with similar capabilities in other languages, such as Python mechanize and Ruby Mechanize.

Finally, when you need Javascript, the usual approach is to use a web browser which is driven by a browser automation framework. Selenium and Watir are popular choices; see also Are there any good tools besides SeleniumRC that can fetch webpages including content post-painted by JavaScript?

6

My favorite is wget, so I'll give an example with that. What you want to do is replicate your browser session as closely as possible, so use the relevant commandline arguments. The ones which are necessary depend on how thoroughly the site checks your browser. --referer (sic) is usually sufficient, but you might also need --user-agent and --load-cookies.

This technique is basically browser spoofing - you're trying to get wget to present itself to the site as the browser. The biggest problem you'll be facing is CAPTCHA, which is not present in the example site you've given. Also, make sure you abide by the site's timing policies (45 seconds in this case), or you'll probably get blocked.

It's probably not possible to download the same file across different sessions - the site grants downloading privileges on the specific file to a specific session.

  • Good answer with comprehensive considerations. Thanks – xiaohan2012 Dec 10 '11 at 1:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.