Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have some "corporative" systems I have to use at work. Some of them are terrible, with frames, pop-ups and such.

I was thinking about automating some stuff using curl or something like that. I need login, click on hyperlinks, set some data, and them do some post.

Is it possible to use curl? Or am I'm going to be better using something like Selenium?

share|improve this question
1  
I would use Python's zope.testbrowser for that. –  phunehehe Apr 14 '11 at 3:19
2  
Do you need to be able to use Javascript? –  intuited Apr 14 '11 at 4:28
1  
2  
Using a session is just sending the session id with each request. Either in a cookie or in GET parameter. wget and curl can use Netscape cookie file, so if your browser also use such one, may be enough to just specify it to the downloader. So, what kind of browser are you using? –  manatwork Dec 9 '11 at 7:35
2  
No luck for a simple solution with those. Firefox and Chrome uses SQLite, Opera has a binary data file format. Looks like you will have to search for the session id and pass it manually to the downloader. –  manatwork Dec 9 '11 at 10:54
show 2 more comments

3 Answers

up vote 9 down vote accepted

Yes, it's fully possible with curl. Most importantly will be to save and reload cookies between uses of curl with --cookie-jar. You can also post form data as needed. I usually use a Firefox add-on called Live HTTP Headers to capture what happens when I navigate a website. It will record any headers, but also any form posts which is very helpful when trying to figure out what to do with curl. I've written bash scripts automating various curl invocations and use pipes and temporary files (courtesy of mktemp) to do some limited processing on the webpage, but if have to do a lot of webpage processing I usually switch to Perl with LibWWW.

share|improve this answer
    
In addition to Firefox Live HTTP Headers, both Safari and Chrome can display the headers broken down by object in the Web Inspector. –  bahamat Jul 20 '12 at 0:17
add comment

For simple cases of downloading the contents of a page, use curl or wget. Both are command line tools designed to download files over HTTP and have many options. In your case, you'll likely need to make these tools look more like a browser; lutzky's answer and penguin359's answer mention some curl and wget options that are useful in that respect.

Sometimes, when you need to log in, it's a lot easier to first log in manually in a web browser, then export the web browser's cookies (extensions like allcookies or Export Cookies for Firefox can help).

If you need to parse the contents of some pages or post forms, you may need fancier tools than curl and wget. Some good tools there are Perl with LWP (libwww) and HTML::TreeBuilder (HTML-Tree) or Python with standard libraries (especially httplib and htmllib).

For more complex interactions with a website, the reference is Perl's WWW::Mechanize. This Perl library defines high-level functions to interact with a website like a web browser does, including POSTing, forms, cookies, but not Javascript. If Perl isn't your cup of tea, this library has imitations with similar capabilities in other languages, such as Python mechanize and Ruby Mechanize.

Finally, when you need Javascript, the usual approach is to use a web browser which is driven by a browser automation framework. Selenium and Watir are popular choices; see also Are there any good tools besides SeleniumRC that can fetch webpages including content post-painted by JavaScript?

share|improve this answer
add comment

My favorite is wget, so I'll give an example with that. What you want to do is replicate your browser session as closely as possible, so use the relevant commandline arguments. The ones which are necessary depend on how thoroughly the site checks your browser. --referer (sic) is usually sufficient, but you might also need --user-agent and --load-cookies.

This technique is basically browser spoofing - you're trying to get wget to present itself to the site as the browser. The biggest problem you'll be facing is CAPTCHA, which is not present in the example site you've given. Also, make sure you abide by the site's timing policies (45 seconds in this case), or you'll probably get blocked.

It's probably not possible to download the same file across different sessions - the site grants downloading privileges on the specific file to a specific session.

share|improve this answer
    
Good answer with comprehensive considerations. Thanks –  xiaohan2012 Dec 10 '11 at 1:23
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.