9

I am downloading the HTML source page of a website using cURL command. The problem is, I do not want most of the contents in the page. I just need the first 100 lines of the source page. Is there a way to stop the download of the page after first few lines?

Currently, I have the below command working but it does not seem to be time effective.

curl -r[0-1] "http://www.freebase.com/m/045c7b" > foo.txt

I tried changing the value from 1 to .5 and .05 and still the entire web page is getting downloaded.

I am looking at time less than a second to execute the above command.

EDIT

From the man page of cURL, I see that "You should also be aware that many HTTP/1.1 servers do not have this feature enabled, so that when you attempt to get a range, you'll instead get the whole document." So if the server is not supporting the range querying, is there some other command in nix environment which will help me accomplishing what am trying to do?

6

You can use head and the -s switch to curl to download a portion of a page. Simply tell head how many lines of output you want.

Example

$ curl -r[0-1] "http://www.freebase.com/m/045c7b" -s 2>&1 | head -10
Warning: Invalid character is found in given range. A specified range MUST 
Warning: have only digits in 'start'-'stop'. The server's response to this 
Warning: request is uncertain.


  <!DOCTYPE html PUBLIC "null" "null">
  <html class="no-js" dir="ltr" lang="en">
    <head>
      <meta charset="utf-8">
      <title>

Details

  • The -10 to head will return 10 lines and then stop. If you wanted more then change it to suit your needs.
  • The -s switch to curl tells it to be silent so it won't display the progress bar while downloading.
  • 1
    It still takes considerable time to download the contents. I was expecting a time range in milli seconds for this to happen. – Ramesh Mar 3 '14 at 21:51
  • @Ramesh - yes I noticed it was rather slow too. Seems to take some time for the server to reply. – slm Mar 3 '14 at 21:54
  • @Ramesh - rather then go at the site using curl you might want to take a look at the API: developers.google.com/freebase – slm Mar 3 '14 at 21:57
  • Hmmm, interesting. My friend was suggesting that in Java if he used freebase API, it took him 300 MS to download the data. I told him it can be reduced considerably using shell script. It seems a problem with the server rather than the shell commands. – Ramesh Mar 3 '14 at 21:59
  • @Ramesh - Yes the slowness in this case is in the database query and the preparation of that data into a response. If the database exposes the the data in other formats just as JSON then you can expedite the response by utilizing that instead of taking the output as HTTP. There is overhead in preparing the response as HTTP that is a waste if the end usecase isn't going to be a human reading the results. – slm Mar 3 '14 at 22:07
1

I haven't really tested this particular application yet, but something tells me you could pair dd and nc here:

$ nc www.website.com 80 <<GOT | dd bs=$BYTE_COUNT iflag=fullblock \
     count=1 of=$OUTFILE
GET / HTTP/1.1
Host: ispconfig.org
Referrer: mypage.com
User-Agent: my-browser
$(printf %b '\r\r')
GOT

Netcat (nc) might require further configuration to get the request headers right but if it's a public site you should be able to finagle a useful output if you care enough to try. Certainly dd will take only as much input as you specify and quit, which will SIGPIPE netcat so that it will immediately follow suit. The only real trick is ironing out the initial handshake - once you start the stream you can drop it whenever you like.

EDIT

Reading slm's comments has prompted me to second that motion; if you can serialize a JSON POST in the correct format that's definitely the way to go for more timely response. Parsing HTML is for the birds, anyway.

One useful trick to that end is to capture your network stream while communicating with the server in your browser, then when your browser sends the POST that gets you what you want send it again as a GET instead and have a look at the results.

1

The head command will usually stop the download before it ends (although for short files it may fill the pipe buffer before the pipe is closed). This is because when a pipe is closed, curl has nowhere to write (file descriptor is closed, write fails).

However, in my experience, the longest thing when downloading is waiting for DNS requests (painful when you are downloading hundreds of files sequentially). This can be helped with a local DNS cache like dnsmasq or, if you are using the same domain name many times with different directory structure, just resolve it to an IP once and do a replace on URL.

To prove my point... try time netstat versus time netstat -n (without cache the difference is dramatic, with cache, it's only bad the first time, then it remembers).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.