Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am using curl to grab the HTML of a website. I want to only grab the first 20 lines, and save this in an external file. I need to grab about 10 URLs, and want to save all of them in the same file. Preferably with the URL of each file above the results. What I came up with is:

curl http://example.com/url1 | head -20 && curl http://example.com/url2 | head -20 > exportfile

However, this yields two issues:

  • This only saves the latest page (and if I put > exportfile after every statement, it overvwrites the file
  • This does not save the actual command in the file

Any way to make this work as I intend?

share|improve this question

2 Answers 2

up vote 3 down vote accepted

You could save all your URLs in a file (say urls.txt, one per line), then loop over them in your script:

#! /bin/bash

# optional, clean up before starting
rm exportfile

while read url ; do
   echo "URL: $url"       >> exportfile
   curl "$url" | head -20 >> exportfile
done < urls.txt

One of the important things is using >> to append. > overwrites.

share|improve this answer
    
Great, works as a charm! –  user Mar 30 '12 at 13:59

Try this (bash):

{ curl http://example.com/url1 | head -20 \
  && curl http://example.com/url2 | head -20; } > exportfile
share|improve this answer
    
Thanks, works fine, but it does not put the URL at the top of the output. –  user Mar 30 '12 at 13:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.