I have a text file with lots of url's in it. I'm using
curl -K "$urls" > $output
to spit the output to my output file. Now for the output of each separate url there is a term, let's say "mortgage", beneath which I do not want any more of the info. Now I know that I can use
to remove all info below the term "mortgage" but if I use it within my script like so
curl -K "$urls" | sed '/mortgage/q' > $output
it removes everything from the entire output below the first instance of "mortgage" from the output of first url in the $urls, but this wipes all of the info from the other url's (including the stuff before their own instance of the word "mortgage") due to the fact that it is working on the entire output, and not for each url.
How can I specify the
sed '/mortgage/q' to act separately on the output of each url in the url file so that it does not affect the output globally?
My url file is pretty simple, in the format (this is just an example):
URL = http://www.bbc.co.uk/sport/rugby-union/34914911 URL = http://stackoverflow.com/questions/9084453/simple-script-to-check-if-a-webpage-has-been-updated
and so on.....
I've conceived a hypothetical way of achieving this but not sure of the code - is there any way I can adapt the
curl -K "$urls" | sed '/mortgage/q' > $output command so that the command loops back after each subsequent url in the
$url file i.e. So that the curl command initially just retrieves the first url in the file, performs the
sed command on that url material, appends to
$output, then loops back to the second url in the file, perform the sed command, append to
$output and so on.... This would mean the required material from each url was included in the output file, but the stuff below 'mortgage' in each url was not. I just don't know how to achieve this with code. Any ideas?