2

I have a text file with lots of url's in it. I'm using

curl -K "$urls" > $output

to spit the output to my output file. Now for the output of each separate url there is a term, let's say "mortgage", beneath which I do not want any more of the info. Now I know that I can use

sed '/mortgage/q'

to remove all info below the term "mortgage" but if I use it within my script like so

curl -K "$urls" | sed '/mortgage/q' > $output

it removes everything from the entire output below the first instance of "mortgage" from the output of first url in the $urls, but this wipes all of the info from the other url's (including the stuff before their own instance of the word "mortgage") due to the fact that it is working on the entire output, and not for each url.

How can I specify the sed '/mortgage/q' to act separately on the output of each url in the url file so that it does not affect the output globally?

My url file is pretty simple, in the format (this is just an example):

URL = http://www.bbc.co.uk/sport/rugby-union/34914911

URL = http://stackoverflow.com/questions/9084453/simple-script-to-check-if-a-webpage-has-been-updated

and so on.....

I've conceived a hypothetical way of achieving this but not sure of the code - is there any way I can adapt the curl -K "$urls" | sed '/mortgage/q' > $output command so that the command loops back after each subsequent url in the $url file i.e. So that the curl command initially just retrieves the first url in the file, performs the sed command on that url material, appends to $output, then loops back to the second url in the file, perform the sed command, append to $output and so on.... This would mean the required material from each url was included in the output file, but the stuff below 'mortgage' in each url was not. I just don't know how to achieve this with code. Any ideas?

  • I can think of an approach but it depends whether your $urls file just has a list of urls or whether it has other curl parameters. Can you show a snip/example of it? – gogoud Nov 25 '15 at 11:15
  • hi gogoud, i've edited the original post with this info, cheers – neilH Nov 25 '15 at 11:33
  • 1
    where's mortgage then? – mikeserv Nov 25 '15 at 11:46
  • mortgage is just an example, as are the url's I have used in the query – neilH Nov 25 '15 at 15:12
  • how can anybody know what you're talking about? your examples do not include anything like the case you say you want to avoid, an example of which you gave as mortgage. we have no idea why you might consider sed /mortgage/q to be adequate in some cases but inadequate in others because we have no context for your problem. you have provided us with input you want (i guess) not the problematic input you have. how can any answer solve a problem to which it is not privy? – mikeserv Nov 25 '15 at 15:51
3

This should do it in two lines:

sed -n 's/\s*URL\s*=\s*\(.*\)/\1/p' /tmp/curl.conf|xargs -I {} curl -O "{}"
sed -n 's/\s*URL\s*=\s*\(.*\)/\1/p' /tmp/curl.conf|xargs -I {} basename "{}"|xargs -I {} sed '/mortgage/q' "{}"

The first sed command on each line extracts the URLs from your urls file (/tmp/curl.conf in the example). In the first line we use curl's -O option to save the output from each page into a file that has the page's name. In the second line we re-examine each of those files and show only the text you are interested in. Of course if the word 'mortgage' doesn't occur in a file then the whole file will be output.

This will leave you with a temporary file for each url in the current directory.

EDIT:

here's a short script that avoids any left-over files, it outputs the result to standard output, you can redirect it from there as you wish:

#!/bin/bash
TMPF=$(mktemp)
# sed command extracts URLs line by line
sed -n 's/\s*URL\s*=\s*\(.*\)/\1/p' /tmp/curl.conf >$TMPF
while read URL; do
    # retrieve each web page and delete any text after 'mortgage' (substitute whatever test you like)
    curl "$URL" 2>/dev/null | sed '/mortgage/q'
done <"$TMPF"
rm "$TMPF"
  • Thanks for the suggestion gogoud. The only concern I have with this is the creation of a separate file for each url. Ideally I need a single output file as I will be performing further global cleaning of the resulting data and having it in separate files may get messy. I suppose with your suggestion I could then introduce a command to join the data from the separate outputs in a single file, but I will wait and see if anyone provides a simpler solution before I accept this answer. Thanks again. – neilH Nov 25 '15 at 15:27
  • I added a short script which outputs a single stream of data and avoids any leftover files – gogoud Nov 25 '15 at 15:39
  • gogoud- ok mate, gonna try this now, will get back to you, thanks. – neilH Nov 25 '15 at 16:14
  • Thanks gogoud, that is working nicely, thanks for your input, i've accepted your answer :) – neilH Nov 25 '15 at 16:44
0

This general trick works even if your curl config file contains miscellaneous options such as user-agent, referer, etc.

First step, assume your config file named curl_config, then use awk '/^[Uu][Rr][Ll]/{print;print "output = dummy/"++k;next}1' curl_config > curl_config2 to create a new curl config file which incremently append different output file names under each url/URL:

Example:

[xiaobai@xiaobai curl]$ cat curl_config
URL = "www.google.com"
user-agent = "holeagent/5.0"

url = "m12345.google.com"
user-agent = "holeagent/5.0"

URL = "googlevideo.com"
user-agent = "holeagent/5.0"
[xiaobai@xiaobai curl]$ awk '/^[Uu][Rr][Ll]/{print;print "output = dummy/"++k;next}1' curl_config  > curl_config2 
[xiaobai@xiaobai curl]$ cat curl_config2
URL = "www.google.com"
output = dummy/1
user-agent = "holeagent/5.0"

url = "m12345.google.com"
output = dummy/2
user-agent = "holeagent/5.0"

URL = "googlevideo.com"
output = dummy/3
user-agent = "holeagent/5.0"
[xiaobai@xiaobai curl]$ 

Then mkdir dummy to create a directory to hold this temporary files. Create inotifywait session(Replace the sed '/google/q' with your sed '/mortgage/q'):

[xiaobai@xiaobai curl]$ rm -r dummy; mkdir dummy;
[xiaobai@xiaobai curl]$ rm final 
[xiaobai@xiaobai curl]$ inotifywait -m dummy -e close_write | while read path action file; do echo "[$file]">> final ; sed '/google/q' "$path$file" >> final; echo "$path$file"; rm "$path$file"; done;
Setting up watches.
Watches established.

Open another bash/terminal session, rm final file if exist, then run your curl with your curl_config2 file created in first step above:

[xiaobai@xiaobai curl]$ curl -vLK curl_config2
...processing

Now take a look to inotifywait session, it will print the latest close written file, sed it and remove it immediately once done:

[xiaobai@xiaobai curl]$ inotifywait -m dummy -e close_write | while read path action file; do echo "[$file]">> final ; sed '/google/q' "$path$file" >> final; echo "$path$file"; rm "$path$file"; done;
Setting up watches.
Watches established.
dummy/1
dummy/3

Finally you can observe your output named final, The [1 and 3] separator is generated from echo "[$file]">> final above:

enter image description here

The reason of remove file immediately is because i assume your output file is large plus many url have to proceed, so it could save disk space to remove it immediately.

  • Hi, thanks for the response, I appreciate the effort you've put in there, but I feel a simpler solution may be possible. I have added some info to the bottom of my original post (see NEW EDIT section) with a hypothetical way I may achieve my goal, but I am unsure of the correct code still. – neilH Nov 25 '15 at 15:41
  • Some of the urls require different options to make it works (e.g. you must provide user-agent when request google), and that's why curl config file exist(i'm not sure how you get it though). And i prefer to provide general solution(with curl config file) which might help someone else, rather than specific to read line by line with strip first 6 characters "URL = ", such as while read -r line; do [ -z "$line" ] && continue; curl -vL "${line:6}" | sed '/mortgage/q' >> "results" ; done < curl_config . – 林果皞 Nov 26 '15 at 8:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.