0

I am trying to make a list of URLs of webpages, based on whether the webpages contain the text "XYZ".

The URLs are of the form:

https://www.website.tld/page.php?var1=???&var2=static

??? is a number starting with 1, incremented by 1 each time, until an error page is encountered (say a page containing the text "ERROR”)

I want to dump the URLs of the positive matches into an output file. I read curl can sequentially scan such URLs, and its output can be passed to grep. However, I am unsure how I can retrieve and save the URL, after the grep output.

  • 3
    I feel like you're asking "If I'm at a known location and drive following given directions, how do I know where I am?" If you ran curl, how do you not know the URL? – G-Man Says 'Reinstate Monica' Dec 8 '18 at 3:54
  • I do know the URL, but how do I output it after grepping on the curl output (webpage content)? – halfbytecode Dec 8 '18 at 4:00
  • 2
    I still don't understand the question.  Given a string, do you know how to output it?  (Hint: echo or printf.) – G-Man Says 'Reinstate Monica' Dec 8 '18 at 4:27
  • Fetch webpages using curl, determine if the webpages (not URLs) contain the required text, output the URLs of positive matches. – halfbytecode Dec 8 '18 at 4:35
  • Rather than adding information in the comments, you should edit the question to improve its quality (see How to Ask). All relevant details should be in the question itself – not in the comments. – Anthony Geoghegan Dec 8 '18 at 19:49
2

It might be easier to generate the URLs without curl:

for ((i=1; i<1000; i++)); do
    url="https://www.website.tld/page.php?var1=${i}&var2=static" 
    if curl -s "$url" | grep -q XYZ; then
        echo "$url" >> positive-matches.txt
    fi
od
| improve this answer | |
  • Thank you. I started working on a script, and your answer helped me. – halfbytecode Dec 8 '18 at 11:27
  • 2
    @halfbytecode I've upvoted this answer but I'd suggest that you also accept it (click on the check mark) to show your appreciation as it answers your question and you don't yet have the ability to upvote answers. – Anthony Geoghegan Dec 8 '18 at 19:43
-1

I have made a working script. Sharing it in case someone finds it helpful. @nohillside 's answer helped me.

#!/bin/bash

count=1

while true
do
    url="https://www.website.tld/page.php?var1=${count}&var2=static"

    text=`curl -s "$url"`

    if echo "$text" | grep -q "ERROR"
    then
        break
    elif echo "$text" | grep -q "XYZ"
    then
        echo "$url" >> matches.txt
    fi

    ((count++))

done
| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.