0

I have a text file with about 50 URLs Am trying to curl each of the URLs in the text file and grep to output a couple of lines from the GREP Match

I cannot understand why it wont work, as if i echo each line, it displays the URL correctly. If i run the CURL manually with a URL and Grep it works. But once in the loop it simply does not output anything.

while IFS= read -r line
do
echo "$line"
curl -s "$line" | grep "Telephone"
done <  links.txt

Sample Input:

https://qguild.co.uk/our-butchers/simpsons-butchers-south-hykeham/
https://qguild.co.uk/our-butchers/simpsons-butchers-spalding/
https://qguild.co.uk/our-butchers/simpsons-butchers-lincoln/
https://qguild.co.uk/our-butchers/simpsons-butchers-2/

Sample Output:

curl -s "https://qguild.co.uk/our-butchers/simpsons-butchers-south-hykeham/" | grep "Telephone"

<p>Simpsons Butchers,<br />Pennell`s Garden Centre,<br />Newark Road, South Hykeham, Lincoln, LN6 9NT. Telephone: 01522 695419<br />Monday - Saturday : 8am - 5.30pm Sunday : 10am - 4pm</p>

<p>Pennells Garden Centre, Newark Road, SOUTH HYKEHAM, LINCOLN LN6 9NT</p>                        
<p>Telephone: <a href="tel:01522-695419">01522 695419</a></p>                        <p>Email: <a href="mailto:[email protected]">[email protected]</a></p>                        
<p>Website: <a href="http://gsimpsonbutchers.co.uk/" target="_blank" rel="noopener noreferrer">http://gsimpsonbutchers.co.uk/</a></p>
2
  • 1
    We need sample input and expected output. Please, read MCVE Commented Feb 13, 2023 at 9:04
  • Provided sample input and output Commented Feb 13, 2023 at 9:22

3 Answers 3

2

Using and :

If you want to keep the most of your code:

dos2unix links.txt

then

while IFS= read -r url; do
     xidel -e '//p[contains(., "Telephone")]/a/text()' -s "$url"
done < links.txt

or even shorter:

xargs < links.txt xidel -e '//p[contains(., "Telephone")]/a/text()' -s

Output

01522 695419
01775 723157
01522 525374
01529 460403
1
  • 1
    Nice clean solution, I was using Grep as i was also wanting the previous matched content, However using your example xidel and xargs i was able to output the information in a much cleaner format. Thanks. Commented Feb 13, 2023 at 10:10
1

Your links.txt file is likely a DOS or Windows text file.

As such, it would have an extra carriage-return character at the end of each line, interfering with the end of the URL strings.

Convert your list of URLs to Unix text format with e.g. dos2unix, then try again.

-2

using input redirection and a while loop here is probably over-complicating things. Using cat and a for loop is simpler in this situation:

IFS=$(printf "\n")
for line in $(cat links.txt)
do
    echo "$line"
    curl -s "$line" | grep "Telephone"
done
2
  • Could you explain what using a for loop would solve here? A for loop is usually used for iterating over a static list (the shell expands the list in memory and iterates over it), while a while loop is used when the input is of indeterminate length, as when reading from a file, for example.
    – Kusalananda
    Commented Feb 13, 2023 at 9:20
  • As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.
    – Community Bot
    Commented Feb 13, 2023 at 17:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .