I have a web page, say https://test.salesforce.com. I would like to search for a pattern, e.g. "src" at the beginning of a line in the entire webpage content. If the pattern is found, I need to retrieve that particular line which starts with the pattern and save it to a text file. I have tried curl and wget but I'm not getting proper output. So far i have tried the following

content=$(wget https://test.salesforce.com -q -O -)  
echo $content | grep -i src > test.txt
  • @Erathiel content=$(wget test.salesforce.com -q -O -) echo $content | grep -i src > test.txt – RRaju May 21 '15 at 17:26
  • Please edit your original post and paste the exact command you've tried, preferably formatted as code. From what you've pasted here it's unclear whether this would even be executed properly. – Erathiel May 21 '15 at 17:33

Try something like

wget -qO- http://www.google.com/ | tr " " \\n | fgrep src= | tr \" \\n | fgrep -v src=

which will, however, output URLs as they are written in the file - which may be relative the the base URL.

  • Thank you for the response When i say web content, I would require only what showing in webpage, No properties or publisher details or anything like that. Is there any way we can get that ? – RRaju May 21 '15 at 18:36
  • In that case you need to analyse the whole tag, including stuff like <img and >. It is very helpful in these cases to know how the webpage code looks like (e.g. coming from the same source, like a CMS) to see where there are line breaks etc. If you need something which can apply to many web pages you need a more sophisticated SGML, XML and HTML5 parser. – Ned64 May 21 '15 at 18:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.