I am trying to extract some URLs from a web page using cURL command. Initially, I use the cURL command as below.

curl www.website.com/

Now, the website contains links to some other websites which am interested in extracting. So, I do a grep on the cURL command as below.

curl www.website.com/ | grep "<a href=" > new1.txt

It is extracting all lines which have <a href= in them. But am particularly interested only in lines which start with <a href= and end with title=

How can I modify the grep command?

  • 2
    If that's all you need, you can do grep "<a href=.*title=" but this can get complicated when parsing HTML.
    – terdon
    Feb 10 '14 at 19:31

This should work:

curl www.website.com/ | grep '^<a href=.*title=$' > new1.txt

This will select all lines that begin with <a href= and end in title=

Just saw Terdon's comment. You can use -P option with grep and make a non-greedy aka lazy alternative as follows:

curl www.website.com/ | grep -P '^<a href=.*?title=$' > new1.txt

Keeping in mind that HTML is not a regular language, and parsing it with regular expressions is nigh-impossible, you could try:

... | grep '^<a href=.*title=.*' > ...

Edit: Saw that you specified lines that start with <a href; the caret takes care of that condition.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.