12

I am extracting URLs from a website using cURL as below.

curl www.somesite.com | grep "<a href=.*title=" > new.txt

My new.txt file is as below.

<a href="http://website1.com" title="something">
<a href="http://website1.com" information="something" title="something">
<a href="http://website2.com" title="some_other_thing">
<a href="http://website2.com" information="something" title="something">
<a href="http://websitenotneeded.com" title="something NOTNEEDED">

However, I need to extract only the below information.

<a href="http://website1.com" title="something">
<a href="http://website2.com" information="something" title="something">

I am trying to ignore the <a href which have information in them and whose title end with NOTNEEDED.

How can I modify my grep statement?

  • Is the output you're showing here correct? The text describing it doesn't make sense along with this example. – slm Feb 10 '14 at 19:55
  • 1
    Aren't you looking for curl www.somesite.com | grep "<a href=.*title=" | grep -v NOTNEEDED > new.txt? – terdon Feb 10 '14 at 21:33
  • @terdon, exactly that was what I was looking for. I can accept it as an answer if you post it. – Ramesh Feb 10 '14 at 21:36
  • Ramesh, it's basically @slm's answer. I have just edited it so you can accept it. – terdon Feb 10 '14 at 21:36
  • oh yeah, I did not realize pipe was this powerful. I have accepted it as an answer. Thanks! – Ramesh Feb 10 '14 at 21:38
16

I'm not fully following your example + the description but it sounds like what you want is this:

$ grep -v "<a href=.*title=.*NOTNEEDED" sample.txt 
<a href="http://website1.com" title="something">
<a href="http://website1.com" information="something" title="something">
<a href="http://website2.com" title="some_other_thing">
<a href="http://website2.com" information="something" title="something">

So for your example:

$ curl www.example.com | grep -v "<a href=.*title=" | grep -v NOTNEEDED > new.txt
  • I have a class in the <a href section. Basically, I do not want that in my output. – Ramesh Feb 10 '14 at 21:31
9

The grep man page says:

-v, --invert-match
    Invert the sense of matching, to select non-matching lines. (-v is specified by POSIX .) 

You can use regular expressions for multiple inversions:

grep -v 'red\|green\|blue'

or

grep -v red | grep -v green | grep -v blue

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.