Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am trying to download all the links from aligajani.com. There are 7 of them, excluding the domain facebook.com which I wan't to ignore. Yes, so that means, I don't want to download links that start with facebook.com domain. Also, I would want them saved in a .txt file, line by line. So there would be 7 lines. Here's what I've tried so far. This just downloads everything. Don't want that.

wget -r -l 1 http://aligajani.com
share|improve this question
1  
Yes, wget downloads whole pages. Why do you think it might only save the links? –  michas Feb 26 at 6:44
    
Well wget has a command that downloads png files from my site. It means, somehow, there must be a command to get all the URLS from my site. I just gave you an example of what I am trying to do currently. –  Ali Gajani Feb 26 at 6:46
2  
You're trying to use completely the wrong tool for the job, this is not at all what wget is designed to do. In future, please don't assume a particular tool for a job when asking for a question; it's not a good way to ask questions in general, but it's especially poor practice in a technical venue. –  Chris Down Feb 26 at 6:52
    
Apologies, I am a noob to wget. I thought wget is powerful functionality built in for tasks like web crawling and more, so I was assuming it would do something like this. I did see the man page for wget and didn't find anything w.r.t what I wanted, so I came here. –  Ali Gajani Feb 26 at 7:08

2 Answers 2

up vote 5 down vote accepted

wget does not offer such an option. Please read its man page.

You could use lynx for this:

lynx -dump -listonly http://aligajani.com | grep -v facebook.com > file.txt

From its man page:

   -listonly
          for -dump, show only the list of links.
share|improve this answer

As others have pointed out, wget is not designed for this. You can however parse its output to get what you want:

$ wget http://aligajani.com -O - 2>/dev/null | 
    grep -oP 'href="\Khttp:.+?"' | sed 's/"//' | grep -v facebook > file.txt

That creates a file called file.txt with the following contents:

http://www.linkedin.com/pub/ali-ayaz-gajani/17/136/799
http://www.quora.com/Ali-Gajani
http://www.mrgeek.me/
http://twitter.com/aligajani
http://www.mrgeek.me
http://aligajani.com
share|improve this answer
    
Please don't parse html with regex: stackoverflow.com/a/1732454/1870481 –  michas Feb 26 at 21:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.