Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am trying to download all the links from aligajani.com. There are 7 of them, excluding the domain facebook.com which I wan't to ignore. Yes, so that means, I don't want to download links that start with facebook.com domain. Also, I would want them saved in a .txt file, line by line. So there would be 7 lines. Here's what I've tried so far. This just downloads everything. Don't want that.

wget -r -l 1 http://aligajani.com
share|improve this question
1  
Yes, wget downloads whole pages. Why do you think it might only save the links? – michas Feb 26 '14 at 6:44
    
Well wget has a command that downloads png files from my site. It means, somehow, there must be a command to get all the URLS from my site. I just gave you an example of what I am trying to do currently. – Ali Gajani Feb 26 '14 at 6:46
2  
You're trying to use completely the wrong tool for the job, this is not at all what wget is designed to do. In future, please don't assume a particular tool for a job when asking for a question; it's not a good way to ask questions in general, but it's especially poor practice in a technical venue. – Chris Down Feb 26 '14 at 6:52
    
Apologies, I am a noob to wget. I thought wget is powerful functionality built in for tasks like web crawling and more, so I was assuming it would do something like this. I did see the man page for wget and didn't find anything w.r.t what I wanted, so I came here. – Ali Gajani Feb 26 '14 at 7:08
up vote 8 down vote accepted

wget does not offer such an option. Please read its man page.

You could use lynx for this:

lynx -dump -listonly http://aligajani.com | grep -v facebook.com > file.txt

From its man page:

   -listonly
          for -dump, show only the list of links.
share|improve this answer

As others have pointed out, wget is not designed for this. You can however parse its output to get what you want:

$ wget http://aligajani.com -O - 2>/dev/null | 
    grep -oP 'href="\Khttp:.+?"' | sed 's/"//' | grep -v facebook > file.txt

That creates a file called file.txt with the following contents:

http://www.linkedin.com/pub/ali-ayaz-gajani/17/136/799
http://www.quora.com/Ali-Gajani
http://www.mrgeek.me/
http://twitter.com/aligajani
http://www.mrgeek.me
http://aligajani.com
share|improve this answer
    
Please don't parse html with regex: stackoverflow.com/a/1732454/1870481 – michas Feb 26 '14 at 21:43

Use the following in terminal:

      wget -r -p -k http://website

or

      wget -r -p -k --wait=#SECONDS http://website

Note: The second one is for websites that may flag you if downloading too quickly; may also cause a loss of service, so use second one for most circumstances to be courteous. Everything will be placed in a folder named the same as website in your root folder directory or whatever directory you have terminal in at time of executing command.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.