Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am trying to download all the links from aligajani.com. There are 7 of them, excluding the domain facebook.com which I wan't to ignore. Yes, so that means, I don't want to download links that start with facebook.com domain. Also, I would want them saved in a .txt file, line by line. So there would be 7 lines. Here's what I've tried so far. This just downloads everything. Don't want that.

wget -r -l 1 http://aligajani.com
share|improve this question
Yes, wget downloads whole pages. Why do you think it might only save the links? – michas Feb 26 '14 at 6:44
Well wget has a command that downloads png files from my site. It means, somehow, there must be a command to get all the URLS from my site. I just gave you an example of what I am trying to do currently. – Ali Gajani Feb 26 '14 at 6:46
You're trying to use completely the wrong tool for the job, this is not at all what wget is designed to do. In future, please don't assume a particular tool for a job when asking for a question; it's not a good way to ask questions in general, but it's especially poor practice in a technical venue. – Chris Down Feb 26 '14 at 6:52
Apologies, I am a noob to wget. I thought wget is powerful functionality built in for tasks like web crawling and more, so I was assuming it would do something like this. I did see the man page for wget and didn't find anything w.r.t what I wanted, so I came here. – Ali Gajani Feb 26 '14 at 7:08
up vote 10 down vote accepted

wget does not offer such an option. Please read its man page.

You could use lynx for this:

lynx -dump -listonly http://aligajani.com | grep -v facebook.com > file.txt

From its man page:

          for -dump, show only the list of links.
share|improve this answer
It dumps the links of a single page. Is there a way to do this recursively? – That Brazilian Guy May 27 at 14:28
Have a look at the man page of lynx. I'm pretty sure it has no option for this. Maybe you want wget -r. Have a look at man wget. Maybe you want to ask a separate question saying what you really want to do and why. – michas May 27 at 14:33

As others have pointed out, wget is not designed for this. You can however parse its output to get what you want:

$ wget http://aligajani.com -O - 2>/dev/null | 
    grep -oP 'href="\Khttp:.+?"' | sed 's/"//' | grep -v facebook > file.txt

That creates a file called file.txt with the following contents:

share|improve this answer
Please don't parse html with regex: stackoverflow.com/a/1732454/1870481 – michas Feb 26 '14 at 21:43

Use the following in terminal:

      wget -r -p -k http://website


      wget -r -p -k --wait=#SECONDS http://website

Note: The second one is for websites that may flag you if downloading too quickly; may also cause a loss of service, so use second one for most circumstances to be courteous. Everything will be placed in a folder named the same as website in your root folder directory or whatever directory you have terminal in at time of executing command.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.