I'm trying to extract the links from a specified url and redirect them to a file.

At first I tried using --spider , this is my code:

wget --spider --force-html -m http://file/path 2>&1 | grep '^--' | awk '{ print $3 }'| grep -v '\.\(css\|js\|png\|gif\|jpg\|ico\|txt\)$'| sort | uniq > links.txt

It works fine if I pass an url which does not have too many webpages , however given a far more complex url it stucks for some really good minutes.

I did some research , I tried to find a way to somehow print the processed output before getting the process itself killed , using timeout -s KILL 30 , but couldn't find any .

So I'm asking you , is there a way to get the links in a decent time ? or at least print the output that it got before interrupting the process ?

Thank you.

  • Please edit your question and i) give us one of the link you are working on so we can test; ii) show us the output you want; iii) explain if you just want the links in the URL itself or if you want to follow them and get all the links from the linked pages as well. If not, I don't understand what you mean by a "URL with many webages". – terdon Dec 22 '15 at 18:58

Take a look at the tee command for your second question:


It can 'split' output of commands to both a file and the console at the same time.

  • We expect answers to be mire comprehensive here. Please edit your answer and explain what this tool is, how to install it and how it would help the OP. As it stands, this is more of a comment than an answer. – terdon Dec 22 '15 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.