As a possible work-around for another issue, is it possible to make wget print the URLs it finds in the target document that it considers as page requisites? I'd rather not build scraping code for the simple task I'm working on.


You could do something like:

wget -nv -p -nd --delete-after <url>

Which will output something like:

2017-06-01 21:37:41 URL:<url> [19101] -> "index.html.3" [1]
2017-06-01 21:37:42 URL:<url>robots.txt [24/24] -> "robots.txt" [1]
2017-06-01 21:37:42 URL:<url>rc/1502.css [19106/19106] -> "1502.css" [1]
2017-06-01 21:37:42 URL:<url>favicon.ico [5390/5390] -> "favicon.ico" [1]
2017-06-01 21:37:42 URL:<url>avatar/nopic.jpg [893/893] -> "nopic.jpg

That could be quickly converted or filtered to something that might suit your needs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.