1

I want to download all the documents pointed to by a web page with wget.

When I run wget, it only downloads the webpage, not all the contents that the web page links to.

wget --content-disposition -v -S -r -nc -np -Kk -px --ignore-length \
     --content-disposition --proxy-user "XYZ" --proxy-passwd "PASS" \
     --no-check-certificate --load-cookies=cookies.txt --keep-session-cookies \
     'https://www.website.com/link/live?func=lab&objId=174914395&objAction=browse&viewType=1'

Wget downloads only one page which is https://www.website.com/link/live?func=lab&objId=174914395&objAction=browse&viewType=1. That webpage contains:

DataStringToVariables( '( {"currentPageNum":1,"totalPages":1,"totalCount":21,"dbTimeTaken":0,"timeTaken":1,"haveBigImages":"fa lse1","myrows":[{"dataId":"177020845","type2":"184","typeName":"Document","name":"Action_Mar_JKill%2Exls", etc.

There are many xls,docs, pdf, etc but wget doesn't download them. How can I download them?

migrated from serverfault.com Aug 14 '13 at 22:07

This question came from our site for system and network administrators.

1

Not sure if I'm getting your question, but I'm going to try and answer anyway.

What you're getting from the URL is JSONP, but JSONP isn't handled by wget at all. wget is just a downloading program, as noted here in the man page for wget:

GNU Wget is a free utility for non-interactive download of files from the Web.

wget does not support any parsing of the data it receives. If you wish to parse the data, use another programming language to import and process the data received here.

Assuming that you're writing some sort of shell script, you may also wish to use jsawk to fetch the data. See this answer for an example.

Once you have parsed the JSON(P), you can go ahead and download the files listed from the JSON(P) data with wget.

  • When passed the -r flag, wget can parse HTML data for links. – user26112 Aug 14 '13 at 23:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.