0

I am learning to use the various wget commands (man wget). But I'm having trouble learning how to download specific files from some sites (on the whole site they are downloaded). On some sites I get on others do not.

For example, why can not I do the same with the files on this site?

https://www.ictsd.org/sites/default/files/review/bridgesweekly22-35a_0.pdf

wget -r -nd -A pdf --accept-regex "review/.*\.pdf" 'https://www.ictsd.org/sites/default/files/'

I'm actually trying to download all the pdf files from the "files" folder and their subfolders.

But even just restricting the download to the "file/review" folder. I can not. I also tried downloading all the pdfs from the site and I could not (on other sites it was possible). Any suggestion? Besides the query to "man wget" where could you learn more about it?

5
  • You made things complicated. The following straightforward command works for me: wget https://www.ictsd.org/sites/default/files/review/bridgesweekly22-35a_0.pdf -- I think there is a reason why you cannot use wild-cards to see all the [pdf] files. You need a web browser to do that, and the web site must be set to allow it.
    – sudodus
    Jan 9, 2019 at 19:13
  • @sudodus but I do not want to download only this files. I want to download all of the /review/*.pdf folder
    – Rafael
    Jan 9, 2019 at 19:16
  • I see, and I am afraid, that it will not work with wget. Maybe or maybe not with some other non-interactive tool. I think the public internet and its websites is made for interactive use, except in some cases, when archive files (containing several compressed files) are available for example via ftp. -- You could get the names (web addresses) of all the review files interactively, save it in a local file and make a shellscript, that uses wget and reads the names from the local file.
    – sudodus
    Jan 9, 2019 at 19:25
  • do you think using the --spider I can pull the links and then with xidel leave only the pdfs and use the "wget -i" with the pdfs? the problem is that I think the spider file would have been quite large (I'll try). Do you have any suggestions for shortening the spider file? @sudodus
    – Rafael
    Jan 9, 2019 at 19:55
  • You can try :-) I have no own experience of using --spider so I can't promise anything. I use wget to get files directly, and I have made some scripts that people can use to fetch files, that I have uploaded. You can ask at the web site, if there is a way to download many of the pdf files via one web address, for example to a compressed archive.
    – sudodus
    Jan 9, 2019 at 20:20

1 Answer 1

Reset to default
3

For wget to be able to grab a whole bunch of files, it needs to be able to find them under the directory you specify. In other words, when you navigate to https://www.ictsd.org/sites/default/files/review/ in a web browser, you should be able to see a link to the pdf there. If the link can be seen in your browser, then it can also be seen by wget.

When I navigate https://www.ictsd.org/sites/default/files/review/ in Firefox, there is a timeout and the error message obtains:

The page isn’t redirecting properly

Since navigating to the directory does not provide an index of the available files, there is no way for wget to see whatever you expect it to see.

Whereas when I put the full path to the particular pdf in the address, Firefox does find it, which is consistent with wget's behaviour.

One can speculate that the website owner has done this on purpose to prevent automated retrieval of all the files at once. If, on the other hand, you believe it is simply an error with the web service, and they have said the files you are after should be visible from the containing directory, you could get in touch with them and let them know about the problem.

Or, if there is some other index linking to all the pdfs, you could possibly use that.

If you know in advance the names of the particular pdfs you want, you could put all the links in a file and have wget read from it like so:

wget -i links.txt
6
  • Thanks! Please see my comment in the question ("Do you think ...") @cryptarch
    – Rafael
    Jan 9, 2019 at 19:59
  • @Rafael I had not heard of xidel before, and haven't used it. Using --spider could help you get the index you want, as @sudodus said, but no guarantees. How do you know there is more than one pdf to obtain? More context might help me expand the answer a bit more. At the end of the day, wget is constrained by the same laws of hypertext transfer protocol as a web browser is: if your browser can't see it, neither can wget.
    – cryptarch
    Jan 9, 2019 at 20:47
  • Each webpage link (ictsd.org/bridges-news/bridges/issue-archive) takes you to a page that has the "download" button. This button takes you to the folder with the pdfs. That is, each link this webpage has a corresponding pdf. This part of the site is organized by year from 2000 to 2018 and each year more than 40 links and their respective pdf
    – Rafael
    Jan 9, 2019 at 21:08
  • @Rafael Yeah, I see. When I tried using --spider it was rejected because the site has disallowed recursion. The --mirror option seems to (kind of) work, except I haven't yet figured out how to get it to harvest the pdfs. But I did notice that the filenames follow a pattern: bridgesweekly<volume>-<issue>[a-z].pdf. That would make it fairly easy to set up a links.txt.
    – cryptarch
    Jan 9, 2019 at 21:29
  • how would you use the pattern to create the wget line?
    – Rafael
    Jan 9, 2019 at 21:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .