I am learning to use the various wget commands (man wget). But I'm having trouble learning how to download specific files from some sites (on the whole site they are downloaded). On some sites I get on others do not.
For example, why can not I do the same with the files on this site?
https://www.ictsd.org/sites/default/files/review/bridgesweekly22-35a_0.pdf
wget -r -nd -A pdf --accept-regex "review/.*\.pdf" 'https://www.ictsd.org/sites/default/files/'
I'm actually trying to download all the pdf files from the "files" folder and their subfolders.
But even just restricting the download to the "file/review" folder. I can not. I also tried downloading all the pdfs from the site and I could not (on other sites it was possible). Any suggestion? Besides the query to "man wget" where could you learn more about it?
wget https://www.ictsd.org/sites/default/files/review/bridgesweekly22-35a_0.pdf
-- I think there is a reason why you cannot use wild-cards to see all the [pdf] files. You need a web browser to do that, and the web site must be set to allow it.wget
. Maybe or maybe not with some other non-interactive tool. I think the public internet and its websites is made for interactive use, except in some cases, when archive files (containing several compressed files) are available for example via ftp. -- You could get the names (web addresses) of all the review files interactively, save it in a local file and make a shellscript, that useswget
and reads the names from the local file.--spider
so I can't promise anything. I use wget to get files directly, and I have made some scripts that people can use to fetch files, that I have uploaded. You can ask at the web site, if there is a way to download many of the pdf files via one web address, for example to a compressed archive.