4

I have read the Wget manual, but unfortunately it does not seem to address my issue, so I would be most grateful if someone could offer me a bit of assistance.

We have a website, (say) website.com, which links directly to (say) website.com/1/, website.com/2/, ... etc.

Now each page website.com/r/, where r is an integer, links to a number of pdf documents. Rather than them being located at website.com/r/doc-i.pdf - which would be convenient - they are all located at website.com/files/doc-i.pdf.

Thus, when I run the command wget -r -l 2 -A pdf website.com, I will of course end up with a big folder named "files", with all the pdf documents contained within it.

I would much prefer, however, that they be organised into different folders named 1, 2, ..., n, that correspond to the page from which they were downloaded. Since I will be downloading in total around 10,000 pdf files, I would rather not have to do this manually.

So how do I tell Wget to organise the files, not by the website directory structure, but by the route in which it took to get to the file?

I hope my explanation is clear, and that this is not too difficult to achieve.

  • There isn't any easy way to do it, as far as I can tell. You could parse the logs and move the files after the fact, but that wouldn't be robust. Plus, if you'd try the same operation again at a later time you'd have to download all PDF files from scratch. It might make sense to do it the other way around: keep the PDFs in the same folder, and add symlinks to classify them. – Satō Katsura Sep 23 '16 at 7:54
  • @SatoKatsura Ah, well that is unfortunate. I guess I'll have to just do it manually then. – user85798 Sep 23 '16 at 8:11
1

(untested) The following needs some tunning, is just a general idea:

### get level1
wget -r -l  website.com/      

#### for each html file otained,
for a in $(find website.com -name '*.html' )
do 
  ### get level 2 but prefix it with the base name
  b=$(basename $a)
  wget -P $b -r -l 1 -A pdf http://$a 
done
  • probably the find will need some tunnig
  • perhaps add something like mv $b/website.com/files FINAL/$b to reduce the levels

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.