wget and preventing files from downloading on a recursive wget

Question

I can use a command like wget -m -A.jpg http://www.mysite.tld/ but that will download every single file, then if it is not a jpg, it will delete the file, like this:

`HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html                             [           <=>                                                     ]   1.73K  --.-KB/s   in 9.5s   

2015-01-13 16:55:52 (186 B/s) - ‘index.html’ saved [1776]

Removing index.html since it should be rejected.`

I would like to download only the .jpg files without having to download every other file on the site and discarding it afterwards, since some of the files are very large and the rest of the files are very numerous.

Answer 1

That's part of wget parsing the tree for links. It only does this with htm/html (i.e. it isn't going to download every non-jpg file, just every non-jpg file that happens to be a .htm or .html file). It has to, it's part of the functionality. See, https://www.gnu.org/software/wget/manual/html_node/Types-of-Files.html

Note: if you can connect to mysite.tld and issue a command to list the files and redirect the output to a text file, you could use wget -i foo.txt to read the download urls from the file rather than having wget work its way through the links to get them.

Example: from remote machine: ls *.jp* > foo.txt then from local machine: wget -i -F foo.txt -B http://www.mysite.tld/foo.jpg (obviously you'll have to move foo.txt to your local machine or otherwise point to it)