Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a list of 900 URLs. Each page contains one image. Some images are duplicates (with same URL). I want to download 900 images, including duplicates.

I was able to download all pages and the embedded images (and ignored all other file types) with wget. But it seems to me that wget "ignores" an image when it was already downloaded before. So I had 900 pages, but only around 850 images.

(How) can I tell wget to download duplicates, too? It could append _1, _2, … at the file name.


My wget command:

wget --input-file=urls.txt --output-file=log.txt --wait 1 --random-wait --page-requisites --exclude-domains code.jquery.com --span-hosts --reject thumbnail*.png -P downloadfolder

share|improve this question
    
Have you tried downloading images with -o flag? –  Karlson Nov 29 '12 at 16:46
    
This should be the default behavior, if you run wget with no other option. What command are you running exactly? –  Gilles Nov 29 '12 at 22:59
    
@Gilles: I added the wget command I used to my question. This is default behavior? Wouldn't that mean that you'd get the very same CSS file (with -p) for each page you download? –  unor Nov 30 '12 at 2:28
    
@Karlson: -o (same as --output-file) activates the log, right? I used this parameter. How could it help me? I tried to make sense out of the log file, but couldn't spot any information about duplicate images. –  unor Nov 30 '12 at 2:31
    
@unor Sorry I meant -O. Which allows you to save with a different name. –  Karlson Nov 30 '12 at 13:40
show 6 more comments

3 Answers

wget doesn't ignore the duplicates, it is most probably overwriting them. wget has a lot of options and I am not aware of all of them but I don't believe there is such an option to selectively add a suffix to discriminate duplicate filenames. You need to build this functionality yourself.

such as

#  mkdir /myarchivedir
#  
#  mkdir /tmp/mytempdir
#  cd /tmp/mytempdir
#  
#  i=1
#  while [ $i -le 900 ] 
#  do
#  wget http://mysite.com/somefile
#  file=$(ls)
#  mv $file /myarchivedir/${i}.${file}
#  (( i=$i+1 ))
#  done

as you can see, even if the $file variable is the same as another one before, since the i is changing value every time, when you move the file to /myarchivedir, it will not overwrite your duplicate named image.

hope this helps

share|improve this answer
    
Why was this downvoted? I'm not that savvy with scripts. I guess this can be used as bash script? Karlson added # in front of every line, but I'd need to remove those because this is the comment symbol? –  unor Dec 1 '12 at 0:23
    
@unor This is not a script. These are commands and # is the command prompt. –  Karlson Dec 28 '12 at 16:56
    
# generally indicates running as root. Is it really necessary to run this script as root? Otherwise, I'd replace these with $, which generally indicate running as a non-root user. –  Naftuli Tzvi Kay Feb 28 '13 at 1:38
add comment

I think your problem is --page-requisites. I suspect it will recognize the duplicates and avoids downloading them again. This is usually what you want. You get around it by calling wget once for each URL and have it download to a different folder each time, like so:

#!/bin/bash
let i=0
while IFS=$'\n' read url; do 
  let i++; 
  wget -nv -P $i --page-requisites "$url"; 
done < urls.txt

However, you only need the single image contained in each of the files. This should do the trick:

wget -i urls.txt -q -O - | ...extract image URLs... | wget -nv -i -

Extracting the URLs from the HTML can be as easy as looking for some verbatim URLs, e.g.

grep -Eo 'http://host/abc/[^"]+.jpg'

Or you need a little more effort. If your 900 URLs point to similar HTML it should not be a problem. In any case this will number the files on name collision.

share|improve this answer
add comment

The reason it's not saving duplicates for you is because you are using the --page-requisites (-p) option. Try adding the --no-directories (-nd) option to your command.

From the manpage (strong emphasis added by me):

When running Wget without -N, -nc, -r, or -p, downloading the same file in the same directory will result in the original copy of file being preserved and the second copy being named file.1. If that file is downloaded yet again, the third copy will be named file.2, and so on. (This is also the behavior with -nd, even if -r or -p are in effect.)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.