I have a list of 900 URLs. Each page contains one image. Some images are duplicates (with same URL). I want to download 900 images, including duplicates.

I was able to download all pages and the embedded images (and ignored all other file types) with wget. But it seems to me that wget "ignores" an image when it was already downloaded before. So I had 900 pages, but only around 850 images.

(How) can I tell wget to download duplicates, too? It could append _1, _2, … at the file name.

My wget command:

wget --input-file=urls.txt --output-file=log.txt --wait 1 --random-wait --page-requisites --exclude-domains code.jquery.com --span-hosts --reject thumbnail*.png -P downloadfolder

  • Have you tried downloading images with -o flag?
    – Karlson
    Nov 29, 2012 at 16:46
  • This should be the default behavior, if you run wget with no other option. What command are you running exactly? Nov 29, 2012 at 22:59
  • @Gilles: I added the wget command I used to my question. This is default behavior? Wouldn't that mean that you'd get the very same CSS file (with -p) for each page you download?
    – unor
    Nov 30, 2012 at 2:28
  • @Karlson: -o (same as --output-file) activates the log, right? I used this parameter. How could it help me? I tried to make sense out of the log file, but couldn't spot any information about duplicate images.
    – unor
    Nov 30, 2012 at 2:31
  • @unor Sorry I meant -O. Which allows you to save with a different name.
    – Karlson
    Nov 30, 2012 at 13:40

3 Answers 3


I think your problem is --page-requisites. I suspect it will recognize the duplicates and avoids downloading them again. This is usually what you want. You get around it by calling wget once for each URL and have it download to a different folder each time, like so:

let i=0
while IFS=$'\n' read url; do 
  let i++; 
  wget -nv -P $i --page-requisites "$url"; 
done < urls.txt

However, you only need the single image contained in each of the files. This should do the trick:

wget -i urls.txt -q -O - | ...extract image URLs... | wget -nv -i -

Extracting the URLs from the HTML can be as easy as looking for some verbatim URLs, e.g.

grep -Eo 'http://host/abc/[^"]+.jpg'

Or you need a little more effort. If your 900 URLs point to similar HTML it should not be a problem. In any case this will number the files on name collision.


The reason it's not saving duplicates for you is because you are using the --page-requisites (-p) option. Try adding the --no-directories (-nd) option to your command.

From the manpage (strong emphasis added by me):

When running Wget without -N, -nc, -r, or -p, downloading the same file in the same directory will result in the original copy of file being preserved and the second copy being named file.1. If that file is downloaded yet again, the third copy will be named file.2, and so on. (This is also the behavior with -nd, even if -r or -p are in effect.)


wget doesn't ignore the duplicates, it is most probably overwriting them. wget has a lot of options and I am not aware of all of them but I don't believe there is such an option to selectively add a suffix to discriminate duplicate filenames. You need to build this functionality yourself.

such as

#  mkdir /myarchivedir
#  mkdir /tmp/mytempdir
#  cd /tmp/mytempdir
#  i=1
#  while [ $i -le 900 ] 
#  do
#  wget http://mysite.com/somefile
#  file=$(ls)
#  mv $file /myarchivedir/${i}.${file}
#  (( i=$i+1 ))
#  done

as you can see, even if the $file variable is the same as another one before, since the i is changing value every time, when you move the file to /myarchivedir, it will not overwrite your duplicate named image.

  • Why was this downvoted? I'm not that savvy with scripts. I guess this can be used as bash script? Karlson added # in front of every line, but I'd need to remove those because this is the comment symbol?
    – unor
    Dec 1, 2012 at 0:23
  • @unor This is not a script. These are commands and # is the command prompt.
    – Karlson
    Dec 28, 2012 at 16:56
  • # generally indicates running as root. Is it really necessary to run this script as root? Otherwise, I'd replace these with $, which generally indicate running as a non-root user. Feb 28, 2013 at 1:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .