1

I'm executing the command:

wget -b --no-clobber -nc -w 0 -Q 0 -r -E -e robots=off -U mozilla -t 0 --no-dns-cache -4 -R gif,jpeg,tif,jpg,pdf,bmp,png,css,js http://www.mywebsitearchive.com

Problem: If I re-run the command (when necessary if wget abruptly stops) I get duplicates for [some] files, such as example.html and example.1.html. Oddly enough a .2.html .3.html is never created. Any idea how to prevent this?

Improve this question

1 Answer 1

Reset to default
4

Try with the -c option:

   -c
   --continue
       Continue getting a partially-downloaded file.  This is useful when
       you want to finish up a download started by a previous instance of
       Wget, or by another program.  For instance:

               wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z

       If there is a file named ls-lR.Z in the current directory, Wget
       will assume that it is the first portion of the remote file, and
       will ask the server to continue the retrieval from an offset equal
       to the length of the local file.

So, something like (I removed --no-clobber, that's what -nc means):

wget -b -c -nc -w 0 -Q 0 -r -E -e robots=off -U mozilla -t 0 --no-dns-cache -4 -R gif,jpeg,tif,jpg,pdf,bmp,png,css,js http://www.mywebsitearchive.com
Improve this answer
6
  • Is there a way to delete the files automatically (i.e: test.1.html) via a -command? If the file isn't complete I just prefer it be gone instead of resuming and having to rm them.
    – Guest
    Commented Mar 10, 2014 at 19:59
  • @Guest remove the -nc option then. This is explained quite clearly in the --no-clobber option of man wget.
    – terdon ♦
    Commented Mar 10, 2014 at 20:02
  • noclobber has to be on I don't want existing files to be overwritten. If I enable continue and noclobber is on what happens?
    – Guest
    Commented Mar 10, 2014 at 20:31
  • @Guest um, you don't want existing files to be overwritten but you do want it to delete the files automatically? How would that be possible? Anyway, please read man wget, it is all explained there.
    – terdon ♦
    Commented Mar 10, 2014 at 20:36
  • 2
    @Guest yes. But your problem is that if you have already downloaded test.html and try to download again, it will create test.1.html. That is what the --no-clobber option does. If you don't want to overwrite it, just don't use --no-clobber (-nc). There are only two possibilities: either existing files are overwritten or they're not. How else could this work?
    – terdon ♦
    Commented Mar 10, 2014 at 21:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .