Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm executing the command:

wget -b --no-clobber -nc -w 0 -Q 0 -r -E -e robots=off -U mozilla -t 0 --no-dns-cache -4 -R gif,jpeg,tif,jpg,pdf,bmp,png,css,js http://www.mywebsitearchive.com

Problem: If I re-run the command (when necessary if wget abruptly stops) I get duplicates for [some] files, such as example.html and example.1.html. Oddly enough a .2.html .3.html is never created. Any idea how to prevent this?

share|improve this question

1 Answer 1

up vote 4 down vote accepted

Try with the -c option:

       Continue getting a partially-downloaded file.  This is useful when
       you want to finish up a download started by a previous instance of
       Wget, or by another program.  For instance:

               wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z

       If there is a file named ls-lR.Z in the current directory, Wget
       will assume that it is the first portion of the remote file, and
       will ask the server to continue the retrieval from an offset equal
       to the length of the local file.

So, something like (I removed --no-clobber, that's what -nc means):

wget -b -c -nc -w 0 -Q 0 -r -E -e robots=off -U mozilla -t 0 --no-dns-cache -4 -R gif,jpeg,tif,jpg,pdf,bmp,png,css,js http://www.mywebsitearchive.com
share|improve this answer
Is there a way to delete the files automatically (i.e: test.1.html) via a -command? If the file isn't complete I just prefer it be gone instead of resuming and having to rm them. –  Guest Mar 10 at 19:59
@Guest remove the -nc option then. This is explained quite clearly in the --no-clobber option of man wget. –  terdon Mar 10 at 20:02
noclobber has to be on I don't want existing files to be overwritten. If I enable continue and noclobber is on what happens? –  Guest Mar 10 at 20:31
@Guest um, you don't want existing files to be overwritten but you do want it to delete the files automatically? How would that be possible? Anyway, please read man wget, it is all explained there. –  terdon Mar 10 at 20:36
@Guest yes. But your problem is that if you have already downloaded test.html and try to download again, it will create test.1.html. That is what the --no-clobber option does. If you don't want to overwrite it, just don't use --no-clobber (-nc). There are only two possibilities: either existing files are overwritten or they're not. How else could this work? –  terdon Mar 10 at 21:15

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.