Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Let's say there is a site www.songs.com, I download songs from that site using wget -r. Then,say , 3 days later, there has been an update on the site and new songs have been added.

Is there any way I download only new songs using wget?

As it stands, if I give command

wget - r

It will download the whole website.

share|improve this question
you are looking for an rsync that is http based, right? – Thorsten Staerk Dec 23 '13 at 17:05
@ThorstenStaerk, I have heard of Rsync, but not ever used it. From what I know, Rsync is a diff tool. So in a broader sense, that is what I want. – Dude Dec 23 '13 at 17:07
up vote 3 down vote accepted

I think you can try using wget --no-clobber, but as mentioned above, you probably want to look into using a solution that is based on rsync rather than HTTP. Presuming that you have SSH access to the server, rsync can use that as a transport mechanism with rsync -za --stats -essh user@host.example.com:/path/to/files /path/to/local/copy/of/files.

Note though that wget --no-clobber -r will only get files that are new since the last check, and won't re-download new copies that have changed. That's why rsync is the better solution for the use-case you present.

Another alternative I found is the GPL software, HTTrack, which mirrors entire web sites, and can pull down subsequent differentials. It can be found here. Windows screen shot, but there are builds and/or source for Windows, OS X, Linux, BSD, and Android(!).

share|improve this answer
No I don't have the ssh access. I am talking about a general blog or music website. – Dude Dec 23 '13 at 17:21
I guess then wget -r --no-clobber is what you need, though you might want to be sure it's okay with the maintainer of the site that you're crawling and cloning it. (: – DopeGhoti Dec 23 '13 at 17:27
@JoeDimaggio, I've found another solution that might be something you could use. See above. – DopeGhoti Dec 23 '13 at 17:39
But, if in my case, only new files are being added, then wget -no clobber will work, right> – Dude Dec 23 '13 at 17:47
Yes, that should work as you need. – DopeGhoti Dec 23 '13 at 18:04

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.