Is there any way to download / mirror a folder structure with many files from sourceforge using wget/curl/similar tool?

I've seen some examples using rsync or scp but I think that works only for SF project owners (cause it needs login, password and filesystem path).


Yes you can do it with wget.

If you are using HTTP

wget --no-check-certificate -e robots=off --user=user --password='pswd' -r URL

If you use FTP :

wget -r ftp://username:password@ftp.your_domain.com/

I usually use httrack for downloading/mirroring web content from a site.

$ httrack http://2011.example.com -K -w -O . -%v --robots=0 -c1 %e0

After it runs you're left with a directory structure that's local and browseable. For example:

$ ls -l
total 304
-rw-r--r--  1 saml saml   4243 Aug 17 10:20 backblue.gif
-rw-r--r--  1 saml saml    828 Aug 17 10:20 fade.gif
drwx------  3 saml saml   4096 Aug 17 10:20 hts-cache
-rw-rw-r--  1 saml saml    233 Aug 17 10:20 hts-in_progress.lock
-rw-rw-r--  1 saml saml   1517 Aug 17 10:20 hts-log.txt
-rw-------  1 saml saml 271920 Aug 17 10:22 hts-nohup.out
-rw-r--r--  1 saml saml   5141 Aug 17 10:20 index.html
drwxr-xr-x 10 saml saml   4096 Aug 17 10:21 2011.example.com

As it downloads you'll see the following type of output:

Bytes saved:    21,89KiB           Links scanned:   12/45 (+4)
Time:   2s                         Files written:   4
Transfer rate:  2,65KiB/s (2,65KiB/s)  Files updated:   1
Active connections:     1          Errors:  7

Current job: parsing HTML file (57%)
 request -  2011.example.com/cgi-bin/hostnames.pl   0B /    8,00KiB

It can be backgrounded and/or aborted and later resumed. This is just the tip of the iceberg in terms of it's features. There is also a GUI for both setting up a download and monitoring it as it progresses.

There is extensive documentation on the httrack website and by googling.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.