Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a site on a server that is basically a bunch of HTML pages, pictures and sounds.

I have lost my password to that server and I need to grab everything that is stored there. I can go page by page and save everything but the site has more than 100 pages.

I am using OSX. I have tried to use wget but I think the server is blocking that.

Is there any alternative I can use to grab that content?

share|improve this question
    
If you have physical access to the server , boot into single user mode and recover your password. debuntu.org/… –  spuder Aug 17 '13 at 18:12
add comment

2 Answers

up vote 13 down vote accepted

If the server is blocking wget, it is most likely doing it on the basis of the "User-agent:" field of the http header, since that is the only way for it to know in the first place. It could also be blocking your IP, in which case using different software won't help, or some scheme which identifies automation on the basis of how rapid a set of requests are (since real people don't browse 100 pages in 3.2 seconds). I have not heard of anyone doing that, but it is possible.

I also have not heard of a way to slow down wget, but there is a way to spoof the user-agent field:

wget --user-agent=""

Will according to the man page drop "User-agent:" completely, since it is not mandatory. If the server doesn't like that, try --user-agent="Mozilla/5.0" which should be good enough.

Of course, it would help if you explained better why you "think the server is blocking that". Does wget say anything, or just time out?

share|improve this answer
    
oh yeah! Thanks. I have missed that on the wget help. It is working now!!!!!!!!!!!!!!!!!!!!!!!! –  Digital Robot Aug 17 '13 at 13:27
4  
wget has a number of options to wait between queries, limit the rate or the amount downloaded. Check the info page for details. –  Stephane Chazelas Aug 17 '13 at 19:38
add comment

I usually use httrack for downloading/mirroring web content from a site.

$ httrack http://2011.example.com -K -w -O . -%v --robots=0 -c1 %e0

After it runs you're left with a directory structure that's local and browseable. For example:

$ ls -l
total 304
-rw-r--r--  1 saml saml   4243 Aug 17 10:20 backblue.gif
-rw-r--r--  1 saml saml    828 Aug 17 10:20 fade.gif
drwx------  3 saml saml   4096 Aug 17 10:20 hts-cache
-rw-rw-r--  1 saml saml    233 Aug 17 10:20 hts-in_progress.lock
-rw-rw-r--  1 saml saml   1517 Aug 17 10:20 hts-log.txt
-rw-------  1 saml saml 271920 Aug 17 10:22 hts-nohup.out
-rw-r--r--  1 saml saml   5141 Aug 17 10:20 index.html
drwxr-xr-x 10 saml saml   4096 Aug 17 10:21 2011.example.com

As it downloads you'll see the following type of output:

Bytes saved:    21,89KiB           Links scanned:   12/45 (+4)
Time:   2s                         Files written:   4
Transfer rate:  2,65KiB/s (2,65KiB/s)  Files updated:   1
Active connections:     1          Errors:  7

Current job: parsing HTML file (57%)
 request -  2011.example.com/cgi-bin/hostnames.pl   0B /    8,00KiB

It can be backgrounded and/or aborted and later resumed. This is just the tip of the iceberg in terms of it's features. There is also a GUI for both setting up a download and monitoring it as it progresses.

There is extensive documentation on the httrack website and by googling.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.