Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am trying to get homework papers with the command but for some reason, it does not return them. Any idea why?

$ wget -A.pdf -r -nd https://noppa.aalto.fi/noppa/kurssi/mat-2.3152/viikkoharjoitukset/

My goal is to download all homework and then merge them with pdftk and then search over them with some program -- but I cannot proceed (without again going to Firefox DownloadThemAll -plugin) until I get this working. Ideas why not working? Have sysadmins set up some bans for wgets or why does it fail?

I get only this robots.txt -file:

$ cat robots.txt 
User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /
Disallow: /cgi-bin/
share|improve this question
    
So what does happen when you type this command? –  Faheem Mitha Dec 20 '11 at 18:26
    
..it downloads only the robots.txt -file but no pdf altough I can see my course homeworks on the site. –  user2362 Dec 20 '11 at 18:35
    
...this is no fake site, official site by Aalto Uni (more here about it). I have tried this elsewhere time-to-time and worked so are there some special restrictions or why wget does not work? Because of https? –  user2362 Dec 20 '11 at 18:37
    
Does regular download via a web browser work? –  Faheem Mitha Dec 20 '11 at 20:01
    
@FaheemMitha: ofc -- used DownloadThemAll -addon in Firefox, now just wanted to use wget to replace it. Rush discovered the problem, an issue with robots.txt. –  user2362 Dec 20 '11 at 21:04
add comment

1 Answer

up vote 3 down vote accepted

Your problem is in robots.txt. Wget looks on it and sees

User-agent: *
Disallow: /

You can use tell wget to ignore the robots.txt file:

wget -A.pdf -r -nd -e robots=off  https://noppa.aalto.fi/noppa/kurssi/mat-2.3152/viikkoharjoitukset/

And it will work.

share|improve this answer
    
Yes this must be right "$wget -d -nd -r -A.pdf SITE 2&>debug", verified by debug msgs: "Not following https://noppa.aalto.fi/noppa/kurssi/mat-2.3152/viikkoharjoitukset/laskari_1_s.1-‌​8__ei_kotihtavaa.pdf because robots.txt forbids it. Decided NOT to load it." -- why have they this kind of stupid restrictions, no search engine can search them?! –  user2362 Dec 20 '11 at 18:49
    
I think it's a kind of protection from site copying. As you can see, google is allowed, so it is protection only from grabbers. –  rush Dec 20 '11 at 18:51
    
what is site grabbing? I know that sysadmins have banned services that provided better service than the official service (...well it is a bit...) and it is currently much harder to find anything because search engines do not index things. I can see only reason for the paranoia to hide information, I have only suspects... but thank you, got it worknig --back to studying. Some people still think we have no internet, kosh. –  user2362 Dec 20 '11 at 18:53
    
I meant site copying. May be it is only my country local slang. (: –  rush Dec 20 '11 at 18:55
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.