Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 2 down vote favorite
1

I am trying to get homework papers with the command but for some reason, it does not return them. Any idea why?

$ wget -A.pdf -r -nd https://noppa.aalto.fi/noppa/kurssi/mat-2.3152/viikkoharjoitukset/

My goal is to download all homework and then merge them with pdftk and then search over them with some program -- but I cannot proceed (without again going to Firefox DownloadThemAll -plugin) until I get this working. Ideas why not working? Have sysadmins set up some bans for wgets or why does it fail?

I get only this robots.txt -file:

$ cat robots.txt 
User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /
Disallow: /cgi-bin/
share|improve this question
So what does happen when you type this command? – Faheem Mitha Dec 20 '11 at 18:26
..it downloads only the robots.txt -file but no pdf altough I can see my course homeworks on the site. – user2362 Dec 20 '11 at 18:35
...this is no fake site, official site by Aalto Uni (more here about it). I have tried this elsewhere time-to-time and worked so are there some special restrictions or why wget does not work? Because of https? – user2362 Dec 20 '11 at 18:37
Does regular download via a web browser work? – Faheem Mitha Dec 20 '11 at 20:01
@FaheemMitha: ofc -- used DownloadThemAll -addon in Firefox, now just wanted to use wget to replace it. Rush discovered the problem, an issue with robots.txt. – user2362 Dec 20 '11 at 21:04

1 Answer

up vote 3 down vote accepted

Your problem is in robots.txt. Wget looks on it and sees

User-agent: *
Disallow: /

You can use tell wget to ignore the robots.txt file:

wget -A.pdf -r -nd -e robots=off  https://noppa.aalto.fi/noppa/kurssi/mat-2.3152/viikkoharjoitukset/

And it will work.

share|improve this answer
Yes this must be right "$wget -d -nd -r -A.pdf SITE 2&>debug", verified by debug msgs: "Not following https://noppa.aalto.fi/noppa/kurssi/mat-2.3152/viikkoharjoitukset/laskari_1_s.1-‌​8__ei_kotihtavaa.pdf because robots.txt forbids it. Decided NOT to load it." -- why have they this kind of stupid restrictions, no search engine can search them?! – user2362 Dec 20 '11 at 18:49
I think it's a kind of protection from site copying. As you can see, google is allowed, so it is protection only from grabbers. – rush Dec 20 '11 at 18:51
what is site grabbing? I know that sysadmins have banned services that provided better service than the official service (...well it is a bit...) and it is currently much harder to find anything because search engines do not index things. I can see only reason for the paranoia to hide information, I have only suspects... but thank you, got it worknig --back to studying. Some people still think we have no internet, kosh. – user2362 Dec 20 '11 at 18:53
I meant site copying. May be it is only my country local slang. (: – rush Dec 20 '11 at 18:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.