When I try to download pages with wget -r using a for loop it works fine but when I try it with -e robots=off it does not stop to that limit which I set for "for" loop, here is my code


lynx --dump  https://en.wikipedia.org/wiki/Cloud_computing |awk '/http/{print $2}'| grep https://en. | grep -v '.svg\|.png\|.jpg\|.pdf\|.JPG\|.php' >Pages.txt

grep -vwE "(http://www.enterprisecioforum.com/en/blogs/gabriellowy/value-data-platform-service-dpaas)" Pages.txt > newpage.txt

rm Pages.txt
egrep -v "#|$^" newpage.txt>try.txt
awk '!a[$0]++' try.txt>new.txt
rm newpage.txt
rm try.txt
mkdir -p htmlpagesnew
cd htmlpagesnew
for i in $( cat ../new.txt );
if [ $j -lt 10 ];
    let j=j+1;
    echo $j
    wget  -N -nd -r -e robots=off --wait=.25 $i  ;
find * -name "*" -exec md5sum '{}' + | perl -ane '$k{$F[0]}++; system("rm \"@F[1 
.. $#F]\"") if $k{$F[0]}>1'

find * -name '*.[[:digit:]]' -exec rm {} +

rm *[0-9]

find -type f |sed 's,\(.*\)/\(.*\)$,\1/\2\t\1/\L\2,'|sort|uniq -D -f 1|cut -f 
1|xargs rm

ls |wc -l

echo "done"

1 Answer 1


Your for loop is fine (except that it could be written more efficiently by using break when $j -ge 10, so that it does not continue iterating through the remaining elements.

The problem is you're expecting wget ... -r to fetch only one file per iteration. That's not how it works; the loop is only counting the number of invocations of wget ... -r and each such invocation can download an unbounded number of files.

The -e robots=off is unrelated, except that using robots.txt might result in stopping the recursion a lot sooner if many of the files are actually blocked to crawlers.

You're not really supposed to use -e robots=off. Depending on your jurisdiction, there could be legal consequences to doing so (I agree this sucks, scraping should be legal, but that's the situation at present).

  • so what could be the solution to download only 10 pages recursively using wget
    – Explorer
    Nov 29, 2016 at 6:00
  • i agree it is illegal to do this but this is part of my course assignment i have to complete@DepressedDaniel
    – Explorer
    Nov 29, 2016 at 6:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .