I want to download all *httpd* RPM file from remote CentOS mirror and I am trying he following command but it doesn't seem to work

[root@yum foo]# wget -r --no-parent -A "*httpd*" https://mirrors.edge.kernel.org/centos/7.5.1804/os/x86_64/Packages/

I see that it created a directory structure, but there are no files in the directories.

[root@yum foo]# ls

what I am doing wrong?

1 Answer 1


The web site you're trying to do this on has a robots.txt file that includes

User-Agent: *
Disallow: /

When in recursive mode, wget obeys this file, and so refuses to do recursive copies.

  • any solution here or any other tool like curl ?
    – Satish
    Dec 6, 2018 at 3:02
  • 1
    Use a different mirror that doesn't have a robots.txt ? eg wget -r --no-parent -A "*httpd*" https://mirrors.xmission.com/centos/7.5.1804/os/x86_64/Packages/ works as expected Dec 6, 2018 at 3:12
  • if the robots file is indeed the problem, you may also use -e robots=off to ignore the robots file Sep 10, 2021 at 9:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .