5

I am working on CentOS 6.4 and I am new to this operating system.

I was downloading a 5 GB file using wget command. I observed that it was trying to download the file from different IP addresses (54.240.168.41), which was blocked by the proxy server. So I got this specific IP address opened by the network support and the download started working.

Since it was a huge file, I left it to complete the execution overnight. Next morning, due to some network error, the download stopped. only 42% was downloaded. I tried to download the file using -c option of the wget command. However, wget keeps trying to connect to different IP addresses starting with 54.xxx.xxx.xxx, except the IP address 54.240.168.41.

My question is, how I would tell wget to download from a specific IP addresses which is NOT blocked by the network?.

This is the command that I am executing

wget --continue http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.2.0/HDP-2.3.2.0-centos6-rpm.tar.gz 
2
  • 2
    What's the actual command you used?
    – muru
    Nov 16, 2015 at 3:27
  • I have edited the question to have actual command.
    – KurioZ7
    Nov 16, 2015 at 3:32

3 Answers 3

13

This worked for me when switching DNS, and needed to access old server by IP but specified host header to route to my account at old server.

wget http://198.38.82.5/something.tar.gz --header "Host: domain-at-server.net"
2
  • My linux box exposes 3 IP4 addresses and a range of IP6 addresses. Can I set host header to one of these addresses, and wget will download through that network interface?
    – P i
    Oct 3, 2021 at 21:16
  • technically you could download by using IP directly. It depends how the server is configured, my solution was for shared hosting service where multiple domains are hosted from same IP. If you use Nginx on your VPS then you can configure host names per IP, and then you provide wget with the host name assigned to IP you want to use Oct 5, 2021 at 21:00
3

I resolved this by using the option --no-dns-cache. When this option was given, it started listing the IP address 54.240.168.41 in the list of IPs it was trying to connect to.

wget --no-dns-cache --continue http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.2.0/HDP-2.3.2.0-centos6-rpm.tar.gz
2

If you use a hostname instead of an IP in your command, you run the risk that the DNS entry for that hostname changes between runs. Instead, use the actual IP:

wget --continue http://54.240.168.41/HDP/centos6/2.x/updates/2.3.2.0/HDP-2.3.2.0-centos6-rpm.tar.gz 

But this won't help if the server has actually moved to a separate IP. Consider asking your proxy admins to provide an exemption by hostname instead of by IP.

1
  • 4
    You'd actually probably need to provide --header 'Host: public-repo-1.hortonworks.com' as well, since the server side may serve different content under different names. Nov 16, 2015 at 5:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.