32

I have a server in USA (Linux box B), and my home PC (Linux box A), and I need download a file from website C,

The issue is, it is very slow to download a file direct from A, so I need download the file when I log in B, and sftp get the file from A.

Is there any way that I can download file and use B as proxy directly through only one line command?

7 Answers 7

51

(Strange situation, doesn't something like the triangle inequality hold for internet routing?)

Anyway, try the following, on A, ssh into B with a -D argument,

ssh -D 1080 address-of-B

which acts as a SOCKS5 proxy on 127.0.0.1:1080, which can be used by anything supporting SOCKS5 proxied connections. Apparently, wget can do this, by using the environment variable

export SOCKS_SERVER=127.0.0.1:1080
wget http://server-C/whatever

Note that sometimes curl is more handy (i.e. I'm not sure if wget can do hostname lookups via SOCKS5; but this is not one of your concerns I suppose); also Firefox is able to work completely through such a SOCKS5 proxy.

Edit I've just now noticed that you're looking for a one-line solution. Well, how about

ssh address-of-B 'wget -O - http://server-C/whatever' >> whatever

i.e. redirection the wget-fetched output to stdout, and redirecting the local output (from ssh running wget remotely) to a file.

This seems to work, the wget output is just a little confusing ("saved to -"), you can get rid of it by adding -q to the wget call.

5
  • 6
    Apparently the SOCKS_SERVER environment variable has no effect on Ubuntu 14.04.
    – kasperd
    Jun 4, 2015 at 15:13
  • 6
    According to this other answer, wget has no support for SOCKS_SERVER variable. That answer suggests using curl --socks5-hostname or curl --socks5. Jun 18, 2015 at 13:30
  • @kasperd, so, is it just ubuntu where it doesn't work, or is the variable not related to wget at all? it appears that the variable might be for socksify instead.
    – cnst
    Sep 5, 2015 at 4:23
  • @cnst I have seen no evidence that any version of wget has socks support. socksify is not a setting, it is a separate program.
    – kasperd
    Sep 5, 2015 at 6:25
  • so, basically, if you want SOCKS_SERVER to have any effect on wget, then you have to install ports.su/security/dante, and then run socksify wget instead of just wget
    – cnst
    Sep 5, 2015 at 7:38
3

Another approach could be that you normally log in into B, where you start a screen session. There you do the wget of your files - all into one directory.

And there the program can happily run; you just detach from screen, but let it run in the background.

If the downloads are finished (maybe even earlier), you can fech the data from B to A using rsync (my preference).

2

Inspired by another answer to another question, I suggest using proxychains-ng (which is the newer version of proxychains).

  1. Download, compile, and optionally install proxychains-ng.
  2. Create a proxychains.conf file in the current directory, or at ~/.proxychains/proxychains.conf, or at /etc/proxychains.conf.
    • Alternatively, create one file anywhere else, or with another name, and specify if through -f command-line argument, or through PROXYCHAINS_CONF_FILE environment variable.
    • There is a sample config file available. The most relevant options are at the very end.
  3. In your proxychains.conf file, add:

    [ProxyList]
    socks5 127.0.0.1 1234
    
  4. Run ssh -D 1234 your_host_b. This will make ssh listen on port 1234 on localhost, and use your remote host as a SOCKS proxy.

    • Alternatively, run ssh -ND 1234 your_host_b instead. -N will prevent ssh from running any command on the remote server (i.e. it won't open a shell).
  5. Run: proxychains4 yourcommandhere yourparametershere. See some examples:
    • proxychains4 wget -O - http://ifconfig.co/
    • proxychains4 -q links http://ifconfig.co/
2

Option 0:

In order to use wget with a SOCKS5 proxy from ssh, you have to install the security/dante package in order to use the SOCKS_SERVER option with the socksify utility.

sudo pkg_add dante

Subsequently, you open an SSH connection in the background:

ssh -N -C -D1080 user@hostB &

And use wget through a SOCKS5 proxy through socksify:

env SOCKS_SERVER=127.0.0.1:1080 socksify wget http://website-C

Option 1:

Just pipe the file to stdout on the server, and read it from stdin on your workstation.

ssh -C user@hostB "wget -O- http://website-C" >> file-from-website-C
1

You can do a ssh tunnel from box A to box B and add to the routing table in box A, that website C is reachable via tunnel to box B. You have to allow packet forwarding on the box B.

Here you can see a very good step-by-step tutorial...

1

You would need to create a tunnel on machine B tha would redirect the call to website C. But I'm puzzled as why this would be faster, unless your ISP as some restrictions.

I don't know a oneliner, but this isn't much more complicated.

On machine A, you do (I took 11111 randomly, you can take whatever you want as long as it is > 1024, or you would need to be root)

ssh -f -C -N -L 11111:C:80 username@B

The username on B is the one you use to connect to B. This should create a tunnel on port 11111 on machine B that redirect to port 80 (web site in HTTP use 443 for HTTPS) on machine C (I hope I did not mess the order ;) )

Then you can download the file directly from machine A via machine B. I'm assuming the file is at http://C/path/to/file so you would then use:

wget http://B:11111/path/to/file
2
  • Wouldn't this work only if the file is accessible without HTTP 1.1 name-based virtual hosting? Since the GET request issued by wget would state the hostname as B, not C.
    – user
    May 16, 2012 at 11:39
  • @MichaelKjörling I don't have enough knowledge to answer you.
    – Huygens
    May 21, 2012 at 7:36
1

You can do this via port forwarding (ssh tunneling). Here's a resource: http://www.jfranken.de/homepages/johannes/vortraege/ssh2_inhalt.en.html#ToC9

Essentially, you should set up port forwarding on B. When A issues wget to B, B will forward the packets to C and send the results back to A.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .