I need to download a large file (1GB). I also have access to multiple computers running Linux, but each is limited to a 50kB/s download speed by an admin policy.

How do I distribute downloading this file on several computers and merge them after all segments have been downloaded, so that I can receive it faster?

  • 7
    Download it at home, and sneakernet it in via usb thumbdrive? – WernerCD Aug 31 '14 at 16:47
  • 2
    I remember stuff like this with old sun b&w station when I was at the university. Just check if you have enough space to save all the contents, one of my friend has been expelled after blocking all computers of the lab (full tmp). – Kartoch Sep 1 '14 at 18:58
  • If there is no download restriction, how fast can the other end send the file? Are there transfer restrictions between computers on the LAN? – Sun Sep 4 '14 at 3:10
  • @SunWKim No. There is no specific restriction there. – B Faley Sep 4 '14 at 4:24

The common protocols HTTP, FTP and SFTP support range requests, so you can request part of a file. Note that this also requires server support, so it might or might not work in practice.

You can use curl and the -r or --range option to specify the range and eventually just catting the files together. Example:

curl -r 0-104857600         -o distro1.iso 'http://files.cdn/distro.iso'
curl -r 104857601-209715200 -o distro2.iso 'http://files.cdn/distro.iso'

And eventually when you gathered the individual parts you concatenate them:

cat distro* > distro.iso

You can get further information about the file, including its size with the --head option:

curl --head 'http://files.cdn/distro.iso'

You can retrieve the last chunk with an open range:

curl -r 604887601- -o distro9.iso 'http://files.cdn/distro.iso'

Read the curl man page for more options and explanations.

You can further leverage ssh and tmux to ease running and keeping track of the downloads on multiple servers.

  • 15
    Note: careful, when using cat distro* > ... check the sorting of the files as the * expanded by your shell could sort it like this: distro1.iso distro10.iso distro11.iso ... and thus concatenating in the wrong order. – Sebastian Sep 1 '14 at 7:55
  • 8
    a fix for @Sebastian's note would be: cat distro{1..10}.iso – nonchip Sep 1 '14 at 10:46
  • 1
    That solution is shell specific and not portable. cat $(seq -fdist%g.iso 1 10) should be more predictable, but it fails in csh, though. Replacing $(…) with backtics seems to work in most shells. – Marco Sep 1 '14 at 13:30
  • 3
    @Marco, seq is not a portable command either. You can use distro001.iso, distro002.iso... distroy010.iso – Stéphane Chazelas Sep 1 '14 at 15:45
  • Is the admin policy, 50 kB/s per transfer connection, or the total bandwidth allowed on the computer. If it is the former, the answer can be utilized on the same computer rather than having to log into different workstations. – Sun Sep 4 '14 at 16:45

It would take about 5.5 hours to rode load a 1 gigabyte file at 50 kilobytes per second.

It seems the effort to coordinate multiple computers to get partials may save some time.

You can look at bittorrent and utilize web seeding along with transfers via peer exchange. Each client can receive pieces and share completed pieces with in the local area network (LAN). You end up with the same 1gb file on each computer but the merging of pieces is automated for you.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.