1

I'm downloading files in parallel through lftp onto a remote server, and I use the commands:

lftp ftp:*url*

cd into directory

mirror --parallel=10

I'm downloading a total of 365 files separated into 12 directories. When I use this command it takes a few hours, which isn't a deal breaker but I figured the more files I download in parallel the faster it will go. Obviously there has to be a point of diminishing returns, as I expect downloading all 365 files in parallel would overload the system. (Each file is little over a gigabyte). Plus, as I have increased the number of files to download in parallel, i increasingly get the message waiting to reconnect....trying in 30s etc. as I believe I am overtaxing the system.

Does anyone have thoughts on efficient ways to download files in parallel? Thanks in advance

1
  • Downloading N files at the same time, means that N processes are sending/recieving packets AT THE SAME TIME. Whichever part of the network between local and remote will be overloaded and slow down. – waltinator Dec 7 '20 at 19:03
0

The more you run in parallel, the more load you put on the sender.

If the server has 1000 TCP connections and the bandwidth is limited, then you will get 0.1% of the server's bandwidth if you have 1 connection.

If you have 10 connections, you will get 1% of the server's bandwidth.

In general my recommendations would be:

  • If you are not in a hurry: use a single connection
  • If you control the server, do whatever you like
  • If you are in a hurry and you do not control the server, use caution: Some will have no problems accepting 1000 parallel requests. Others will struggle at 100. And if you are adding 10 extra parallel requests, that may strain the server.

In your situation it sounds as if you are in the first situation: You are not in a hurry, so you should use a single connection. The "Connection refused" is also a sign that the server is overloaded. If you want to use more than 1 connection, consider talking to whomever owns the server.

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.