How can I parallelise the upload of a directory by FTP?

Question

I need to upload a directory with a rather complicated tree (lots of subdirectories, etc.) by FTP. I am unable to compress this directory, since I do not have any access to the destination apart from FTP - e.g. no tar. Since this is over a very long distance (USA => Australia), latency is quite high.

Following the advice in How to FTP multiple folders to another server using mput in Unix?, I am currently using ncftp to perform the transfer with mput -r. Unfortunately, this seems to transfer a single file at a time, wasting a lot of the available bandwidth on communication overhead.

Is there any way I can parallelise this process, i.e. upload multiple files from this directory at the same time? Of course, I could manually split it and execute mput -r on each chunk, but that's a tedious process.

A CLI method is heavily preferred, as the client machine is actually a headless server accessed via SSH.

Answer 1

lftp would do this with the command mirror -R -P 20 localpath - mirror syncs between locations, and -R uses the remote server as the destination , with P doing 20 parallel transfers at once.

As explained in man lftp:

   mirror [OPTS] [source [target]]

   Mirror specified source directory to local target directory. If  target
   directory ends with a slash, the source base name is appended to target
   directory name. Source and/or target can be URLs pointing  to  directo‐
   ries.


        -R,    --reverse                 reverse mirror (put files)
        -P,    --parallel[=N]            download N files in parallel

Answer 2

You could try using gnu parallel and curl to automate it.

then you could do something such as:

find . -t f -name "*.pdf" | parallel -j 4 curl -T {} ftp://ftp.site.com --user me:pass

This will run 4 jobs per cpu uploading all pdfs in working path.