I need to upload a directory with a rather complicated tree (lots of subdirectories, etc.) by FTP. I am unable to compress this directory, since I do not have any access to the destination apart from FTP - e.g. no tar. Since this is over a very long distance (USA => Australia), latency is quite high.

Following the advice in How to FTP multiple folders to another server using mput in Unix?, I am currently using ncftp to perform the transfer with mput -r. Unfortunately, this seems to transfer a single file at a time, wasting a lot of the available bandwidth on communication overhead.

Is there any way I can parallelise this process, i.e. upload multiple files from this directory at the same time? Of course, I could manually split it and execute mput -r on each chunk, but that's a tedious process.

A CLI method is heavily preferred, as the client machine is actually a headless server accessed via SSH.

  • Would rsync be an option? edit apparently not, as it doesn't work over ftp. Might be worth asking the destination server admin whether he's willing to give you sftp access - you wouldn't be able to log in, but rsync would work. – Shadur Jan 20 '14 at 10:57

lftp would do this with the command mirror -R -P 20 localpath - mirror syncs between locations, and -R uses the remote server as the destination , with P doing 20 parallel transfers at once.

As explained in man lftp:

   mirror [OPTS] [source [target]]

   Mirror specified source directory to local target directory. If  target
   directory ends with a slash, the source base name is appended to target
   directory name. Source and/or target can be URLs pointing  to  directo‐

        -R,    --reverse                 reverse mirror (put files)
        -P,    --parallel[=N]            download N files in parallel

You could try using gnu parallel and curl to automate it.

then you could do something such as:

find . -t f -name "*.pdf" | parallel -j 4 curl -T {} ftp://ftp.site.com --user me:pass

This will run 4 jobs per cpu uploading all pdfs in working path.

  • The is one possible approach, but has the drawback of needing to authenticate once for every file - again, a lot of overhead made worse by high latency. – Bob Jan 22 '14 at 7:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.