13

I need to upload a directory with a rather complicated tree (lots of subdirectories, etc.) by FTP. I am unable to compress this directory, since I do not have any access to the destination apart from FTP - e.g. no tar. Since this is over a very long distance (USA => Australia), latency is quite high.

Following the advice in How to FTP multiple folders to another server using mput in Unix?, I am currently using ncftp to perform the transfer with mput -r. Unfortunately, this seems to transfer a single file at a time, wasting a lot of the available bandwidth on communication overhead.

Is there any way I can parallelise this process, i.e. upload multiple files from this directory at the same time? Of course, I could manually split it and execute mput -r on each chunk, but that's a tedious process.

A CLI method is heavily preferred, as the client machine is actually a headless server accessed via SSH.

1
  • Would rsync be an option? edit apparently not, as it doesn't work over ftp. Might be worth asking the destination server admin whether he's willing to give you sftp access - you wouldn't be able to log in, but rsync would work. Commented Jan 20, 2014 at 10:57

2 Answers 2

18

lftp would do this with the command mirror -R -P 20 localpath - mirror syncs between locations, and -R uses the remote server as the destination , with P doing 20 parallel transfers at once.

As explained in man lftp:

   mirror [OPTS] [source [target]]

   Mirror specified source directory to local target directory. If  target
   directory ends with a slash, the source base name is appended to target
   directory name. Source and/or target can be URLs pointing  to  directo‐
   ries.


        -R,    --reverse                 reverse mirror (put files)
        -P,    --parallel[=N]            download N files in parallel
3

You could try using gnu parallel and curl to automate it.

then you could do something such as:

find . -t f -name "*.pdf" | parallel -j 4 curl -T {} ftp://ftp.site.com --user me:pass

This will run 4 jobs per cpu uploading all pdfs in working path.

1
  • The is one possible approach, but has the drawback of needing to authenticate once for every file - again, a lot of overhead made worse by high latency.
    – Bob
    Commented Jan 22, 2014 at 7:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .