Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need to upload a directory with a rather complicated tree (lots of subdirectories, etc.) by FTP. I am unable to compress this directory, since I do not have any access to the destination apart from FTP - e.g. no tar. Since this is over a very long distance (USA => Australia), latency is quite high.

Following the advice in How to FTP multiple folders to another server using mput in Unix?, I am currently using ncftp to perform the transfer with mput -r. Unfortunately, this seems to transfer a single file at a time, wasting a lot of the available bandwidth on communication overhead.

Is there any way I can parallelise this process, i.e. upload multiple files from this directory at the same time? Of course, I could manually split it and execute mput -r on each chunk, but that's a tedious process.

A CLI method is heavily preferred, as the client machine is actually a headless server accessed via SSH.

share|improve this question
    
Would rsync be an option? edit apparently not, as it doesn't work over ftp. Might be worth asking the destination server admin whether he's willing to give you sftp access - you wouldn't be able to log in, but rsync would work. –  Shadur Jan 20 at 10:57

2 Answers 2

up vote 5 down vote accepted

lftp would do this with the command mirror -R -P 20 localpath - mirror syncs between locations, and -R uses the remote server as the destination , with P doing 20 parallel transfers at once.

As explained in man lftp:

   mirror [OPTS] [source [target]]

   Mirror specified source directory to local target directory. If  target
   directory ends with a slash, the source base name is appended to target
   directory name. Source and/or target can be URLs pointing  to  directo‐
   ries.


        -R,    --reverse                 reverse mirror (put files)
        -P,    --parallel[=N]            download N files in parallel
share|improve this answer

You could try using gnu parallel and curl to automate it.

then you could do something such as:

find . -t f -name "*.pdf" | parallel -j 4 curl -T {} ftp://ftp.site.com --user me:pass

This will run 4 jobs per cpu uploading all pdfs in working path.

share|improve this answer
    
The is one possible approach, but has the drawback of needing to authenticate once for every file - again, a lot of overhead made worse by high latency. –  Bob Jan 22 at 7:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.