I need to transfer a very large folder over FTP, without creating a file first

Question

I am running an Amazon EC2 instance that is limited in disk space. I have a very large folder that I need to FTP to my local machine, preferably without creating an intermediate tar file, since there might not be enough disk space.

Can I use a pipe for this? If so, can someone help me with the command?

Note While FTP isn't the best way of transferring the files, is my only option in this case.

Answer 1

Use just about any FTP client other than the basic ftp program.

• Store your password in ~/.netrc and use wget:

  wget -r ftp://username@example.com/path/to/directory
  

  Keep in mind that the -r option has a default maximum depth of 5. You can use -l to increase it.

• Use lftp:

  open username@example.com
  mirror /path/to/directory
  

• Use ncftp:

  open -u username example.com
  get -R /path/to/directory
  

  Or its ncftpget command:

  ncftpget ftp://username@example.com//path/to/directory
  

Alternatively, mount the FTP server as a directory and access it like you would any other directory. There are several FUSE filesystems for FTP.

• AVFS:

  mountavfs
  cat file_containing_your_password >~/.avfs/#ftp_ctl:username@example.com/password
  cp -Rp ~/.avfs/#ftp:username@example.com/path/to/directory .
  

• CURLFTPFS: store your password in ~/.netrc, then

  mkdir ~/example.com
  curlftpfs -o user=username example.com ~/example.com
  cp -Rp example.com/path/to/directory .
  

Answer 2

For this kind of transfer FTP may not be your best choice. Using rsync or scp may work better for this. Try something like:

scp -r user@host:folder  .

If you need to fetch updates from the server, then rsync should work better.

EDIT: If you are stuck with FTP, then the use a recursive wget command, or get an FTP client which supports recursive gets.