2

I'm downloading a large amount of data from an SFTP server: multiple folders, multiple files. The graph of everything I'm downloading looks like a forest full of branches.

The current idea for this project is to:

  1. FTP all files to the local environment
  2. Decompress all files once they're all in the local environment
  3. Run data integrity checks on each file
  4. Push all decompressed data that pases step three to a database system with automatically provisioned storage

I am storing all the data locally on an EBS volume attached to an EC2 instance in AWS. After beginning step one above, I come back to my instance to find the EBS volume full. So, I double the size of the hard drive, restart step one--and find the volume is full again several hours later.

Having written out the procedure above, I can see value in developing some scripts on my end that iteratively process multiple subsets of the data so as to avoid having a very large footprint at one time. But, I was hoping to be lazy and just provision one, massive EBS volume for a few hours, do the work all in one pass, and save myself several hours of dev time.

Given that I only have SFTP access to the data ( no ssh; no other protocols ) is there an FTP command that will tell me the footprint of all of these files on disk? I'd like to use that information to know a good lower bound for what size EBS volume I'll provision.

1 Answer 1

2

lftp has a built-in du command which works over sftp (or any protocol lftp supports, presumably) by recursively listing everything and summing it up.

$ lftp sftp://user@hostname/
lftp HOST:~> du -s
… wait a while …
1656724 .

lftp is packaged in Debian, Ubuntu, and several RPM-based distros.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.