I'm downloading a large amount of data from an SFTP server: multiple folders, multiple files. The graph of everything I'm downloading looks like a forest full of branches.
The current idea for this project is to:
- FTP all files to the local environment
- Decompress all files once they're all in the local environment
- Run data integrity checks on each file
- Push all decompressed data that pases step three to a database system with automatically provisioned storage
I am storing all the data locally on an EBS volume attached to an EC2 instance in AWS. After beginning step one above, I come back to my instance to find the EBS volume full. So, I double the size of the hard drive, restart step one--and find the volume is full again several hours later.
Having written out the procedure above, I can see value in developing some scripts on my end that iteratively process multiple subsets of the data so as to avoid having a very large footprint at one time. But, I was hoping to be lazy and just provision one, massive EBS volume for a few hours, do the work all in one pass, and save myself several hours of dev time.
Given that I only have SFTP access to the data ( no ssh; no other protocols ) is there an FTP command that will tell me the footprint of all of these files on disk? I'd like to use that information to know a good lower bound for what size EBS volume I'll provision.