8

I'm attempting to download a year's worth of data from an NOAA FTP Server using wget (or ncftpget). However, it takes way longer than it should due to FTP's overhead (I think). For instance, this command

time wget -nv -m ftp://ftp:email@ftp.ncdc.noaa.gov/pub/data/noaa/2015 -O /weather/noaa/2015

Or similarly, via ncftpget

ncftpget -R -T -v ftp.ncdc.noaa.gov /weather/noaa/ /pub/data/noaa/2015

Yields a result of. 53 minutes to transfer 30M!

FINISHED --2015-01-03 16:21:41--
Total wall clock time: 53m 32s
Downloaded: 12615 files, 30M in 7m 8s (72.6 KB/s)

real    53m32.447s
user    0m2.858s
sys 0m8.744s

When I watch this transfer, each individual file transfers quite quickly (500kb/sec) but the process of downloading 12,000 relatively small files incurs an enormous amount of overhead and slows the entire process down.

My Questions:

  1. Am I assessing the situation correctly? I realize it's hard to tell without knowing the servers but does FTP really suck this much when transferring tons of small files?
  2. Are there any tweaks to wget or ncftpget to enable them to play nicer with the remote FTP server? Or perhaps some kind of parallelism?
6

Here's how I ended up doing solving this using the advice from others. The NOAA in this case has an FTP and an HTTP resource for this, so what I wrote a script that does the following:

  1. ncftpls to get a list of files
  2. sed to complete the filepaths to a full list of http files
  3. aria2c to quickly download them all

Example script:

# generate file list
ncftpls ftp://path/to/ftp/resources > /tmp/remote_files.txt

# append the full path, use http
sed -i -e 's/^/http:\/\/www1\.website\.gov\/pub\/data\//' /tmp/remote_files.txt

# download using aria2c
aria2c -i /tmp/remote_files.txt -d /filestore/2015

This runs much faster and is probably kinder to the NOAA's servers. There's probably even a clever way to get rid of that middle step, but I haven't found it yet.

|improve this answer|||||
  • Excellent solution, works out of the box! Thanks. – markusN Jan 28 '16 at 9:53
2
  1. Your assessment is correct. From a pure numbers standpoint, the overhead is slowing down the download
  2. Use aria2c. Aria2c opens many parallel connections to the ftp server to download a set of files faster. Just be sure that your server supports multiple concurrent connections from the same host

Alternatively, if you have write access on your host (in this case, I would assume you are logging in anonymously and do not have write access) zip/tar the files before download

|improve this answer|||||
  • If you can compress as well this may make it even faster. – ctrl-alt-delor Jan 3 '15 at 23:32
  • Can aria2c support recursive FTP? I looked at the documentation and couldn't find a mention of this, like I want to download an entire folder... – Tom Hayden Jan 4 '15 at 0:28
  • It doesn't support it, however, using standard ftp to enumerate files, then passing that list to aria will still be faster than recursive standard ftp – Outurnate Jan 4 '15 at 1:42
  • @TomHayden - I'm not aware that you can exactly, but certainly you can list a folder and feed it the results. Also there is the RPC interface with which you can do practically anything. 742: good answer - came here to say this but you beat me to it. – mikeserv Jan 4 '15 at 1:43
  • @mikeserv thanks. Props to finding the RPC – Outurnate Jan 4 '15 at 1:57
1

Yes, you are correct. Unfortunately while the ftp specs offer an alternate transfer mode that is capable of piplining multiple files over a single data connection, the last time I did a survey of available software ( which admittedly was back in 1998 ), none supported it. I don't think the situation has changed because interest in ftp has been rather low. Most organizations with such data sets provide access to them via http, or in larger archives to avoid this problem. You should check if NOAA does this.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.