I'm attempting to download a year's worth of data from an NOAA FTP Server using wget (or ncftpget). However, it takes way longer than it should due to FTP's overhead (I think). For instance, this command
time wget -nv -m ftp://ftp:firstname.lastname@example.org/pub/data/noaa/2015 -O /weather/noaa/2015
Or similarly, via ncftpget
ncftpget -R -T -v ftp.ncdc.noaa.gov /weather/noaa/ /pub/data/noaa/2015
Yields a result of. 53 minutes to transfer 30M!
FINISHED --2015-01-03 16:21:41-- Total wall clock time: 53m 32s Downloaded: 12615 files, 30M in 7m 8s (72.6 KB/s) real 53m32.447s user 0m2.858s sys 0m8.744s
When I watch this transfer, each individual file transfers quite quickly (500kb/sec) but the process of downloading 12,000 relatively small files incurs an enormous amount of overhead and slows the entire process down.
- Am I assessing the situation correctly? I realize it's hard to tell without knowing the servers but does FTP really suck this much when transferring tons of small files?
- Are there any tweaks to wget or ncftpget to enable them to play nicer with the remote FTP server? Or perhaps some kind of parallelism?