I have a script that's meant to download a certain number of files from a remote server. It only needs to do this every 24 hours, since they are JSON sources for a database on my server. The files are updated on the remote server around midnight GMT, and my script runs an hour after that to ensure they are properly updated already.
The problem is that I consistently notice it fails to download at least twenty or more out of the 132 files, except it doesn't think it's failed at all (I see 200 OK). They are JSONs, so they are at most 8KB in size. In the wget logfile, I see this:
--2013-09-21 12:01:10-- http://services.runescape.com/m=itemdb_rs/api/graph/19227.json Reusing existing connection to services.runescape.com:80. HTTP request sent, awaiting response... 200 OK Length: 0 [text/html] Saving to: `./jsons/19227.json' 0K 0.00 =0s 2013-09-21 12:01:10 (0.00 B/s) - `./jsons/19227.json' saved [0/0]
This doesn't make any sense. There's no rhyme or reason to the failures. I re-tried many times and each time it wrote 0-byte files at random, not failing on the same files each time. The frustrating part is there are no errors anywhere, so nothing gets caught in the error log...
no-clobber doesn't matter in this case. The files are meant to be overwritten, as they become out-of-date every 24 hours, and even "good data" from the day before is "bad data" today.
Is there anywhere I could improve my script to check filesize or whatever before downloading? I tried on my Mac at home and got the same exact result, even using "spider mode" to check if it exists first. The most frustrating part is if I were to paste the URL into a browser, it loads the whole JSON just as it should...I take it "retries" won't help as wget is not running into any HTTP errors anyway.