1

I'm trying to download a series of podcast episodes using cURL. I know how to use a range of numbers to download several files matching a pattern, but what I can't figure out is how to have multiple ranges within a single URL.

Example URLs:

http://www.example.com/01_episode_1.mp3
http://www.example.com/02_episode_2.mp3
http://www.example.com/03_episode_3.mp3
...
http://www.example.com/52_episode_52.mp3

If I use

curl -O http://www.example.com/[01-52]_episode_[1-52].mp3

I get files like this:

01_episode_1.mp3
01_episode_2.mp3
01_episode_3.mp3
...
01_episode_52.mp3
02_episode_1.mp3
02_episode_2.mp3
02_episode_3.mp3
...
02_episode_52.mp3
03_episode_1.mp3
03_episode_2.mp3
03_episode_3.mp3
...
03_episode_52.mp3
etc...

But most of them are obviously not going to be valid files.

How can I use cURL to grab the files I want? Or maybe I'm trying to use the wrong tool for the job?

This is on OS X 10.10.3 and whatever version of cURL is installed by default on that platform.

1

How about:

for i in {1..52}; do curl -O `printf "http://www.example.com/%02d_episode_%d.mp3" $i $i`; done
  • Perfect! I figured a loop might be required by my command line knowledge just wasn't up to snuff. Thanks! – Patrick Wynne Jul 10 '15 at 2:23
1

While a weaker solution to this particular case, it's fairly useful in similar issues: curl features range globbing. I stumbled upon this thread, so someone else might too.

curl http://www.example.com/[01-52]_episode_[1-52].mp3 -o "ep#1_#2.mp3"
find . type -f -empty -delete

Word of explanation:

  • Uses curl's range globbing feature
  • Weaker than a loop here, since there are two almost identical ranges which actually are meant to be identical (and with exception of leading zeroes in first - are).
  • Chose -o in favour of -O to use local name and show #1 #2 ... #n feature of saving the files while using range globbing in curl.
  • You could just choose 2nd number with -o "#2.mp3" but this would overwrite the file! DLing 01ep1.mp3 and 02ep1.mp3 saves just 1.mp3
  • Testing on example.com will yield peculiar results since that page always replies with 200 and same page, just so you know, that's how IANA set it up.
  • Feature docs: https://ec.haxx.se/cmdline-globbing.html
  • find call is here since I don't know the range maximum
  • This saves all the files: if the range is [1-10] and there are just pages 1 and 2, it will save empty files for 3-9 (both inclusive). Hence: find . -empty -delete which deletes empty files. Obviously, due to aforementioned issue with example.com website this won't work there.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.