3

I'm trying to download a series of podcast episodes using cURL. I know how to use a range of numbers to download several files matching a pattern, but what I can't figure out is how to have multiple ranges within a single URL.

Example URLs:

http://www.example.com/01_episode_1.mp3
http://www.example.com/02_episode_2.mp3
http://www.example.com/03_episode_3.mp3
...
http://www.example.com/52_episode_52.mp3

If I use

curl -O http://www.example.com/[01-52]_episode_[1-52].mp3

I get files like this:

01_episode_1.mp3
01_episode_2.mp3
01_episode_3.mp3
...
01_episode_52.mp3
02_episode_1.mp3
02_episode_2.mp3
02_episode_3.mp3
...
02_episode_52.mp3
03_episode_1.mp3
03_episode_2.mp3
03_episode_3.mp3
...
03_episode_52.mp3
etc...

But most of them are obviously not going to be valid files.

How can I use cURL to grab the files I want? Or maybe I'm trying to use the wrong tool for the job?

This is on OS X 10.10.3 and whatever version of cURL is installed by default on that platform.

2 Answers 2

5

While a weaker solution to this particular case, it's fairly useful in similar issues: curl features range globbing. I stumbled upon this thread, so someone else might too.

curl http://www.example.com/[01-52]_episode_[1-52].mp3 -o "ep#1_#2.mp3"
find . type -f -empty -delete

Word of explanation:

  • Uses curl's range globbing feature
  • Weaker than a loop here, since there are two almost identical ranges which actually are meant to be identical (and with exception of leading zeroes in first - are).
  • Chose -o in favour of -O to use local name and show #1 #2 ... #n feature of saving the files while using range globbing in curl.
  • You could just choose 2nd number with -o "#2.mp3" but this would overwrite the file! DLing 01ep1.mp3 and 02ep1.mp3 saves just 1.mp3
  • Testing on example.com will yield peculiar results since that page always replies with 200 and same page, just so you know, that's how IANA set it up.
  • Feature docs: https://ec.haxx.se/cmdline-globbing.html
  • find call is here since I don't know the range maximum
  • This saves all the files: if the range is [1-10] and there are just pages 1 and 2, it will save empty files for 3-9 (both inclusive). Hence: find . -empty -delete which deletes empty files. Obviously, due to aforementioned issue with example.com website this won't work there.
3

How about:

for i in {1..52}; do curl -O `printf "http://www.example.com/%02d_episode_%d.mp3" $i $i`; done
1
  • Perfect! I figured a loop might be required by my command line knowledge just wasn't up to snuff. Thanks! Commented Jul 10, 2015 at 2:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .