wget has an ability to recursively download files, however it does it one file at a time.
I would like to pass in a directory URL, and for each URL it encounters in the recursion for it to spawn off a downloading process.
One way I was thinking to do this is to somehow use
wget to print out the URLs it encounters, and then feeding those URLs into separate instances of
wget URL_1 &,
wget URL_2 & etc).