5

I'm going to wget youtube search page where the URL is horrible. This is example when searching searchkeyword sorted by upload date:

wget http://www.youtube.com/results?search_type=videos&search_query=searchkeyword&search_sort=video_date_uploaded&suggested_categories=26%2C27%2C22%2C28%2C24

and this one is page 2 of the search result:

wget http://www.youtube.com/results?search_type=videos&search_query=searchkeyword&search_sort=video_date_uploaded&suggested_categories=26%2C27%2C22%2C28%2C24&page=2

The wget give me misleading page for both pages.

9

Wrapping the URL in single quotation marks should do the trick. Also, you will want to give a name to the downloaded file as well (otherwise it will have the weird name that wget guessed from the URL). So

wget 'http://www.youtube.com/results?search_type=videos&search_query=searchkeyword&search_sort=video_date_uploaded&suggested_categories=26%2C27%2C22%2C28%2C24' -O nice_name.html

Explanation as per laebshade's comment (thanks for reminding me): The many funny characters in the URL will be interpreted by the shell as special commands or other control sequences. From the top of my head, ~ ! # ^ & * will all mean something else to the shell (depends on what shell you are using). You can use \ to escape them one by one, or quote the whole thing.

  • 4
    It would be good to explain the reason for using quotes. The reason it's not giving you the right URL is because the URL contains an '&' (ampersand) character, which your shell interprets as a fork() to background; in other words, your shell sends the first part of the command wget http://youtube.com/results?search_type=videos& to the background, then treats the next part of the "command" as a new one, until it reaches another &, and so on and so forth. – laebshade Jul 17 '11 at 14:38
  • @laebshade Very helpful explanation. :) – Aquarius_Girl Aug 29 '11 at 10:01
4

Don't use wget for downloading individual pages. It is better suited for things like recursive downloads of whole sites. Try curl instead.

Also, make sure you quote your arguments on the command line so that you don't run into shell globing problems with characters like ?.

curl 'http://www.youtube.com/results?search_type=videos&search_query=searchkeyword&search_sort=video_date_uploaded&suggested_categories=26%2C27%2C22%2C28%2C24&page=2' \
    > results.html
  • Instead of > results.html you can also use curl http:://url.com/randompage -o results.html – bollovan Jul 17 '11 at 11:49
  • Perfect: Curl obviously has different magic embedded. ^wget^curl made it work in my case (pulling a JSON file of a google spreadsheet.) Hat's off to you Good Sir! – Sherwood Botsford Jan 2 '17 at 18:19
3

Use youtube-dl for downloading Youtube videos. It's a epic python script. It req's Python. If you have Ubuntu, install with this command: sudo apt-get install youtube-dl otherwise you can directly download it from the website: http://rg3.github.com/youtube-dl/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.