Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm going to wget youtube search page where the URL is horrible. This is example when searching searchkeyword sorted by upload date:

wget http://www.youtube.com/results?search_type=videos&search_query=searchkeyword&search_sort=video_date_uploaded&suggested_categories=26%2C27%2C22%2C28%2C24

and this one is page 2 of the search result:

wget http://www.youtube.com/results?search_type=videos&search_query=searchkeyword&search_sort=video_date_uploaded&suggested_categories=26%2C27%2C22%2C28%2C24&page=2

The wget give me misleading page for both pages.

share|improve this question
add comment

3 Answers

up vote 9 down vote accepted

Wrapping the URL in single quotation marks should do the trick. Also, you will want to give a name to the downloaded file as well (otherwise it will have the weird name that wget guessed from the URL). So

wget 'http://www.youtube.com/results?search_type=videos&search_query=searchkeyword&search_sort=video_date_uploaded&suggested_categories=26%2C27%2C22%2C28%2C24' -O nice_name.html

Explanation as per laebshade's comment (thanks for reminding me): The many funny characters in the URL will be interpreted by the shell as special commands or other control sequences. From the top of my head, ~ ! # ^ & * will all mean something else to the shell (depends on what shell you are using). You can use \ to escape them one by one, or quote the whole thing.

share|improve this answer
4  
It would be good to explain the reason for using quotes. The reason it's not giving you the right URL is because the URL contains an '&' (ampersand) character, which your shell interprets as a fork() to background; in other words, your shell sends the first part of the command wget http://youtube.com/results?search_type=videos& to the background, then treats the next part of the "command" as a new one, until it reaches another &, and so on and so forth. –  laebshade Jul 17 '11 at 14:38
    
@laebshade Very helpful explanation. :) –  TheIndependentAquarius Aug 29 '11 at 10:01
add comment

Use youtube-dl for downloading Youtube videos. It's a epic python script. It req's Python. If you have Ubuntu, install with this command: sudo apt-get install youtube-dl otherwise you can directly download it from the website: http://rg3.github.com/youtube-dl/

share|improve this answer
add comment

Don't use wget for downloading individual pages. It is better suited for things like recursive downloads of whole sites. Try curl instead.

Also, make sure you quote your arguments on the command line so that you don't run into shell globing problems with characters like ?.

curl 'http://www.youtube.com/results?search_type=videos&search_query=searchkeyword&search_sort=video_date_uploaded&suggested_categories=26%2C27%2C22%2C28%2C24&page=2' \
    > results.html
share|improve this answer
    
Instead of > results.html you can also use curl http:://url.com/randompage -o results.html –  bollovan Jul 17 '11 at 11:49
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.