Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Site contents

<html>
  <body>
   <a href="http://www.sagar.com/" title="sagar">Sagar</a>
   <a href="http://www.sagarcom/1/" title="sagar1">Sagar Link1</a>
   <a href="http://www.sagar.com/2/song.mp3">Download this</a>
  </body>
</html>

link_source_file.txt contains url for the above content

## Below wget command
wget -r -l1 -H -t1 -nd -N -np -A.mp3 -erobots=off -i link_source_file.txt

I want to download a specific type of file as mentioned by -A. Here it is mp3. The above wget command it will go till one depth of the link as per -t1.

It follows the first link http://www.sagar.com/ and downloads the index.html file. Then figures out that doesn't have any file type as .mp3 then it removes index.html

I want wget to follow only links which have a extension of *.mp3 and not any other links. How can I tweak the command for this requirement?

share|improve this question
    
wget downloads song.mp3 for me with the command you gave. Isn't that what you wanted? pavuk may be a suitable alternative to wget if you want to do more advanced matching on URLs. –  frostschutz Apr 6 '13 at 12:02
    
@frostschutz Yes, it downloads *.mp3 from the site. But i want to exclude any index file downloading, trying to use it optimally with minimum downloads. –  Samar Panda Apr 6 '13 at 16:02
    
Wget will know the file is not mp3 after fetching the file headers at the earliest. The fact the link says .../index.html doesn't mean the server will return "index.html" - it may normally return 404 or a redirect, but it may be as well configured to return whatever, no reason not to reply with song.mp3 if such is the server admin's fancy. If you know for a fact a every, and only link with "mp3" in file extension will return an mp3 file, filter it yourself before passing that to wget. –  SF. Aug 10 '13 at 13:13

2 Answers 2

I don't think you can. After all, using the -r recursive download option basically asks wget to look for links (in -l levels), and links are only found in html files. So wget thus needs to download and parse the html files, looking for links to both more html files and mp3 files.

If you already created a list of links, why don't you just use fgrep ".mp3" to create a new list with only mp3 files, and use this as link_source_file for wget? As wget won't download any html files, it won't find more links, so you can drop the -r -l and probably a few other options too - just keep -i (and -A, though the list already only contains just mp3 files). Use the -x option to force the creation of a file hierarchy, even without -r.

In short, if you want to traverse a site recursively, you must allow wget to download and parse html files, even if it doesn't keep them.


An alternative would be to use wget to download just a few levels of html files. Then harvest the links from these files manually and create a list with all the mp3 files you found (lynx, sed and fgrep would be of great help for this). Or if you know something about the organization of the site, generate a list of the files you want in some other way (e.g. using a shellscript to create a list with lines like:

http://www.sagar.com/1/song.mp3
http://www.sagar.com/2/song.mp3
http://www.sagar.com/3/song.mp3
http://www.sagar.com/100/song.mp3
share|improve this answer

The manual approach with grep.

Start in an empty directory and download all the HTML-files in your list non-recursively. Add the --convert-links options so any relative links to MP3 files will be converted to full ones. So you have only files like the one you posted above with one or more links to MP3 files.

Then you can do:

(for f in *
 do
     grep -P --only-matching '(?<=href=")[^ "]*\.mp3' foo.html "$f"
     # rm "$f"
 done) | xargs wget

Leave out the # before the rm if you want to delete the downloaded non-mp3 files.

This may still fail if e.g. a host refuses to download when the referrer is missing or wrong.

share|improve this answer
    
The rm will remove the input files even for an error, like when the connection fails. Not good. It will also delete any other files in the directory, including mp3 files. I'll edit to comment out the rm. (it could be a mv to a subdir, or using a && between grep and rm at least.) –  Volker Siegel Aug 20 at 17:58
    
@VolkerSiegel You're absolutely right about the lack of error handling, it wasn't the focus of the answer. As for the rm, I did write you start with an empty directory, so there are only HTML files in there when you run the command - if it fails, you redownload them. –  frostschutz Aug 20 at 18:06
    
Of course, it works mostly fine if following the full description - but people are sometimes coming here because they do not know much about shell scripts etc. What does not work, even with full description, is: When there is an error for some of the input files, they will be deleted, but there are some resulting mp3 files. It's easy to miss that not all were downloaded then, and not possible to find out which failed, if the scrollback buffer or the terminal is not long enough. –  Volker Siegel Aug 20 at 18:12
    
Awfully high expectations for a quick&dirty solution :) –  frostschutz Aug 20 at 18:25

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.