2

Site contents

<html>
  <body>
   <a href="http://www.sagar.com/" title="sagar">Sagar</a>
   <a href="http://www.sagarcom/1/" title="sagar1">Sagar Link1</a>
   <a href="http://www.sagar.com/2/song.mp3">Download this</a>
  </body>
</html>

link_source_file.txt contains url for the above content

## Below wget command
wget -r -l1 -H -t1 -nd -N -np -A.mp3 -erobots=off -i link_source_file.txt

I want to download a specific type of file as mentioned by -A. Here it is mp3. The above wget command it will go till one depth of the link as per -t1.

It follows the first link http://www.sagar.com/ and downloads the index.html file. Then figures out that doesn't have any file type as .mp3 then it removes index.html

I want wget to follow only links which have a extension of *.mp3 and not any other links. How can I tweak the command for this requirement?

Improve this question
3
  • wget downloads song.mp3 for me with the command you gave. Isn't that what you wanted? pavuk may be a suitable alternative to wget if you want to do more advanced matching on URLs.
    – frostschutz
    Commented Apr 6, 2013 at 12:02
  • @frostschutz Yes, it downloads *.mp3 from the site. But i want to exclude any index file downloading, trying to use it optimally with minimum downloads.
    – Samar Panda
    Commented Apr 6, 2013 at 16:02
  • Wget will know the file is not mp3 after fetching the file headers at the earliest. The fact the link says .../index.html doesn't mean the server will return "index.html" - it may normally return 404 or a redirect, but it may be as well configured to return whatever, no reason not to reply with song.mp3 if such is the server admin's fancy. If you know for a fact a every, and only link with "mp3" in file extension will return an mp3 file, filter it yourself before passing that to wget.
    – SF.
    Commented Aug 10, 2013 at 13:13

2 Answers 2

Reset to default
1

I don't think you can. After all, using the -r recursive download option basically asks wget to look for links (in -l levels), and links are only found in html files. So wget thus needs to download and parse the html files, looking for links to both more html files and mp3 files.

If you already created a list of links, why don't you just use fgrep ".mp3" to create a new list with only mp3 files, and use this as link_source_file for wget? As wget won't download any html files, it won't find more links, so you can drop the -r -l and probably a few other options too - just keep -i (and -A, though the list already only contains just mp3 files). Use the -x option to force the creation of a file hierarchy, even without -r.

In short, if you want to traverse a site recursively, you must allow wget to download and parse html files, even if it doesn't keep them.

An alternative would be to use wget to download just a few levels of html files. Then harvest the links from these files manually and create a list with all the mp3 files you found (lynx, sed and fgrep would be of great help for this). Or if you know something about the organization of the site, generate a list of the files you want in some other way (e.g. using a shellscript to create a list with lines like:

http://www.sagar.com/1/song.mp3
http://www.sagar.com/2/song.mp3
http://www.sagar.com/3/song.mp3
http://www.sagar.com/100/song.mp3
Improve this answer
0

The manual approach with grep.

Start in an empty directory and download all the HTML-files in your list non-recursively. Add the --convert-links options so any relative links to MP3 files will be converted to full ones. So you have only files like the one you posted above with one or more links to MP3 files.

Then you can do:

(for f in *
 do
     grep -P --only-matching '(?<=href=")[^ "]*\.mp3' foo.html "$f"
     # rm "$f"
 done) | xargs wget

Leave out the # before the rm if you want to delete the downloaded non-mp3 files.

This may still fail if e.g. a host refuses to download when the referrer is missing or wrong.

Improve this answer
4
  • The rm will remove the input files even for an error, like when the connection fails. Not good. It will also delete any other files in the directory, including mp3 files. I'll edit to comment out the rm. (it could be a mv to a subdir, or using a && between grep and rm at least.)
    – Volker Siegel
    Commented Aug 20, 2014 at 17:58
  • @VolkerSiegel You're absolutely right about the lack of error handling, it wasn't the focus of the answer. As for the rm, I did write you start with an empty directory, so there are only HTML files in there when you run the command - if it fails, you redownload them.
    – frostschutz
    Commented Aug 20, 2014 at 18:06
  • Of course, it works mostly fine if following the full description - but people are sometimes coming here because they do not know much about shell scripts etc. What does not work, even with full description, is: When there is an error for some of the input files, they will be deleted, but there are some resulting mp3 files. It's easy to miss that not all were downloaded then, and not possible to find out which failed, if the scrollback buffer or the terminal is not long enough.
    – Volker Siegel
    Commented Aug 20, 2014 at 18:12
  • Awfully high expectations for a quick&dirty solution :)
    – frostschutz
    Commented Aug 20, 2014 at 18:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .