6

How can I extract from a certain number of files all the URIs like magnet: file:? I'd like to have one URI per line as the output. I'd like to use bash to achieve this, but any suggestion is welcome. I'm using debian 7.

The input is a bunch of html files with random names. The URIs will all be in a href HTML tags like: <a href='magnet:?xt=something1:something1:somerandomcharacters'>link text</a>. There isn't a scheme of how the URIs are stored in each file, there may be n URIs per file and they may be wherever in the files.

Expected output should be a list with values separated by new line character like:

magnet:?xt.......
magnet:?xt.......
magnet:?xt.......
3
  • Can you give inputs and exptected output?
    – cuonglm
    Jul 13, 2014 at 17:31
  • Nb. The OP has acknowledged the URIs may be enclosed in ' >_<
    – goldilocks
    Jul 13, 2014 at 17:54
  • 1
    Use the right tool: perl-uri-find...
    – jasonwryan
    Jul 13, 2014 at 18:24

3 Answers 3

Reset to default
5

I suppose the URI ends with a space: grep -o 'magnet://[^ ]*' filename

Update: grep -o "magnet:?xt[^']*" filename

Update: cat * | grep -o "magnet:?xt[^']*" or grep -oh "magnet:?xt[^']*" *

7
  • I updated my question with some important details I did not mention before.
    – giovi321
    Jul 13, 2014 at 17:44
  • Anyway, they all end with a '.
    – giovi321
    Jul 13, 2014 at 17:45
  • +1 WRT the ending, you could either tailor this to the task, or use a set of allowed characters. Unfortunately, by the RFC, ' is reserved, meaning it is valid in URIs -- so you'll need to go the "tailor to the specific situation" route, as a generic solution would have to include the final ', which in this case you don't want.
    – goldilocks
    Jul 13, 2014 at 17:52
  • So if I use grep -o "magnet:?xt[^']*" filename i will obtain the uri with the ' inclueded? I could use a simple find and replace script to get rid of the ' then, right?
    – giovi321
    Jul 13, 2014 at 19:05
  • This is working great. I used * instead of filename, the only problem is that the output is always like filename:magnet:?xt...... and I don't know how to get rid of the filename:. Any suggestions?
    – giovi321
    Jul 13, 2014 at 19:12
3

With GNU grep, you can try:

$ echo "<a href='magnet:?xt=something1:something1:somerandomcharacters'>link text</a>" | 
grep -oP "magnet.*(?=')"
magnet:?xt=something1:something1:somerandomcharacters
0

You can have both single and double quotes as delimiter and last week I've seen someone use no delimiter at all, which is frustrating if you try to use the character before the protocol part as end indicator. Also you can have relative paths and they are valid links too. And you can have a href-attribute in an a-tag or a src-attribute in an img-tag, and so on.

The correct way would be to parse the whole document as html/xml and then traverse all elements, extract all attributes from them and match the value to your pattern. If you care about relative paths, you have to expand those too with the URL you used to download. PHP or Perl should have functions for this DOM tree parsing. Every other solution with grep or similar will probably skip some of the URLs or truncate too early.

Maybe curl or wget have some cool feature to create that list, most probably wget in spider mode.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .