Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

How can I extract from a certain number of files all the URIs like magnet: file:? I'd like to have one URI per line as the output. I'd like to use bash to achieve this, but any suggestion is welcome. I'm using debian 7.

The input is a bunch of html files with random names. The URIs will all be in a href HTML tags like: <a href='magnet:?xt=something1:something1:somerandomcharacters'>link text</a>. There isn't a scheme of how the URIs are stored in each file, there may be n URIs per file and they may be wherever in the files.

Expected output should be a list with values separated by new line character like:

magnet:?xt.......
magnet:?xt.......
magnet:?xt.......
share|improve this question
    
Can you give inputs and exptected output? –  Gnouc Jul 13 at 17:31
    
Nb. The OP has acknowledged the URIs may be enclosed in ' >_< –  goldilocks Jul 13 at 17:54
1  
Use the right tool: perl-uri-find... –  jasonwryan Jul 13 at 18:24

3 Answers 3

up vote 4 down vote accepted

I suppose the URI ends with a space: grep -o 'magnet://[^ ]*' filename

Update: grep -o "magnet:?xt[^']*" filename

Update: cat * | grep -o "magnet:?xt[^']*" or grep -oh "magnet:?xt[^']*" *

share|improve this answer
    
I updated my question with some important details I did not mention before. –  giovi321 Jul 13 at 17:44
    
Anyway, they all end with a '. –  giovi321 Jul 13 at 17:45
    
+1 WRT the ending, you could either tailor this to the task, or use a set of allowed characters. Unfortunately, by the RFC, ' is reserved, meaning it is valid in URIs -- so you'll need to go the "tailor to the specific situation" route, as a generic solution would have to include the final ', which in this case you don't want. –  goldilocks Jul 13 at 17:52
    
So if I use grep -o "magnet:?xt[^']*" filename i will obtain the uri with the ' inclueded? I could use a simple find and replace script to get rid of the ' then, right? –  giovi321 Jul 13 at 19:05
    
This is working great. I used * instead of filename, the only problem is that the output is always like filename:magnet:?xt...... and I don't know how to get rid of the filename:. Any suggestions? –  giovi321 Jul 13 at 19:12

With GNU grep, you can try:

$ echo "<a href='magnet:?xt=something1:something1:somerandomcharacters'>link text</a>" | 
grep -oP "magnet.*(?=')"
magnet:?xt=something1:something1:somerandomcharacters
share|improve this answer

You can have both single and double quotes as delimiter and last week I've seen someone use no delimiter at all, which is frustrating if you try to use the character before the protocol part as end indicator. Also you can have relative paths and they are valid links too. And you can have a href-attribute in an a-tag or a src-attribute in an img-tag, and so on.

The correct way would be to parse the whole document as html/xml and then traverse all elements, extract all attributes from them and match the value to your pattern. If you care about relative paths, you have to expand those too with the URL you used to download. PHP or Perl should have functions for this DOM tree parsing. Every other solution with grep or similar will probably skip some of the URLs or truncate too early.

Maybe curl or wget have some cool feature to create that list, most probably wget in spider mode.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.