Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have an HTML file with javascript and CSS in the source. Listed in the JS is a series of URLs' embedded with other meta-data. I want to use awk to extract the URLs (all enclosed in double quotes with the http:// prefix) and dump the urls to stdout. But I do not know how to use awk, but it seems to be the tool to use.

{
title: "Dsssat",
artist: "cxpl djij awsoj e",
mp3: "http://somesite.com/seal/dsssat.mp3",
},
share|improve this question

2 Answers 2

up vote 4 down vote accepted

Why use awk? sed is better at this:

sed -ne 's/.*\(http[^"]*\).*/\1/p' < foo.js
share|improve this answer
    
awk = typically used as a data extraction sed = stream editor, which an apply transformation –  Ronaldo Nascimento Nov 25 '12 at 22:46
    
Now, how do I use curl to grab each one? –  Ronaldo Nascimento Nov 25 '12 at 22:48
    
I'd say perl is more commonly used for data extraction, awk is getting increasingly rare :) –  Dennis Kaarsemaker Nov 25 '12 at 22:48
1  
sed -ne 's/.*(http[^"]*).*/\1/p' < foo.js | xargs curl –  Dennis Kaarsemaker Nov 25 '12 at 22:49
1  
@RonaldoNascimento Note that this only extracts the last URL of each line. This may or may not matter depending on your file format. –  Gilles Nov 25 '12 at 22:55

You can use grep. To include the double quotes:

grep -o '"http://[^"]*"' myfile.html

To exclude the double quotes:

grep -o 'http://[^"]*' myfile.html

Edit

You may want to do some further filtering to ensure that you only match the URLs in the JavaScript objects:

grep -o 'mp3: "http://[^"]*"' myfile.html | grep -o '"http://[^"]*"'

grep -o 'mp3: "http://[^"]*"' myfile.html | grep -o 'http://[^"]*'
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.