I have an HTML file with javascript and CSS in the source. Listed in the JS is a series of URLs' embedded with other meta-data. I want to use awk to extract the URLs (all enclosed in double quotes with the http:// prefix) and dump the urls to stdout. But I do not know how to use awk, but it seems to be the tool to use.

title: "Dsssat",
artist: "cxpl djij awsoj e",
mp3: "http://somesite.com/seal/dsssat.mp3",

You can use grep. To include the double quotes:

grep -o '"http://[^"]*"' myfile.html

To exclude the double quotes:

grep -o 'http://[^"]*' myfile.html


You may want to do some further filtering to ensure that you only match the URLs in the JavaScript objects:

grep -o 'mp3: "http://[^"]*"' myfile.html | grep -o '"http://[^"]*"'

grep -o 'mp3: "http://[^"]*"' myfile.html | grep -o 'http://[^"]*'

Why use awk? sed is better at this:

sed -ne 's/.*\(http[^"]*\).*/\1/p' < foo.js
  • awk = typically used as a data extraction sed = stream editor, which an apply transformation – Ronaldo Nascimento Nov 25 '12 at 22:46
  • Now, how do I use curl to grab each one? – Ronaldo Nascimento Nov 25 '12 at 22:48
  • 1
    sed -ne 's/.*(http[^"]*).*/\1/p' < foo.js | xargs curl – Dennis Kaarsemaker Nov 25 '12 at 22:49
  • 2
    @RonaldoNascimento Note that this only extracts the last URL of each line. This may or may not matter depending on your file format. – Gilles 'SO- stop being evil' Nov 25 '12 at 22:55
  • 1
    That depends on what you want to do. If you want to learn scripting on linux, then yes: focus on languages like perl and python. sed and awk are nice for oneliners but not for bigger tasks. – Dennis Kaarsemaker Nov 25 '12 at 23:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.