2

I have a HTML document that looks (when oversimplified) like this:

<html>
  <body>
    <a href="...">...</a>
    <a href="...">...</a>
    <a href="...">...</a>
    ...
  </body>
</html>

What I'd like to do would be to extract the URLs in line-delimited output. Enter xmllint:

$ xmllint --html --xpath //a/@href
href="..." href="..." href="..."

It's getting the attribute, the whole attribute including the name, and it's outputting them space-delimited. How can I just get a list of lines with the values of the href attribute? I want output like this:

...
...
...

where ... is the URL found in the href attribute of each a element.

How can I format this output properly?

  • 1
    Does this help? – eyoung100 Jul 31 '15 at 4:28
  • 1
    don't do that thing in that link! – mikeserv Jan 15 '16 at 19:19
1

Given file.html:

<html>
  <body>
    <a href="url1">link text 1</a>
    <a href="url2">link text 2</a>
    <a href="url3">link text 3</a>
    ...
  </body>
</html>

We can use Unix pipes to send existing xmllint's output, to sed and see this result:

$ xmllint --html --xpath //a/@href input.html | sed 's/ href="\([^"]*\)"/\1\n/g'
url1
url2
url3

Explanation

With xmllint alone, we only get:

$ xmllint --html --xpath //a/@href input.html
 href="url1" href="url2" href="url3"%
  • the trailing % indicates there is no trailing newline

One of the benefits of Unix-like systems is we can benefit from Doug McIlroy's pipes feature, so we don't have to have one program try to do everything, we are in fact encouraged to combine programs to suit our needs.

So, finding xmllint's output unsatisfactory, we pipe to combine it with our sed command, which:

  • searches for href="URL" units
  • using \( \) grouping to surround the URL part
  • and replacing it with \1\n so it references the group we defined around the URL, while also adding a new line after that matched \1

In this way we combine xmllint and sed to obtain the desired line-delimited output, one URL per line.

0

Have you considered using sed:

sed -n 's/.*href="\([^"]*\).*/\1/p'

  • that gets only one per line - and i mean input lines not output lines. and so you lose <a href="http...."> </a> <a href="http...."> the first. it also gets <div text_attribute="bla...href="... all of this ... " – mikeserv Jan 15 '16 at 19:13
  • You are right. I used the sample from above which has each href on separate lines. @mikeserv – cesar Jan 15 '16 at 19:19
  • 1
    you can do it - you just need to be more conservative. the .* is usually an issue for these kinds of things. – mikeserv Jan 15 '16 at 19:20
  • 1
    this might help as an example. and there's this. – mikeserv Jan 15 '16 at 19:27
  • no problem. let me know if you work it out so i can upvote a worthy edit. – mikeserv Jan 15 '16 at 19:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.