# Grouping and capturing works for toy problem, not real world

Toy problem:

``````\$ echo "foo <a href="/topic/null-hypothesis/" data-sc="text link:topic link">Null hypothesis</a> bar" | sed -E 's@.*<a href=/topic/[^>]*>([^<]*)</a>.*@\1@'
Null hypothesis
``````

Real world (sed does not filter anything):

``````\$ cat *html | grep '<a href="/topic' | sed -E 's@.*<a href=/topic/[^>]*>([^<]*)</a>.*@\1@'
<a href="/topic/null-hypothesis/" data-sc="text link:topic link">Null hypothesis</a>,
<a href="/topic/approximation/" data-sc="text link:topic link">Approximation</a>,
<a href="/topic/estimation-methods/" data-sc="text link:topic link">Estimation methods</a>,
<a href="/topic/statistical-variance/" data-sc="text link:topic link">Statistical variance</a>,
<a href="/topic/identifiability/" data-sc="text link:topic link">Identifiability</a>,
<a href="/topic/preliminary-estimates/" data-sc="text link:topic link">Preliminary estimates</a>,
<a href="/topic/matrix-inversion/" data-sc="text link:topic link">Matrix inversion</a>
``````

What change needs to made to get 'Null hypothesis'?

PS:

``````\$ cat *html | grep -n10 '<a href="/topic' | sed -E 's@.*<a href=/topic/[^>]*>([^<]*)</a>.*@\1@'
538-
539-
540-
541-
542-
543-
544-        <div class="topics-list mtl">
545-            <p class="hide">You can always find the topics here!</p>
546-            <strong>Topics:</strong>
547-
548:                <a href="/topic/null-hypothesis/" data-sc="text link:topic link">Null hypothesis</a>,
549-
550:                <a href="/topic/approximation/" data-sc="text link:topic link">Approximation</a>,
551-
552:                <a href="/topic/estimation-methods/" data-sc="text link:topic link">Estimation methods</a>,
553-
554:                <a href="/topic/statistical-variance/" data-sc="text link:topic link">Statistical variance</a>,
555-
556:                <a href="/topic/identifiability/" data-sc="text link:topic link">Identifiability</a>,
557-
558:                <a href="/topic/preliminary-estimates/" data-sc="text link:topic link">Preliminary estimates</a>,
559-
560:                <a href="/topic/matrix-inversion/" data-sc="text link:topic link">Matrix inversion</a>
561-
562-        </div>
563-
564-        <div class="mvl left">
565-
566-
567-
568-
569-<div id="flag-description" aria-live="assertive">
570-    <a class="hover" data-qa="give-feedback" data-toggle="flag-reason" href="#" title="Give feedback on the topics for this item.">
``````

ṔS2: the complete *html file: https://pastebin.com/RLnWXKWe

• How could `href=/topic/` match `href="/topic/`. Double quotes can't be just inserted anywhere. – choroba Jun 30 '19 at 19:52
• Because `echo "foo <a href="/topic/null-hypothesis/" data-sc="` has a double-quoted string `foo <a href=`, an unquoted string `/topic/null-hypothesis/`, a second double-quoted string ` data-sc=`, and so on, meaning the quotes are not present when `sed` sees the text. I believe it would work otherwise in both cases with the appropriate changes. – Michael Homer Jun 30 '19 at 22:39
• This is what the toy thing should have been: `\$ echo "foo <a href=\"/topic/null-hypothesis/\" data-sc=\"text link:topic link\">Null hypothesis</a> bar" | sed -E 's@.*<a href=\"/topic/[^>]*>([^<]*)</a>.*@\1@' Null hypothesis` Actually, not necessary to escape the dq in the replacement. – Erwann Jul 1 '19 at 1:59

## 2 Answers

`cat *html | grep -oE '\"\/.*\/\"' | awk -F'/' '{print \$(NF-1)}'`
This should work fine.

You should try running your first command just up to (but not including) the `|` — i.e., just the `echo` command.

``````\$ echo "foo <a href="/topic/null-hypothesis/" data-sc="text link:topic link">Null hypothesis</a> bar"
foo <a href=/topic/null-hypothesis/ data-sc=text link:topic link>Null hypothesis</a> bar
``````

See anything wrong?  Were you expecting `href=` and `data-sc=` to be followed by quoted strings?

Your `echo` command is wrong.  The `"` at the beginning doesn’t match the `"` at the end; it matches the first one it finds:

``````\$ echo "foo <a href="/topic/null-hypothesis/" data-sc="text link:topic link">Null hypothesis</a> bar"
↑▲▲▲▲▲▲▲▲▲▲▲▲↑.......................↑▲▲▲▲▲▲▲▲▲↑....................↑▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲↑
``````

The characters with `▲` symbols under them are between quotes — so you have successfully quoted the `<` and `>` characters, which would have caused a mess it they weren’t quoted.  The characters with `.` under them aren't quoted.  And the quotes themselves aren’t quoted!

The simplest way to quote a quote is to use the other kind of quote — i.e., change the first and last `"` to `'`.

Then fix your sed command to work with the correct toy data, like this:

`sed -E 's@.*<a href="/topic/[^>]*>([^<]*)</a>.*@\1@'`
Note the added `"` after the `href=`.

and you should be fine with the real data.

• "You should try running your first command just up to (but not including) the | — i.e., just the echo command." this answer comes after this comment, so I'm not sure it's still relevant. – Erwann Jul 1 '19 at 3:48