7

I have the following in a txt file:

<ol><li><b><a href="/page1/Mark_Yato" title="Mark Yato">Mark Yato</a> ft. MarkAm &amp; <a href="/page1/Giv%C4%93on" title="Givēon">Givēon</a> - <a href="/page1/Mark_Yato:Thuieo" title="Mark Yato:Thuieo">Thuieo</a> (7)</b></li>
<li><b><a href="/page1/The_Central" title="The Central">The Central</a> - <a href="/page1/The_Central:AHTIOe oie" title="The Central:AHTIOe oie">AHTIOe oie</a> (7)</b></li>
<li><b><a href="/page1/Taa_Too_A" title="Taa Too A">Taa Too A</a> - <a href="/page1/Taa_Too_A:ryhwtyw w" title="Taa Too A:ryhwtyw w">ryhwtyw w</a> (8)</b></li>

and am trying to make it output as the following:

Mark Yato ft. MarkAm & Givēon - Thuieo
The Central - AHTIOe oie
Taa Too A - ryhwtyw w

To achieve this, I thought I would try removing '<', '>' and everything between them so it's left with just the list I'm trying to get.

I tried the following sed command already:

sed 's/<[^()]*>//g'

but this is outputting just the following:

(7)
(7)
(8)

What am I doing wrong and how can I fix the sed command or translate it into awk if it is better use for that?

5
  • 2
    Is your text file actually an HTML document (i.e. a web page)? Or does it really only have snippets like the example? – roaima Jun 3 at 23:16
  • 4
    Looks like you simply mis-typed <> as (). – Toby Speight Jun 4 at 7:28
  • Use pup to parse html – annahri Jun 4 at 8:01
  • 1
    The pattern <[^()]*> tells to match a <, then any number of characters (*) that are not ( or ) ([^()]), then a >. On the first line, that matches the very first <, then everything up to a> just before the space and (7). And repeats for the last two tags. The [^()] could match the > and space too, but it needs to track back to let the > be matched too. – ilkkachu Jun 5 at 0:41
  • Have you tried jQuery? php.net/manual/en/function.strip-tags.php – Levente Jun 6 at 20:03
25

Parsing markup with regular expressions is notoriously problematic.

While not an issue with your sample data, angle brackets may appear in tag attributes, comments and possibly other places, making regular expressions that match from < to > unreliable.

You should resort to tools that implement a markup parser.

For instance, using pandoc (version >= 2.8) with your sample data (without adding the missing </ol> tag):

$ pandoc -f html -t plain file 
Mark Yato ft. MarkAm & Givēon - Thuieo (7)

The Central - AHTIOe oie (7)

Taa Too A - ryhwtyw w (8)

You may then easily post-process this output as regular text to remove empty lines and other unwanted parts:

$ pandoc -f html -t plain file |
  sed -e '/^$/d' -e 's/[[:blank:]]*([[:digit:]]*)$//'
Mark Yato ft. MarkAm & Givēon - Thuieo
The Central - AHTIOe oie
Taa Too A - ryhwtyw w

Note that, before version 2.8, pandoc used to convert any emphasized text to all-caps when generating output in plain format. The <b> tag in your list items would trigger this behavior (more on this in the changelog or the relevant commit on GitHub).

Depending on your actual input data, a workaround could be to use markdown as pandoc's input format, either explicitly:

pandoc -f markdown -t plain file

or implicitly, considering it is what pandoc automatically defaults to (pandoc -t plain file).

3
  • I do wonder how often > characters appear in the "wrong" places for this in HTML documents in the wild. I mean, I've done some HTML scraping with regexes back when, and never even stopped to think about the possibility. In any case, you could probably write a regex to match tags with attributes inside, it'd just be more hairy. (But just recognizing the tags should still be possible, since I don't think there's any nesting there. Checking if the end tags match the start tags or such would be different, though.) – ilkkachu Jun 5 at 0:35
  • 1
    @ilkkachu I agree that > is unlikely to be found in unexpected places, but I believe that crafting reliable regular expressions is likely not worth the effort (for instance, I just realized that tags don't need to be opened and closed on the same line) - unless the issue at hand came with really specific assumptions/constraints. Also, my point is to encourage "better" practices, under the usual assumption that others will apply what they find in these Q/As to their own, different problems. – fra-san Jun 5 at 10:09
  • ("tags don't need to be opened and closed on the same line" is ambiguous; I meant that < and the corresponding > can be on different lines). – fra-san Jun 5 at 10:30
9

You're almost there - regex matches are "greedy", so you need to tell the pattern that the closing > character is not allowed inside the pattern. Put another way, the [^()]* portion inside the pattern will match as much text as it can "greedily". If you don't tell the pattern to exclude the closing > from this portion of the pattern, the opening < and closing > that the Regex uses are not necessarily the ones that are paired from an HTML point of view.

Use this instead:

sed -e 's/<[^>]*>//g'

This forces the regex to delete every HTML tag, not a bigger block of text that has < and > at the ends, as well as < or > in the middle.

1
  • Might break on something like <span class="<test>">Hello, world</span> – aragaer Jun 5 at 11:52
6

You could use php to strip all HTML tags and to convert the HTML entities back to normal characters:

$ <file php -r 'echo htmlspecialchars_decode(strip_tags(file_get_contents("php://stdin")), ENT_HTML5);'
Mark Yato ft. MarkAm & Givēon - Thuieo (7)
The Central - AHTIOe oie (7)
Taa Too A - ryhwtyw w (8)

To additionally remove blanks (spaces, tabs) followed by an opening (, followed by one or more numbers and a closing ) at the line endings with sed:

$ <file php -r 'echo htmlspecialchars_decode(strip_tags(file_get_contents("php://stdin")), ENT_HTML5);' |
    sed 's/[[:blank:]]*([[:digit:]][[:digit:]]*)$//'
Mark Yato ft. MarkAm & Givēon - Thuieo
The Central - AHTIOe oie
Taa Too A - ryhwtyw w
6

so this command should do the trick...

sed -i 's/<[^>]*>//g' file

result file should be this

Mark Yato ft. MarkAm &amp; Givēon - Thuieo (7)
The Central - AHTIOe oie (7)
Taa Too A - ryhwtyw w (8)

From there you can do this

sed -i 's/([0-9])//g' file

and it should give you the desired output of this

Mark Yato ft. MarkAm &amp; Givēon - Thuieo
The Central - AHTIOe oie
Taa Too A - ryhwtyw w

EDIT: update second command to look neater.

EDIT: Alternative oneliner

sed -i 's/<[^>]*>//g;s/([0-9])//g' file
3
  • The problem is angle bracket symbols inside quotes, or quoted items inside angled brackets. May not be a problem in this case but be aware.... – Stilez Jun 4 at 10:18
  • @Stilez wouldn't any such visual angle brackets actually be represented by &lt; and &gt; in the HTML and so wouldn't be an issue. – Ed Morton Jun 4 at 13:24
  • No. The OP says this is the content of a text file. As a text file, assume the content is literal. If it meant &gt/lt; then the file content would say so. – Stilez Jun 4 at 13:47
4

Using xmlstarlet:

xmlstarlet fo -H file |
xmlstarlet sel -E latin1 -t -v '//li' -nl 2>/dev/null |
xmlstarlet unesc | sed 's/ [^ ]*$//'

This uses xmlstarlet to convert the HTML fragment to a well-formed HTML document (the 1st command). It then extracts the value of each li node (The 2nd command). Finally, it decodes any HTML entities (&amp; for example). The final sed command just deletes anything after the last space on each line (there are numbers in parentheses that shouldn't be part of the output).

The output given the document in the question:

Mark Yato ft. MarkAm & Givēon - Thuieo
The Central - AHTIOe oie
Taa Too A - ryhwtyw w
3

If you have lynx, then with nolist option the following command would work as:

lynx -dump -nolist -force_html file | awk '{sub(/^[[:space:]]*[0-9]+\. /,""); }1'
lynx -dump -nolist -force_html --display-charset=UTF-8 --assume-charset=UTF-8 file | awk '{sub(/^[[:space:]]*[0-9]+\. /,""); }1'

## Mark Yato ft. MarkAm & Givēon - Thuieo (7)
## The Central - AHTIOe oie (7)
## Taa Too A - ryhwtyw w (8)

To remove characters from last space, regex / [^ ]*$/ would work as suggested in this answer. To get desired output, the command would be changed to:

lynx -dump -nolist -force_html file | awk '{sub(/^[[:space:]]*[0-9]+\. /,""); sub(/ [^ ]*$/,""); }1'
lynx -dump -nolist -force_html --display-charset=UTF-8 --assume-charset=UTF-8 file | awk '{sub(/^[[:space:]]*[0-9]+\. /,""); sub(/ [^ ]*$/,""); }1'

## Mark Yato ft. MarkAm & Givēon - Thuieo
## The Central - AHTIOe oie
## Taa Too A - ryhwtyw w

If file extension is html, then -force_html may be omitted in lynx command as suggested by @Căcărău. The above command may not work properly in some cases. Therefore it is better to include --display-charset=UTF-8 --assume-charset=UTF-8 to lynx command because UTF-8 is common character encoding on internet.

8
  • 1
    fwiw, use -force_html and lynx won't care about the file extensions – Căcărău Jun 4 at 8:14
  • 1
    You should mention that your awk scripts require GNU awk for \s or change each \s to [[:space:]] and then it'll behave the same in any POSIX awk. – Ed Morton Jun 4 at 13:34
  • 1
    @EdMorton Thanks. I use gawk only. Edited. – Prabhjot Singh Jun 4 at 13:44
  • 1
    You're welcome. If you always test your scripts with both gawk --lint and gawk --posix before posting them then between them they'll tell you about most (or maybe all) of the places you're using non-POSIX gawk extensions and then you can either say the script is gawk-only or change it to use a POSIX equivalent. Try, for example, awk --lint -v FS="" 'BEGIN{print gensub(/\s/,"y",1,"foo")}' and awk --posix -v FS="" 'BEGIN{print gensub(/\s/,"y",1,"foo")}'. – Ed Morton Jun 4 at 13:53
  • 1
    Use both --lint and --posix in separate calls to gawk. They each identify some issues that the other doesn't and when you use them together in single call (e.g. awk --lint --posix -v FS="" 'BEGIN{print gensub(/\s/,"y",1,"foo")}') the --lint assumes you want to ONLY check POSIX constructs so the output isn't as useful as you won't get any specific warnings about using gawk extensions. – Ed Morton Jun 4 at 14:01
1

For all of you regex lovers seeking the perfect match, one that does take into account the <>'s inside quotes would be:

sed -E 's/<([^">]|"[^"]*")*>//g'

Which means: "REPLACE a < followed by any number of ((not a > or a ") OR (a ", but then it must be followed by any number of (not a ") and then a closing ") WITH nothing, globally."

It requires extended regexes (-E) to be able to branch (i.e. the "OR").

New contributor
B. Launay is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
0

Instead of having sed remove the contents, I could see using it to insert newlines, followed by an egrep to remove the angled bracket lines. sed 's/</\n</g' file | sed 's/>/>\n/g' | egrep -v "<|>" First sed puts a return character before opening bracket. Second sed puts a return character after the closing bracket. The grep returns all lines without brackets.

1
  • ignore my answer... I didn't take into account the additional brackets inside the lines. – Dan Jun 4 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.