0

I have a text file td.txt which has this content

More here || <--field="data.com":CHK test text--> || test 2 test x @ -- 
||<!-- more data 
here --> ||
echo "||<!--field="data.com":CHK test text-->|| test 2 test x @ -- ||<--field="data.com":CHK more data here -->|| test data

I wish to remove all instances of <!-- and all text inside the tags -->

I have this sed expression cat td.txt | sed 's/<!--[^P]*-->//g'

which does remove the tags and their contents as desired, but not when the <!-- tag starts on one line and ends on another line with -->.

The result using the above expression is

More here || <--field="data.com":CHK test text--> || test 2 test x @ -- 
||<!-- more data 
here --> ||
echo "|||| test data

showing the

<!-- more data 
    here -->

tag still present

How do I remove all instances of <!-- and all text inside the tags --> and
<!-- and all text
inside the tags -->

using sed or grep or any other tool ?

1
  • 1
    If you are dealing with XML documents, then consider including some real and well-formed example documents. The text that you have quoted now is not XML and one can't work with it unless one modifies it substantially. In short, please include a representative sample of the document that you're working with. (it's easy to strip comments out of an XML document).
    – Kusalananda
    Aug 10, 2021 at 11:53

3 Answers 3

6

To delete comments from an XML document, you may use xmlstarlet:

xmlstarlet ed -d '//comment()' file.xml

It does not matter if the comments span multiple lines.

The xmlstarlet tool supports in-place editing through its -L (--inplace) option.

6
  • Thanks. The data has already been stripped back from xml and the first operation, while it's still xml, needs to be something else, therefore the sample presented is not wholly unrepresentative. I can see xmlstarlet is handy, but would prefer a more dynamic solution like grep or sed that will work over multi-line tags of any format.
    – Kes
    Aug 10, 2021 at 12:07
  • 2
    @Kes I would strongly suggest that you do not destroy the document structure by using tools unaware of XML to modify the file. It is much more convenient and safer (and efficient) to work with XML parsers than with text editing tools. Now, you can no longer know whether -- ends a comment or whether it's part of non-tag data, without actually implementing an XML parser in sed or whatever tool you end up using.
    – Kusalananda
    Aug 10, 2021 at 12:11
  • There have been so many vulnerabilities in XML parsing libraries, that I wouldn't be so sure about the safer part. Aug 10, 2021 at 14:52
  • 1
    @StéphaneChazelas You know that bit better than I do. When I said "safer" I meant that it is easier to get or modify the correct data without accidentally fetching or modifying parts of the document that shouldn't be fetched or modified. Is there a better word for this?
    – Kusalananda
    Aug 10, 2021 at 14:55
  • Thanks @kusalananda "Now, you can no longer know whether -- ends a comment or whether it's part of non-tag data". I don't understand becasue -- is not the search termination term, rather --> is, so parsing for -- does not occur with non xmlstarlet tools if the search is performed correctly?
    – Kes
    Aug 12, 2021 at 13:36
3

I'd use perl:

perl -0777 -pe 's/<!--.*?-->//sg' < your-file

Where -p enables the sed mode, -0777 changes the record separator to an impossible character which means the whole file is processed as one record (slurp mode) and s///sg replaces all matches of the <!--.*?--> regexps with nothing, the s flag so that . also matches on newline characters, and g to replace all occurrences, not just the first.

That doesn't care whether the input is valid XML or not, as it makes no attempt at parsing the input as XML, so it will work even if the input is not valid XML, but in theory, it could end up removing things that are not XML comments if those <!-- or --> occur in <!CDATA[ parts for instance.

With sed, you can do something similar with:

sed ':1
     $!{
       N;b1
     }
     s/_/_u/g; s/(/_</g; s/)/_>/g; # escape _, ( and ) characters
     s/<!--/(/g; s/-->/)/g; # replace <!-- and --> with
                            # single characters ( and )

     s/([^)]*)//g; # remove the comments

     s/)/-->/g; s/(/<!--/g; # revert earlier replacement
     s/_>/)/g; s/_</(/g; s/_u/_/g; # revert earlier escaping'

As standard sed doesn't have perl's *? operator (the non-greedy version of *), we're left with using ([^)]*) instead to match on ( followed by any number of characters other than ) followed by ) after having replaced the <!-- with ( and --> with ), itself after having saved away the ( and ) characters that were originally in the input as _< and _> (and _ as _u).

Note the loop beforehand that slurps the whole input into the pattern space (beware some sed implementations have a low limit on the size of the pattern space, not GNU sed though).

3
  • Thanks. I have gnu sed v4.8 and perl v5.34.0. Both more or less up to date. The perl expression removes <!-- my text here --> when all on the same line, but does not remove the tags when across multiple lines ie <!-- my text \n\n\n\ here -->
    – Kes
    Aug 10, 2021 at 15:55
  • @Kes, did you see the update with the added s flag which was missing in the initial version of my answer? (thanks to @fra-san for spotting it). Aug 10, 2021 at 15:56
  • @urcodebetterznow. Thanks. I had forgotten to revert the replacements after the comments had been removed. Should be fixed now. Aug 10, 2021 at 17:51
1
sed '/<!--/!b;:1;/-->/!{N;b1};s/-->/\a/;s/<!--.*\a//' file

When the <!-- tag appears, we check for the presence of --> tag, if it is not there, add the line to the buffer and return to label 1. Replace the first --> tag with a character not found in text files \r, \f, or \a. Remove from the tag <!-- to the anchor.

Let's complicate the task. Suppose several search tags may appear in a line at once:

awk 'NR>1 {sub(/-->/,"\r"); sub(/.*\r/, "")}1' RS='<!--' ORS= file

Probably, only GNU awk is suitable for the last solution, because if, say, there is no separator <!-- In the file, then it will be completely read in to the buffer.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .