1

Suppose you have the following file:

...
<tag2>
    a
    b
    c
    a
</tag2>
...
<tag2>
    x
    y
    y
    z
    x    
</tag2>

How to remove the duplicate lines inside each <tag2> like the example below?

...
<tag2>
    a
    b
    c
</tag2>
...
<tag2>
    x
    y   
    z
</tag2>

I would like to search every file in the directory and sub directories and removed these duplicates.

  • Does the file have that exact layout, consistently? Is there any structure to the a, b, etc? Can you use a real XML parser here? – Michael Homer Apr 18 at 20:26
  • @MichaelHomer not really, the number of any tags are unknown. And the order and number of <tag2> tags are also uknown. – orangesky Apr 18 at 20:28
1

XSLT 2.0 solution:

<xsl:template match="tag2">
  <tag2>
    <xsl:value-of select="distinct-values(tokenize(., '&#xa;'))"/>
  </tag2>
</xsl:template>
0

Unsure how complex your file could be, but for the example given this appears to work.

$ awk '/^<[a-z]/{print;delete z}!/^</{z[$0]=1}/^<\//{for(x in z){print x}print}' file1
<tag2>
    a
    b
    c
</tag2>
<tag2>
    x
    y
    z
</tag2>
$

Commented version

awk '/^<[a-z]/ {         # If start tag
         print           #     Print line
         delete z        #     Clear array
     } !/^</ {           # If not a tag
         z[$0]=1         #     Store line
     } /^<\// {          # If end tag
         for(x in z) {   #     For each array entry
             print x     #         Print array entry
         }
         print           #     Print end tag
     }' file1
  • Using regular expressions to process XML is fine for a one-off, but never do it as part of a production process. It will only work if the XML conforms to the exact rules that the script assumes, and in production you want something that can handle any legal XML. – user32929 Apr 19 at 7:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.