How to remove every pattern match and write it in new file

Question

So i have a long string that have no blanks and i want to remove every time the pattern match. This is how the string looks like:

bdy11<entryunit="phoenix"><name>costperdps</name><value>27.75</value></entry><entryunit="phoenix"><name>damage</name><value>5</value></entry><entrbdy623yunit="phoenix"><name>dps</name><value>9.01</value></entry><entryunit="phoenix"><name>dpsplusbonus</name><value>18.02</value></entry><entryunit="phoenix"><name>dpsplusupgbonus</name><value>23.42</value></entry><entryunit="phoenix"><name>dpsplusupgrades</name><value>14.41</value></entry><entryunit="phoenix"><name>range</name><value></value></entry><entryunit="phoenix"><name>targets</name><value>air</value></entry><entryunit="phoenix"><name>upgrade</name><value>1</value></entry><entryunit=bdy624"phoenix"><name>upgrade_damage</name><value></value></entry><entryunit="photoncannon"><name>attack</name><value>1</value></entry><entryunit="photoncannon"><name>bonus_damage</name><value></value></entry><entryunit="photoncannon"><name>bonus_type</name><value></value></entry><entryunit="photoncannon"><name>cooldown</name><value>1.25</value></entry><entryunit="photoncannon"><name>costperdps</name><value>9.38</value></entry><entryunit="photoncannon"><name>damage</name><value>20</value><bdy625/entry><entryunit="photoncannon"><name>dps</name><value>16.00</value></entry><entryunit="photoncannon"><name>dpsplusbonus</name><value></value></entry><entryunit="photoncannon"><name>dpsplusupgbonus</name><value></value></entry><entryunit="photoncannon"><name>dpsplusupgrades</name><value>16.00</value></entry><entryunit="photoncannon"><name>range</name><value></value></entry><entryunit="photoncannon"><name>targets</name><value>both</value></entry><entryunit="photoncannon"><name>upgrbdy626ade</name><value></value></entry><entryunit="photoncannon"><name>upgrade_damage</name><value></value></entry><entryunit="probe"><name>attack</name><value>1</value></entry><entryunit="probe"><name>bonus_damage</name><value></value></entry><entryunit="probe"><name>bonus_type</name><value></value></entry><entryunit="probe"><name>cooldown</name><value>1.50</value></entry><entryunit="probe"><name>costperdps</name><value>15.00</value></entry><entryunit="probe"><name>damage</name><value>5</valubdy627e></entry><entryunit="probe"><name>dps</name><value>3.33</value></entry><entryunit="probe"><name>dpsplusbonus</name><value></value></entry><entryunit="probe"><name>dpsplusupgbonus</name><value></value></entry><entryunit="probe"><name>dpsplusupgrades</name><value>3.33</value></entry><entryunit="probe"><name>range</name><value></value></entry><entryunit="probe"><name>targets</name><value>ground</value></entry><entryunit="probe"><name>upgrade</name><value></value></entry><entryunit="probe">bdy628<name>upgrade_damage</name><value></value></entry><entryunit="sentry"><name>attack</name><value>1</value></entry><entryunit="sentry"><name>bonus_damage</name><value></value></entry><entryunit="sentry"><name>bonus_type</name><value></value></entry><entryunit="sentry"><name>cooldown</name><value>1.00</value></entry><entryunit="sentry"><name>costperdps</name><value>25.00</value></entry><entryunit="sentry"><name>damage</name><value>6</value></entry><entryunit="sentry"><name>dps</name><value>6bdy629.00</value></entry><entryunit="sentry"><name>dpsplusbonus</name><value></value></entry><entryunit="sentry"><name>dpsplusupgbonus</name><value></value></entry><entryunit="sentry"><name>dpsplusupgrades</name><value>9.00</value></entry><entryunit="sentry"><name>range</name><value></value></entry><entryunit="sentry"><name>targets</name><value>both</value></entry><entryunit="sentry"><name>upgrade</name><value>1</value></entry><entryunit="sentry"><name>upgrade_damage</name><value></value></entrbdy630y><entryunit="stalker"><name>attack</name><value>1</value></entry><entryunit="stalker"><name>bonus_damage</name><value>4</value></entry><entryunit="stalker"><name>bonus_type</name><value>armored</value></entry><entryunit="stalker"><name>cooldown</name><value>1.44</value></entry><entryunit="stalker"><name>costperdps</name><value>25.20</value></entry><entryunit="stalker"><name>damage</name><value>10</value></entry><entryunit="stalker"><name>dps</name><value>6.94</value></entry><entryunit="bdy631stalker"><name>dpsplusbonus</name><value>9.72</value></entry><entryunit="stalker"><name>dpsplusupgbonus</name><value>11.81</value></entry><entryunit="stalker"><name>dpsplusupgrades</name><value>9.03</value></entry><entryunit="stalker"><name>range</name><value></value></entry><entryunit="stalker"><name>targets</name><value>both</value></entry><entryunit="stalker"><name>upgrade</name><value>1</value></entry><entryunit="stalker"><name>upgrade_damage</name><value></value></entry><entryunit="vbdy632oidray"><name>attack</name><value>1</value></entry><entryunit="voidray"><name>bonus_damage</name><value>4</value></entry><entryunit="voidray"><name>bonus_type</name><value>armored</value></entry><entryunit="voidray"><name>cooldown</name><value>.60</value></entry><entryunit="voidray"><name>costperdps</name><value>35.00</value></entry><entryunit="voidray"><name>damage</name><value>6</value></entry><entryunit="voidray"><name>dps</name><value>10.00</value></entry><entryunit="voidray">bdy633<name>dpsplusbonus</name><value>16.67</value></entry><entryunit="voidray"><name>dpsplusupgbonus</name><value>21.67</value></entry><entryunit="voidray"><name>dpsplusupgrades</name><value>15.00</value></entry><entryunit="voidray"><name>range</name><value></value></entry><entryunit="voidray"><name>targets</name><value>both</value></entry><entryunit="voidray"><name>upgrade</name><value>1</value></entry><entryunit="voidray"><name>upgrade_damage</name><value></value></entry><entryunit="vobdy634idraycharged"><name>attack</name><value>1</value></entry><entryunit="voidraycharged"><name>bonus_damage</name><value>8</value></entry><entryunit="voidraycharged"><name>bonus_type</name><value>armored</value></entry><entryunit="voidraycharged"><name>cooldown</name><value>.60</value></entry><entryunit="voidraycharged"><name>costperdps</name><value>26.25</value></entry><entryunit="voidraycharged"><name>damage</name><value>8</value></entry><entryunit="voidraycharged"><name>dps</nbdy635ame><value>13.33</value></entry><entryunit="voidraycharged"><name>dpsplusbonus</name><value>26.67</value></entry><entryunit="voidraycharged"><name>dpsplusupgbonus</name><value>36.67</value></entry><entryunit="voidraycharged"><name>dpsplusupgrades</name><value>18.33</value></entry><entryunit="voidraycharged"><name>range</name><value></value></entry><entryunit="voidraycharged"><name>targets</name><value>both</value></entry><entryunit="voidraycharged"><name>upgrade</name><value>1<bdy636/value></entry><entryunit="voidraycharged"><name>upgrade_damage</name><value>1</value></entry><entryunit="zealot"><name>attack</name><value>2</value></entry><entryunit="zealot"><name>bonus_damage</name><value></value></entry><entryunit="zealot"><name>bonus_type</name><value></value></entry><entryunit="zealot"><name>cooldown</name><value>1.20</value></entry><entryunit="zealot"><name>costperdps</name><value>7.50</value></entry><entryunit="zealot"><name>damage</name><value>8</value></entrybdy637><entryunit="zealot"><name>dps</name><value>13.33</value></entry><entryunit="zealot"><name>dpsplusbonus</name><value></value></entry><entryunit="zealot"><name>dpsplusupgbonus</name><value></value></entry><entryunit="zealot"><name>dpsplusupgrades</name><value>18.33</value></entry><entryunit="zealot"><name>range</name><value></value></entry><entryunit="zealot"><name>targets</name><value>ground</value></entry><entryunit="zealot"><name>upgrade</name><value>1</value></entry><entryunit="zealotbdy638"><name>upgrade_damage</name><value></value></entry></record>trl000000204000001410

I want to remove every time there is bdy followed by 2 digits and bdy followed by 3 digits. So far I tried to use sed with regular expression but I believe that there is a mistake in the regular expression

sed 's/(.*bdy\d\d)//g; s/(.*bdy\d\d\d)//g' newTarget

I plan to use pipe to transform the text with

sed -e "
  s|<tag>|\n&|
  s/.*\n//
  s|</tag>|&\n|
  /\n/P;D
" newTarget > newTarget2

To get each tag on new line so its easier to extract the data between the tags but before i do that i need to remove bdy followed by the numbers because occurs in between tags.

Answer 1

I presume you want to transform your string into valid XML. You will not only need to remove the bdy strings but also need to fix the entry tag and its unit attribute (currently written together as entryunit), add the missing <record> start tag at the start and remove the trl string at the very end.

You may do this with echo and sed, and then reformat it for readability using xmlstarlet:

{
    echo '<record>'
    sed -e 's/bdy[0-9]\{2,3\}//g' \
        -e 's/entryunit/entry unit/g' \
        -e 's/trl[0-9]*$//' file
} | xmlstarlet fo

Or, on a single line,

{ echo '<record>'; sed -e 's/bdy[0-9]\{2,3\}//g' -e 's/entryunit/entry unit/g' -e 's/trl[0-9]*$//' file; } | xmlstarlet fo

The xmlstarlet bit is only necessary if you want to reformat the document nicely. Without reformatting, an XML parser would still be able to read it without issue.

The sed command first deletes all the bdy strings. We match one or two digits after the substring bdy using [0-9]\{2,3\}, which means "two or three digits between 0 and 9".

The entryunit string is corrected into entry unit using a straightforward substitution.

We delete the trl string and the trailing string of digits with the last substitution.

This, with xmlstarlet reformatting, should give you output looking like this:

<?xml version="1.0"?>
<record>
  <entry unit="phoenix">
    <name>costperdps</name>
    <value>27.75</value>
  </entry>
  <entry unit="phoenix">
    <name>damage</name>
    <value>5</value>
  </entry>

[...]

  <entry unit="zealot">
    <name>upgrade</name>
    <value>1</value>
  </entry>
  <entry unit="zealot">
    <name>upgrade_damage</name>
    <value/>
  </entry>
</record>

Regardless of whether you reformat the resulting XML document with xmlstarlet or not later, you may extract data using the tool like so:

xmlstarlet sel -t -v '//entry[@unit="photoncannon" and name="cooldown"]/value' -nl file

The above outputs the value of the value node belonging to the entry whose unit attribute is photoncannon and whose name child node's value is cooldown. In other words, it gets the cool-down value for the photon cannon.

With parameters taken from shell variables:

unit=photoncannon
name=cooldown
xmlstrlet sel -t \
    --var unit="'$unit'" \
    --var name="'$name'" \
    -v '//entry[@unit=$unit and name=$name]/value' -nl file

(Note the specific quoting used; the internal variables $unit and $name must contain XML-encoded strings, which means they need to have explicit quotes added.)

You could obviously use any other XML parser to do the same thing. The following uses xq (from here), which is an XML parser sitting on top of jq, the JSON parser, which means it first translates your XML document into JSON, and then calls jq to parse the resulting data:

unit=photoncannon
name=cooldown
xq -r \
    --arg unit "$unit" \
    --arg name "$name" \
    '.record.entry[] | select(."@unit" == $unit and .name == $name).value' file