1

I have the following XML piece:

<value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">+380554446363</value>
<value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">+380554446364</value>
<value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">+380554446365</value>

I am trying to replace the <value> tag contents with their SHA-1 hashes using the following command:

cat test.xml | sed "s/>[+]\([0-9][0-9]*\)<\/value>/>+$(echo \\1 | sha1sum | cut -f1 -d' ')<\/value>/g"

It fails by replacing all found cases with the same incorrect value.

Expected:

<value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">34df370575e3528b31daef8633cb539119a3b028</value>
<value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">d93767c769fd51bcf9eb25f95932559b24bae812</value>
<value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">20338c1f048bed553b6cce76eaf1d388ba7686f5</value>

Got:

<value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">+cbcac786fef5abeb39fe473ab6abe554978a8156</value>
<value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">+cbcac786fef5abeb39fe473ab6abe554978a8156</value>
<value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">+cbcac786fef5abeb39fe473ab6abe554978a8156</value>

What could I be doing wrong? TIA.

4
  • 3
    This would be easier if the XML was well-formed. It's currently missing a root node and the values nodes lacks its end tag. Should we just improvise those bits?
    – Kusalananda
    Commented Dec 22, 2021 at 23:13
  • Yes, the only lines that actually matter are the ones with the <value> tag. I'll edit the question to reflect that. Commented Dec 22, 2021 at 23:13
  • Do you realise that your SHA checksums are all incorrect? They hash not only the value but also a trailing newline Commented Dec 28, 2021 at 18:19
  • I completely missed that, thank you for pointing it out. Commented Dec 28, 2021 at 20:21

4 Answers 4

4

Don't edit XML with sed or awk. Use a proper xml parser, e.g. python's xml.etree:

input.xml:

<values>
    <value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">+380554446363</value>
    <value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">+380554446364</value>
    <value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">+380554446365</value>
</values>
#!/usr/bin/env python3
import hashlib
import xml.etree.ElementTree as ET

tree = ET.parse('input.xml')
root = tree.getroot()

for value in root:
    value.text = hashlib.sha256(value.text.encode('utf-8')).hexdigest()


tree.write('output.xml')

output.xml:

<values>
    <value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">4b2bdff20d17dc4ae7ad99937399530b39bd7a63f7348375c547d01565c11898</value>
    <value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">8d3b1452d83ef803f8043eaddbe2fc58b0fd42c8ad5abc554e78796548f75ddb</value>
    <value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">06bd986417875e536401306180c51b7ded4eeab179933e0026a103bc8dc9eee7</value>
</values>
4
  • In normal circumstances I would be doing that but, unfortunately, in this particular case I have to solve it using just sed. Commented Dec 22, 2021 at 23:34
  • Does that module do sha1 too, as the question does not seem want to use sha256.
    – Kusalananda
    Commented Dec 22, 2021 at 23:41
  • It does, although it's not really relevant in my case. Commented Dec 22, 2021 at 23:55
  • 2
    "in this particular case I have to solve it using just sed". No you don't. As a professional, you should insist on using the proper tools for the job. Commented Dec 23, 2021 at 11:13
3
+50

The sha1sum is evaluating the SHA-1 of the constant string "\\1" instead of the first SED regex match:

$ echo \\1 | sha1sum
cbcac786fef5abeb39fe473ab6abe554978a8156  -

The shell performs all the various expansions (e.g. command substitutions) before executing the command (in this case, sed). Thus, shell expands

cat test.xml | sed "s/>[+]\([0-9][0-9]*\)<\/value>/>+$(echo \\1 | sha1sum | cut -f1 -d' ')<\/value>/g"

to

cat test.xml | sed "s/>[+]\([0-9][0-9]*\)<\/value>/>+cbcac786fef5abeb39fe473ab6abe554978a8156<\/value>/g"

It then runs two processes, one running

cat test.xml

and another running

sed "s/>[+]\([0-9][0-9]*\)<\/value>/>+cbcac786fef5abeb39fe473ab6abe554978a8156<\/value>/g"

with the STDOUT of the first process piped to the STDIN of the second process.

In order for what you're trying to do to work, sed would have to be able to execute other executables from within sed. I don't believe sed can do that, so you'll have to do it some other way.

You can do it using sed, e.g. this is one way

for a in `cat test.xml | sed -E 's,^.*>(\+[0-9]+)<\/value>$,\1,'`; do echo "$a" | sha1sum | cut -f1 -d' '; done >2nd
cat test.xml | sed -E 's,>\+[0-9]+</value>$,>,' >1st
paste -d '' 1st 2nd | sed -E 's,$,</value>,'

You've also not included the "+" in the brackets enclosing the first sed match string, from the SHA1 sums you've provided as the expected results, I believe you want the "+" included, so I've corrected that as well.

1
  • Thank you! That seems to answer more of my questions than I was able to figure out so far. However, I notice that, when running cat test.xml | sed "s/>[+]\([0-9][0-9]*\)<\/value>/>+$(echo \\1)<\/value>/g" (without piping the result to sha1sum), it seems to process the backreference correctly and outputs not the constant string "\\1" but the actual matched values, unmodified. What could be the case here? Commented Dec 28, 2021 at 20:48
1

GNU sed substitute command has an /e modifier which executes as vash code, the rhs of the s/// aka, the modified pattern space.

sed -E "
  s/'/&\\\\&&/g
  s:(<.*>)([+][0-9]+)(</value>.*):printf '%s%s%s' '\\1' \"\$(echo '\\2'|sha1sum|cut -d' ' -f1)\" '\\3':e
" test.xml

<value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">34df370575e3528b31daef8633cb539119a3b028</value>
<value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">d93767c769fd51bcf9eb25f95932559b24bae812</value>
<value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">20338c1f048bed553b6cce76eaf1d388ba7686f5</value>
1

If you're on a system with an compiled with crypto extension functions you could say:

xmlstarlet sel -N crypto='http://exslt.org/crypto' \
  -t -m '//value' -e '{name()}' -c '@*' -v 'crypto:sha1(.)' -b -n \
file.xml

Output:

<value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">3bacaac1e104d7bfde5ff462d461d3b4b917c037</value>
<value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">808b3e7bb8c8c7c06c328a082bac64205837531b</value>
<value id="1" creatorId="0" creationTime="1639487132" expirationTime="1639573532">8c31811a3b8d816f719ad5c78b58810b9b6cd4b7</value>

where the checksums differ from the desired output as they're not computed from texts including newlines.


Running xmlstarlet transform --show-ext 2>&1 | grep crypto at my end outputs:

{http://exslt.org/crypto}md4
{http://exslt.org/crypto}sha1
{http://exslt.org/crypto}md5
{http://exslt.org/crypto}rc4_decrypt
{http://exslt.org/crypto}rc4_encrypt

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .