XML command line (shell script) manipulation

Question

How to manipulate XML from command line in shell script?

There are many commands for manipulating tabular data, substituting environment variable or replacing text fragments with regex, but I haven't found anything for XML.

My build script need to insert a tag with content within the main tag of the xml document, and I find it an overkill to install java, perl or python in OS for that purpose (my scripts are done in gitlab with docker images, so doing my job with tools available in maven:3.5-jdk-8 image would be a dream).

I don't want to manipulate XML with sed, although in my build script it would work, because it's evil.

Example: I have the following xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>  
  <!-- a lot of other tags-->
</project>  

And I want to insert the following block:

<distributionManagement>
    <repository>
        <id>private-releases</id>
        <url>https://my.private.server.com/nexus/repository/maven-releases/</url>
    </repository>
</distributionManagement>

inside the project tag (and it fully doesn't matter if it will be on the begin or on the end.

Answer 1

XMLStarlet (http://xmlstar.sourceforge.net/overview.php) is written in C and uses libxml2 and libxslt.

Given the XML document

<?xml version="1.0"?>
<root>
  <tag>data</tag>
</root>

a subnode to root may be inserted using

xml ed -s '/root' -t elem -n 'newtag' -v 'newdata' file.xml

which produces

<?xml version="1.0"?>
<root>
  <tag>data</tag>
  <newtag>newdata</newtag>
</root>

Inserting many things (using the original file.xml at the top here):

xml ed -s '/root' -t elem -n 'newtag' \
       -s '/root/newtag' -t elem -n 'subtag' -v 'subdata' file.xml

This produces

<?xml version="1.0"?>
<root>
  <tag>data</tag>
  <newtag>
    <subtag>subdata</subtag>
  </newtag>
</root>

---

For the example in the question:

xml ed -N x="http://maven.apache.org/POM/4.0.0" \
       -s '/x:project' -t elem -n 'distributionManagement' \
       -s '/x:project/distributionManagement' -t elem -n 'repository' \
       -s '/x:project/distributionManagement/repository' -t elem -n 'id' \
         -v 'private-releases' \
       -s '/x:project/distributionManagement/repository' -t elem -n 'url' \
         -v 'https://my.private.server.com/nexus/repository/maven-releases/' \
    file.xml

Result:

<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>
  <!-- a lot of other tags-->
  <distributionManagement>
    <repository>
      <id>private-releases</id>
      <url>https://my.private.server.com/nexus/repository/maven-releases/</url>
    </repository>
  </distributionManagement>
</project>

---

Inserting a previously prepared XML file at a location in the XML:

Assuming the original XML from the question is in file.xml and the additional bits that should go in the new distributinManagement node are in new.xml (but not the node tag itself), one could do the following to insert new.xml in the root node:

xml ed -N x="http://maven.apache.org/POM/4.0.0" \
       -s '/x:project' -t elem -n 'distributionManagement' \
       -v "$(<new.xml)" file.xml | xml unesc | xml fo

XMLStarlet will automatically escape data that needs escaping, such as < and > characters. The xml unesc bit unescapes the inserted data (it actually unescapes the whole document, which may or may not be an issue), and xml fo reformats the resulting XML document.

The result is

<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>
  <!-- a lot of other tags-->
  <distributionManagement>
    <repository>
      <id>private-releases</id>
      <url>https://my.private.server.com/nexus/repository/maven-releases/</url>
    </repository>
  </distributionManagement>
</project>

I'm a tiny bit uneasy about doing it this way, "but it works".

See also this related question on StackOverflow: https://stackoverflow.com/questions/29298507/xmlstarlet-xinclude-xslt

Answer 2

I find it an overkill to install java, perl or python in OS for that purpose (my scripts are done in gitlab with docker images, so doing my job with tools available in maven:3.5-jdk-8 image would be a dream).

it probably still is overkill, but if you are only concerned with the container's size you could use a very lightweight language such as Lua or Guile.

from the Lua docs:

Adding Lua to an application does not bloat it. The tarball for Lua 5.3.4, which contains source code and documentation, takes 297K compressed and 1.1M uncompressed. The source contains around 24000 lines of C. Under 64-bit Linux, the Lua interpreter built with all standard Lua libraries takes 246K and the Lua library takes 421K.