9

How to manipulate XML from command line in shell script?

There are many commands for manipulating tabular data, substituting environment variable or replacing text fragments with regex, but I haven't found anything for XML.

My build script need to insert a tag with content within the main tag of the xml document, and I find it an overkill to install java, perl or python in OS for that purpose (my scripts are done in gitlab with docker images, so doing my job with tools available in maven:3.5-jdk-8 image would be a dream).

I don't want to manipulate XML with sed, although in my build script it would work, because it's evil.

Example: I have the following xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>  
  <!-- a lot of other tags-->
</project>  

And I want to insert the following block:

<distributionManagement>
    <repository>
        <id>private-releases</id>
        <url>https://my.private.server.com/nexus/repository/maven-releases/</url>
    </repository>
</distributionManagement>

inside the project tag (and it fully doesn't matter if it will be on the begin or on the end.

  • post your input xml and expected output – RomanPerekhrest Feb 9 '18 at 12:12
  • So the specific requirements is for an XML parser that can be invoked from the command line that is not implemented in any of the major scripting languages, but a freestanding C or C++ (or other compiled) utility? – Kusalananda Feb 9 '18 at 12:17
  • @Kusalanda I've specified I'm running scipts within docker containers, so it's most important for me to add as little to the docker image as possible. – 9ilsdx 9rvj 0lo Feb 9 '18 at 12:21
  • If you have an image with maven and a jdk then Java sounds like the best option to me.... why do you consider Java heavyweight in this case? – Daniel Pryden Feb 9 '18 at 17:36
  • It's probably worth asking this question on Stack Overflow and tagging with maven -- I suspect there's a better way to do what you're trying to do within Maven itself. – Daniel Pryden Feb 9 '18 at 17:38
10

XMLStarlet (http://xmlstar.sourceforge.net/overview.php) is written in C and uses libxml2 and libxslt.

Given the XML document

<?xml version="1.0"?>
<root>
  <tag>data</tag>
</root>

a subnode to root may be inserted using

xml ed -s '/root' -t elem -n 'newtag' -v 'newdata' file.xml

which produces

<?xml version="1.0"?>
<root>
  <tag>data</tag>
  <newtag>newdata</newtag>
</root>

Inserting many things (using the original file.xml at the top here):

xml ed -s '/root' -t elem -n 'newtag' \
       -s '/root/newtag' -t elem -n 'subtag' -v 'subdata' file.xml

This produces

<?xml version="1.0"?>
<root>
  <tag>data</tag>
  <newtag>
    <subtag>subdata</subtag>
  </newtag>
</root>

For the example in the question:

xml ed -N x="http://maven.apache.org/POM/4.0.0" \
       -s '/x:project' -t elem -n 'distributionManagement' \
       -s '/x:project/distributionManagement' -t elem -n 'repository' \
       -s '/x:project/distributionManagement/repository' -t elem -n 'id' \
         -v 'private-releases' \
       -s '/x:project/distributionManagement/repository' -t elem -n 'url' \
         -v 'https://my.private.server.com/nexus/repository/maven-releases/' \
    file.xml

Result:

<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>
  <!-- a lot of other tags-->
  <distributionManagement>
    <repository>
      <id>private-releases</id>
      <url>https://my.private.server.com/nexus/repository/maven-releases/</url>
    </repository>
  </distributionManagement>
</project>

Inserting a previously prepared XML file at a location in the XML:

Assuming the original XML from the question is in file.xml and the additional bits that should go in the new distributinManagement node are in new.xml (but not the node tag itself), one could do the following to insert new.xml in the root node:

xml ed -N x="http://maven.apache.org/POM/4.0.0" \
       -s '/x:project' -t elem -n 'distributionManagement' \
       -v "$(<new.xml)" file.xml | xml unesc | xml fo

XMLStarlet will automatically escape data that needs escaping, such as < and > characters. The xml unesc bit unescapes the inserted data (it actually unescapes the whole document, which may or may not be an issue), and xml fo reformats the resulting XML document.

The result is

<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>
  <!-- a lot of other tags-->
  <distributionManagement>
    <repository>
      <id>private-releases</id>
      <url>https://my.private.server.com/nexus/repository/maven-releases/</url>
    </repository>
  </distributionManagement>
</project>

I'm a tiny bit uneasy about doing it this way, "but it works".

See also this related question on StackOverflow: https://stackoverflow.com/questions/29298507/xmlstarlet-xinclude-xslt

  • It looks interesting, although for inserting more than one single tag the syntax is quite long. Only that in ubuntu it is named 'xmlstarlet'. Is it possible to insert the content of other file as tag, assuming that content is a valid xml? – 9ilsdx 9rvj 0lo Feb 9 '18 at 13:52
  • @9ilsdx9rvj0lo See updated answer. – Kusalananda Feb 9 '18 at 17:45
  • "it actually unescapes the whole document, which may or may not be an issue". Yep massive issue, all the existing &amp; were unencoded causing XML to no longer be valid :( – rob Apr 18 at 9:25
1

I find it an overkill to install java, perl or python in OS for that purpose (my scripts are done in gitlab with docker images, so doing my job with tools available in maven:3.5-jdk-8 image would be a dream).

it probably still is overkill, but if you are only concerned with the container's size you could use a very lightweight language such as Lua or Guile.

from the Lua docs:

Adding Lua to an application does not bloat it. The tarball for Lua 5.3.4, which contains source code and documentation, takes 297K compressed and 1.1M uncompressed. The source contains around 24000 lines of C. Under 64-bit Linux, the Lua interpreter built with all standard Lua libraries takes 246K and the Lua library takes 421K.

  • It's worth considering simply adding LUA to maven container, thanks for the tip. – 9ilsdx 9rvj 0lo Feb 12 '18 at 7:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.