Structured data format to XML

Question

I have some output in some structured data format that I need to convert into XML.

My use case is that Palo Alto firewalls deliver configuration output via the command line interface which cannot be used for restoring the configuration. It needs to be converted into XML to be able to use it.

The format is (just a snippet of the huge file to get the idea):

unknown-applications {
  unknown-tcp {
    destinations-per-hour 10;
    sessions-per-hour 10;
    session-length {
      maximum-bytes 100;
      minimum-bytes 50;
    }
  }
  unknown-udp {
    destinations-per-hour 10;
    sessions-per-hour 10;
    session-length {
      maximum-bytes 100;
      minimum-bytes 50;
    }
  }
}

and needs to be

<unknown-applications>
  <unknown-tcp>
    <destinations-per-hour>10</destinations-per-hour>
    <sessions-per-hour>10</sessions-per-hour>
    <session-length>
      <maximum-bytes>100</maximum-bytes>
      <minimum-bytes>50</minimum-bytes>
    </session-length>
  </unknown-tcp>
  <unknown-udp>
    <destinations-per-hour>10</destinations-per-hour>
    <sessions-per-hour>10</sessions-per-hour>
    <session-length>
      <maximum-bytes>100</maximum-bytes>
      <minimum-bytes>50</minimum-bytes>
    </session-length>
  </unknown-udp>
</unknown-applications>

Is there something like a parser to achieve this?

EDIT: as comments didnt allow me to post this much: thanks, looks promising!

some parts are not converted right because i didnt list an example for it: e.g.

          import {
            network {
              interface [ ethernet1/5 ethernet1/6];
            }
          }

gets converted to

<import>
<network>
<interface>[</interface>
</network>
</import>

but should be

         <import>
            <network>
              <interface>
                <member>ethernet1/5</member>
                <member>ethernet1/6</member>
              </interface>
            </network>
          </import>

i see that there are some specific things like [] making it members, so i am not sure how feasible it is to do it manually... the original format is supposed to be in xml as well

Answer 1

Since your input format is non-standard, you need to write a parser to handle it. There are two approaches: you can write it "by hand" in any procedural programming language (typically as a top-down recursive descent parser), or you can use some kind of parser generator. In the latter approach you define the BNF of the grammar of your input and the tool constructs the parser for you. If you can use Scala, it makes this very easy.

Another parser generator often used in the XML world is REx - it's handy because it will generate a parser in XQuery or XSLT, which makes the XML generation side of the problem very easy; it's a great piece of software but sadly it's very poorly documented. Another XML-oriented tool is Stephen Pemberton's "Invisible XML" (https://homepages.cwi.nl/~steven/ixml/) -- if REx is great software with poor documentation, then Invisible XML, sadly, is great documentation with not much published software.

Another cheap-and-cheerful approach occurs to me: with a fairly simple editor script based on regular expressions, you could:

• put names in quotes and add a terminating colon
• replace semicolons by commas

and then you would have something that many JSON parsers would accept (it's not quite conformant, because your semicolons are terminators rather than separators, but many JSON parsers tolerate this.)

Answer 2

The following is a simple awk program that assumes that the data looks exactly like what you are showing, i.e. that

1. the opening "tags" are on lines that only contains a label and a {,
2. lines with data on them contains exactly a label followed by a single word and a ;, and that
3. each } occurs on lines with nothing else on them.

It uses a stack to keep track of what tag it's currently working with, and pushes new tags onto the stack when they occur in the input. When a } is found, the end tag for the top of the stack is outputted. Data lines are detected by the fact that they don't end with either a lone { or } character.

No attempt is made to validate the data in any way.

$NF == "{" {
        stack[++top] = $1                       # push tag to stack
        printf "<%s>\n", stack[top]             # output opening tag
        next
}

$NF == "}" {
        printf "</%s>\n", stack[top--]          # output closing tag + pop stack
        next
}

{
        sub(";$", "", $2)                       # remove ; from EOL
        printf "<%s>%s</%s>\n", $1, $2, $1      # output tag with data
}

For your example data, awk -f script.awk data.in produces

<unknown-applications>
<unknown-tcp>
<destinations-per-hour>10</destinations-per-hour>
<sessions-per-hour>10</sessions-per-hour>
<session-length>
<maximum-bytes>100</maximum-bytes>
<minimum-bytes>50</minimum-bytes>
</session-length>
</unknown-tcp>
<unknown-udp>
<destinations-per-hour>10</destinations-per-hour>
<sessions-per-hour>10</sessions-per-hour>
<session-length>
<maximum-bytes>100</maximum-bytes>
<minimum-bytes>50</minimum-bytes>
</session-length>
</unknown-udp>
</unknown-applications>

which, when run through xmlstarlet fo, gets transformed into

<?xml version="1.0"?>
<unknown-applications>
  <unknown-tcp>
    <destinations-per-hour>10</destinations-per-hour>
    <sessions-per-hour>10</sessions-per-hour>
    <session-length>
      <maximum-bytes>100</maximum-bytes>
      <minimum-bytes>50</minimum-bytes>
    </session-length>
  </unknown-tcp>
  <unknown-udp>
    <destinations-per-hour>10</destinations-per-hour>
    <sessions-per-hour>10</sessions-per-hour>
    <session-length>
      <maximum-bytes>100</maximum-bytes>
      <minimum-bytes>50</minimum-bytes>
    </session-length>
  </unknown-udp>
</unknown-applications>