1

I have a folder that contains 12,000+ XML files. I need to get a list of files within that folder that meet certain criteria.

In the XML file there is a node called /BillingData/InvoiceLinesList/InvoiceLines. There could be one or more InvoiceLines within the InvoiceLinesList. In the InvoiceLines, I need to search for a tag called <charge> whose value is 99 and also where there is a tag in the same InvoiceLines called <chargeType> where the value is D.

What's the best way to do that? Using awk I thought I might be able to do it, but I couldn't figure a way of searching for multiple conditions as I'm not great at awk. I saw a potential method on here using xmlstarlet but that only looks for one or other value in a single tag rather than separate values in more than one tag.

  • 1
    for xml, use xml tools. do not parse XML/HMTL/json/yaml without the appropriate tool. When you desperately need to hammer a nail and have no hammer, you cn use a heavy object instead but it's uncomfortable and risky. It's the same with parsing XML, if you can use appropriate tool do it, it will be easier safer and the result will only be nicer. – Kiwy Sep 21 '18 at 10:14
2

Both awk and sed are generally unsuitable for parsing XML (and other similar formats such as JSON and YAML etc.). For this XML example, for example, we don't know what order the nodes in InvoiceLines are stored or whether they are delimited by newlines or not. The XML format does not care about these things, but an awk or sed script would fall over easily unless exceptional care was taken to cover all possible cases (including the various possible encodings of the data), in which case you would have had to write an XML parser anyway.

So, using an XML parser, like the one built into xmlstarlet, is the right thing to want to do.


The following command would print the filename of the input file if at least one of the required nodes was found in the file file.xml. If multiple InvoiceLines nodes matched, the filename would be printed multiple times with a newline in-between. This means that we, from the start, disqualify filenames containing newlines.

xmlstarlet sel \
    -t -m '/BillingData/InvoiceLinesList/InvoiceLines[chargeType = "D" and charge = "99"]' \
    --inp-name -nl file.xml

The XPATH query would match all InvoiceLines that had subnodes chargeType and charge with the specified values. Using @charge instead of charge would have tested against a charge attribute in the InvoiceLines node, by the way.

Applying this on all XML files in a single directory:

xmlstarlet sel \
    -t -m '/BillingData/InvoiceLinesList/InvoiceLines[chargeType = "D" and charge = "99"]' \
    --inp-name -nl ./*.xml

If there are too many files and the above throws an error, you may use xargs:

printf '%s\n' ./*.xml | xargs xmlstarlet -t -m ...

Or, find (this will also search subdirectories):

find . -type f -name '*.xml' -exec xmlstarlet -t -m ... {} +

Pipe the result through uniq if you want to make the file listing unique.


I used the following XML to test the above:

<BillingData>
    <InvoiceLinesList>
        <InvoiceLines>
            <chargeType>D</chargeType>
            <charge>99</charge>
        </InvoiceLines>
        <InvoiceLines>
            <chargeType>D</chargeType>
            <charge>99</charge>
        </InvoiceLines>
        <InvoiceLines>
            <chargeType>E</chargeType>
            <charge>99</charge>
        </InvoiceLines>
    </InvoiceLinesList>
</BillingData>
  • Does xmlstarlet still load the whole file in memory before proceding? Good luck with that ;-) – mosvy Sep 21 '18 at 10:58
  • @mosvy Do you know how big the files are? The main part of the answer is the XPATH query. This could be applied with some other framework if needs be. – Kusalananda Sep 21 '18 at 11:06
  • @Kusalananda That looks amazing. The XML structure you used as a test is correct. There are other nodes but their names are different. The XML is well formed so I will give your example a shot and see how it goes. Kudos dude! – snert Sep 21 '18 at 12:16
  • So as a single command I tried xmlstarlet sel \ -t -m '/BillingData/InvoiceLinesList/InvoiceLines[chargeType = "D" and charge = "99"]' \ --inp-name -nl ./*.xml but it gave me an error: error in arguments: no -t or --template options found – snert Sep 21 '18 at 13:32
  • @snert the backslashes ( \ ) escapes the newlines in my command. If you put it on a single line, then remove them. – Kusalananda Sep 21 '18 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.