how to compare two xml files having same data in different lines?

Question

I have two files have same data but in different lines.

File 1:

<Identities>
    <Identity>
        <Id>048206031415072010Comcast.USR8JR</Id>
        <UID>ccp_test_79</UID>
        <DisplayName>JOSH CCP</DisplayName>
        <FirstName>JOSH</FirstName>
        <LastName>CCP</LastName>
        <Role>P</Role>
        <LoginStatus>C</LoginStatus>
    </Identity>
    <Identity>
        <Id>089612381523032011Comcast.USR1JR</Id>
        <UID>94701_account1</UID>
        <DisplayName>account1</DisplayName>
        <FirstName>account1</FirstName>
        <LastName>94701</LastName>
        <Role>S</Role>
        <LoginStatus>C</LoginStatus>
    </Identity>
</Identities>

File 2 :

<Identities>
    <Identity>
        <Id>089612381523032011Comcast.USR1JR</Id>
        <UID>94701_account1</UID>
        <DisplayName>account1</DisplayName>
        <FirstName>account1</FirstName>
        <LastName>94701</LastName>
        <Role>S</Role>
        <LoginStatus>C</LoginStatus>
    </Identity>
    <Identity>
        <Id>048206031415072010Comcast.USR8JR</Id>
        <UID>ccp_test_79</UID>
        <DisplayName>JOSH CCP</DisplayName>
        <FirstName>JOSH</FirstName>
        <LastName>CCP</LastName>
        <Role>P</Role>
        <LoginStatus>C</LoginStatus>
    </Identity>
</Identities>

If I use diff file1 file2 command I am getting below response:

1,10d0
<     <Identities>
<         <Identity>
<             <Id>048206031415072010Comcast.USR8JR</Id>
<             <UID>ccp_test_79</UID>
<             <DisplayName>JOSH CCP</DisplayName>
<             <FirstName>JOSH</FirstName>
<             <LastName>CCP</LastName>
<             <Role>P</Role>
<             <LoginStatus>C</LoginStatus>
<         </Identity>
20a11,20
>     <Identities>
>         <Identity>
>             <Id>048206031415072010Comcast.USR8JR</Id>
>             <UID>ccp_test_79</UID>
>             <DisplayName>JOSH CCP</DisplayName>
>             <FirstName>JOSH</FirstName>
>             <LastName>CCP</LastName>
>             <Role>P</Role>
>             <LoginStatus>C</LoginStatus>
>         </Identity>

But I need to get no difference, because these files having same data in different lines.

Answer 1

You can achieve what you want with the help of a small Python script (you'll need Python installed, as well as the lxml toolkit).

tagsort.py:

#!/usr/bin/python

import sys
from lxml import etree

filename, tag = sys.argv[1:]

doc = etree.parse(filename, etree.XMLParser(remove_blank_text=True))
root = doc.getroot()
root[:] = sorted(root, key=lambda el: el.findtext(tag))
print etree.tostring(doc, pretty_print=True)

This script sorts the first-level elements under the XML document root by the content of a second-level element, sending the result to stdout. It's called like this:

$ python tagsort.py filename tag

Once you've got that, you can use process substitution to get a diff based on its output (I've added one element and changed another in your example files to show a non-empty result):

$ diff <(python tagsort.py file1 Id) <(python tagsort.py file2 Id)
4a5
>     <AddedTag>Something</AddedTag>
17c18
<     <Role>X</Role>
---
>     <Role>S</Role>

Answer 2

I had a similar problem and I eventually found: https://superuser.com/questions/79920/how-can-i-diff-two-xml-files

That post suggests doing a canonical xml sort then doing a diff. The following should work for you if you are on linux, mac, or if you have windows something like cygwin installed:

$ xmllint --c14n File1.xml > 1.xml
$ xmllint --c14n File2.xml > 2.xml
$ diff 1.xml 2.xml

Answer 3

It's tagged shell, but honestly I prefer using a scripting language with a parser. In this case perl with XML::Twig.

It goes something like this:

#!/usr/bin/env perl
use strict;
use warnings;

use XML::Twig;

sub compare_by_identity {
   my ( $first, $second ) = @_;
   foreach my $identity ( $first->get_xpath('//Identity') ) {
      my $id = $identity->first_child_text('Id');

      print $id, "\n";
      my $compare_to =
        $second->get_xpath( "//Identity/Id[string()=\"$id\"]/..", 0 );
      if ($compare_to) {
         print "Matching element found for ID $id\n";
         foreach my $element ( $identity->children ) {
            my $tag  = $element->tag;
            my $text = $element->text;
            if ( not $element->text eq $compare_to->first_child_text($tag) ) {
               print "$id, $tag has value $text which doesn't match: ",
                 $compare_to->first_child_text($tag), "\n";
            }
         }
      }
      else {
         print "No matching element for Id $id\n";
      }
   }
}

my $first_file  = XML::Twig->new->parsefile('test1.xml');
my $second_file = XML::Twig->new->parsefile('test2.xml');

compare_by_identity( $first_file,  $second_file );
compare_by_identity( $second_file, $first_file );

I'm explicitly comparing one 'Identity' element at a time, and checking that all the fields in one, exist in the other, with the same value.

And then reversing that, because the second file might have extra entries.

Answer 4

Using xq from https://kislyuk.github.io/yq/:

xq -sS '.[].Identities.Identity |= sort | .[0] == .[1]' file1.xml file2.xml

This reads in both XML files and translates them into JSON. Each file appears as separate elements of the top-level array, so in the example above, file1.xmlwill be found in .[0] and file2.xml will be found in .[1] (this is done courtesy of the -s option). The keys of each file will be automatically sorted due to the -S option.

The identities of each document are sorted and then the two documents are compared.

The output will be the string true for the data provided in the question, or false if a single change is made to either document.

You may use this in a shell test like so:

if xq -e -sS '.[].Identities.Identity |= sort | .[0] == .[1]' file1.xml file2.xml >/dev/null
then
    echo 'documents are the same'
else
    echo 'documents are different'
fi

The -e option causes the exit status of xq to be zero if the last statement has a true boolean value.

---

For reference, the JSON document created internally for the two files in the question will look like this:

[
  {
    "Identities": {
      "Identity": [
        {
          "DisplayName": "JOSH CCP",
          "FirstName": "JOSH",
          "Id": "48206031415072010Comcast.USR8JR",
          "LastName": "CCP",
          "LoginStatus": "C",
          "Role": "P",
          "UID": "ccp_test_79"
        },
        {
          "DisplayName": "account1",
          "FirstName": "account1",
          "Id": "089612381523032011Comcast.USR1JR",
          "LastName": "94701",
          "LoginStatus": "C",
          "Role": "S",
          "UID": "94701_account1"
        }
      ]
    }
  },
  {
    "Identities": {
      "Identity": [
        {
          "DisplayName": "account1",
          "FirstName": "account1",
          "Id": "089612381523032011Comcast.USR1JR",
          "LastName": "94701",
          "LoginStatus": "C",
          "Role": "S",
          "UID": "94701_account1"
        },
        {
          "DisplayName": "JOSH CCP",
          "FirstName": "JOSH",
          "Id": "048206031415072010Comcast.USR8JR",
          "LastName": "CCP",
          "LoginStatus": "C",
          "Role": "P",
          "UID": "ccp_test_79"
        }
      ]
    }
  }
]