5

Suppose that I have the two files with the following content:

$ cat File1.txt
Apple
orange
watermelon
avocado
lime
$ cat File2.txt
orange
Apple
lime 
watermelon
avocado

Basically, there is no difference, as both have same values. I am using the diff command:

diff File1.txt File2.txt

and it shows files are different as values are misplaced, In my case I require it should have not showed difference. What are the other ways to achieve this, any suggestions are welcome.

4
14

Compare the sorted files.

In bash (or ksh or zsh), with a process substitution:

diff <(sort File1.txt) <(sort File2.txt)

In plain sh:

sort File1.txt >File1.txt.sorted
sort File1.txt >File2.txt.sorted
diff File1.txt.sorted File2.txt.sorted

To quickly see the differences between sorted files, comm can be useful: it shows directly the lines that are in one file but not the other.

comm -12  <(sort File1.txt) <(sort File2.txt) >common-lines.txt
comm -23  <(sort File1.txt) <(sort File2.txt) >only-in-file-1.txt
comm -13  <(sort File1.txt) <(sort File2.txt) >only-in-file-2.txt

If a line is repeated in the same file, the commands above insist on the two files having the same number of repetitions. If you want to treat

foo
bar
foo

as identical to

bar
foo

then remove duplicates when sorting: use sort -u instead of sort.

If you save the output of sort on one file and use it later when the other file is available, note that the two files must be sorted in the same locale. If you do this, you should probably sort in byte order:

LC_ALL=C sort File1.txt >File1.txt.sorted
1
  • 1
    Beware that on some systems, including GNU ones, the output of sort is not deterministic outside the C/POSIX locale, so you could get false positives, so it's best to use LC_ALL=C for all invocations of sort and comm (that would also make it much more efficient). Jun 26 '20 at 9:57
5

Sort the files first (in bash):

diff <(sort file1) <(sort file2)
4

Using awk, you can make a hash index of every distinct input line text, using a command like:

awk 'The magic' Q=A fileA Q=B fileB Q=C fileC ...

'The magic' per input line is:

{ X[$0] = X[$0] Q; }

When you get to the END condition, you iterate over the index of X. Any line that occurred exactly once in each file will be like:

X["Apple"] = "ABC";

A line that appeared once in fileA and three times in fileC would present as "ACCC". You can report any anomalies any way you like, for any number of files. (I once had to implement a 14-way comparison on a safety-critical system that ran on a Main and Standby server, each with a real-time plus Oracle database.)

If you choose to include the line number NR on each tag, and write some interesting patterns, you can make the tags look like:

X["Walrus"] = "A347B38C90"

and report which matching texts were on which lines in the various files.

2
  • See also the ARGIND of GNU awk to avoid having to pass those Q=... arguments. Jun 26 '20 at 6:33
  • While sorting is the standard approach, this way should be more efficient, at least for large files, as it does a simple linear scan through each file without the O(n log n) complexity of a sort.
    – gidds
    Jun 26 '20 at 8:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.