1

I have two large files consisting mainly of numbers in matrix form, and I'd like to use diff (or similar command) to compare these files and determine which numbers are different.

Unfortunately, quite a lot of these numbers differ only by sign, and I'm not interested in those differences. I only care when two numbers are different in magnitude. (i.e. I want 0.523 vs. 0.623, but NOT 0.523 vs. -0.523)

Is it possible to make diff ignore the sign and only print numbers that are different in magnitude?

EDIT: Some input examples, as requested:

File 1:

21   -0.0081318   0.0000000   0.0000000   0.0000000  -0.0138079
22    0.0000000   0.0000000   0.0000000   0.1156119   0.0000000
23    0.0000000   0.0047536   0.0000000   0.0000000   0.0000000

File 2:

21   -0.0081318   0.0000000   0.0000000   0.0000000   0.0032533
22    0.0000000   0.0000000   0.0000000  -0.0250637   0.0000000
23    0.0000000  -0.0047536   0.0000000   0.0000000   0.0000000

Assuming my files are formated mostly like this (except much, MUCH longer), I want to print out the differences, but ignore such differences when they're only in sign. For example, I don't care for 0.0047536 vs. -0.0047536, but I do want to print 0.1156119 vs. -0.0250637.

  • 2
    What does "mainly in matrix form" mean? An input sample might help, as well as an output sample. – RudiC Oct 3 '18 at 21:20
  • Could you just delete / eliminate all minusses from the input? With tr, or sed? – RudiC Oct 3 '18 at 21:21
  • 2
    Please at least provide examples of input and desired output. Do you only want to know IF they differ or the difference or both values or a combination thereof. How should it be displayed? Also I'd suggest referring to either a mathematics suite (e.g. GNU octave) or a suitable programming language (e.g. FORTRAN) for this task. Especially if it is large files. BTW: diff compares whole files or strings and does so line by line, not word by word. Not the right tool here. – Fiximan Oct 3 '18 at 21:39
  • @RudiC Could you show me how I could eliminate all minuses from the input and still leave the format of the text intact ? – johnymm Oct 3 '18 at 23:27
  • Do you know the matrix sizes? Do they vary? How should the output look like? – Fiximan Oct 3 '18 at 23:36
3

Given your shell provided "process substitution" (likes recent bashes), try

diff <(tr '-' ' ' <file1) <(tr '-' ' '<file2)
1,2c1,2
< 21    0.0081318   0.0000000   0.0000000   0.0000000   0.0138079
< 22    0.0000000   0.0000000   0.0000000   0.1156119   0.0000000
---
> 21    0.0081318   0.0000000   0.0000000   0.0000000   0.0032533
> 22    0.0000000   0.0000000   0.0000000   0.0250637   0.0000000
  • This also works. Thanks! I can't believe it was this easy haha. I should be more familiar with basic Unix commands. – johnymm Oct 4 '18 at 14:45
1
$ xdiff(){ diff -bu <($1 "$2") <($1 "$3"); }
$ xdiff 'sed s/-\([.0-9]\)/\1/g' file1 file2

you can do other normalization to the data. for instance, to treat all of 0.01, .01, -.0100, -.01e as the same:

$ norm(){ awk '{for(i=1;i<=NF;i++){$i=$i<0?-$i:+$i};print}' "$@"; }
$ xdiff norm file1 file2
  • This approach seems to delete the minuses, but also displaces the entire line to the left - so diff still displays the line. Is it possible to replace the the minus with an empty space, so the format remains the same ? – johnymm Oct 4 '18 at 0:52
  • 1
    the -b option given to diff should tell it to treat any amount of whitespace as the same, so it shouldn't display the line. but yes, you can replace - by a space by adding a space before the \1 (eg sed s/../\ \1/g). – mosvy Oct 4 '18 at 0:58
  • I'm not on my computer right now but I'll try it as soon as I can and let you know if it works. – johnymm Oct 4 '18 at 1:00
  • When I add a space before "\1" it gives me the error: sed: -e expression #1, char 14: unterminated `s' command – johnymm Oct 4 '18 at 4:47
  • yes, it will be split with IFS. it'll work if you put it in a function though: norm(){ sed 's/-/ /g' "$@"; }; xdiff norm file1 file2. rather than extend that minimal xdiff example, you could use the script from my answer here like this: x 'diff -bu' "sed 's/-/ /g'" file1 file2 – mosvy Oct 4 '18 at 4:57
0

To print lines from file2 whose corresponding absolute values differ from the corresponding fields from the same line in file1, save the lines from file1 in memory, the compare their absolute values:

function abs(x) {
  return x < 0 ? -x : x;
}

# file 1
NR == FNR {
  file1[$1]=$2" "$3" "$4" "$5" "$6
}

# file 2
NR != FNR {
  split(file1[$1], file1fields);
  if ( abs($2) - abs(file1fields[1]) ||
       abs($3) - abs(file1fields[2]) ||
       abs($4) - abs(file1fields[3]) ||
       abs($5) - abs(file1fields[4]) ||
       abs($6) - abs(file1fields[5]) )
        print;
}

Save that in a file, then run awk -f /path/to/that/file file1 file2

  • I'm not sure what you mean by "save the lines from file1 in memory" – johnymm Oct 4 '18 at 4:51
  • @johnymm it means file1 is completely saved to RAM, then file2 is read (more or less) line by line and the corresponding lines are compared. If file1 exceeds your PC's memory, you will have to split the files beforehand. – Fiximan Oct 4 '18 at 10:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.