102

I have some sql dumps that I am looking at the differences between. diff can obviously show me the difference between two lines, but I'm driving myself nuts trying to find which values in the long list of comma-separated values are actually the ones causing the lines to be different.

What tool can I use to point out the exact character differences between two lines in certain files?

12 Answers 12

85

There's wdiff, the word-diff for that.

On desktop, meld can highlight the differences within a line for you.

  • 7
    Colored wdiff: wdiff -w "$(tput bold;tput setaf 1)" -x "$(tput sgr0)" -y "$(tput bold;tput setaf 2)" -z "$(tput sgr0)" file1 file2 – l0b0 Apr 12 '11 at 11:21
  • 40
    For color, install colordiff, then do: wdiff a b | colordiff – philfreo Sep 28 '13 at 3:36
  • Meld actually is extremely slow (minutes) at showing the intra-line differences between to line-based files. – Dan Dascalescu Jun 28 '17 at 22:02
  • There is also dwdiff tool which is mostly compatible with wdiff but also supports colored output and probably some other features. And it is more available in some Linux distributions like Arch. – MarSoft Aug 31 '17 at 0:13
  • 3
    wdiff -n a b | colordiff, advises man colordiff. – Camille Goudeseune May 4 '18 at 17:50
23

I've used vimdiff for this.

Here's a screenshot (not mine) showing minor one or two character differences that stands out pretty well. A quick tutorial too.

  • In my case couldn't spot the difference so opened the files in gvim -d f1 f2 the particular long lines were both highlighted as being different however the actual difference was extra highlighted in red – zzapper Dec 9 '15 at 16:54
  • I've been using vim forever, but had no idea about vimdiff! – mitchus Apr 3 '17 at 16:02
  • And there is diffchar.vim for character-level diffs. – Keelan Nov 16 '17 at 23:52
  • 2
    As much as I love vim and vimdiff, vimdiff's algorithm for highlighting differences in a line is pretty basic. It seems to just strip out the common prefix and suffix, and highlight everything between as different. This works if all of the characters that changed are grouped together, but if they are spread out it doesn't work well. It's also terrible for word-wrapped text. – Laurence Gonsalves Dec 21 '17 at 20:49
  • For long lines as in the OP, vimdiff -c 'set wrap' -c 'wincmd w' -c 'set wrap' a b, suggests stackoverflow.com/a/45333535/2097284. – Camille Goudeseune May 4 '18 at 17:54
21

Just another method using git-diff:

git diff -U0 --word-diff --no-index -- foo bar | grep -v ^@@

grep -v if not interested in positions of the diffs.

  • 2
    This is exactly the behaviour I was trying to mimic -- didn't realize I could use git-diff without one of the files being indexed. – spinup Oct 10 '17 at 0:27
  • 1
    --word-diff is the key option here. Thanks! – user2707671 Aug 17 '18 at 13:04
  • 1
    --no-index is only required if you're in a git working directory and both foo and bar are as well. – xn. Feb 12 at 21:00
6

Here is a "..hair of the dog that bit you" method...
diff got you to this point; use it to take you further...

Here is the output from using the sample line pairs... indicates a TAB

Paris in the     spring 
Paris in the the spring 
             vvvv      ^

A ca t on a hot tin roof.
a cant on a hot  in roof 
║   v           ^       ^

the quikc brown box jupps ober the laze dogs 
The☻qui ckbrown fox jumps over the lazy dogs 
║  ║   ^ ║      ║     ║    ║          ║     ^

Here is the script.. You just need to ferret out the line pairs somehow.. (I've used diff only once (twice?) before today, so I don't know its many options, and sorting out the options for this script was enough for me, for one day :) .. I think it must be simple enough, but I'm due for a coffee break ....

#
# Name: hair-of-the-diff
# Note: This script hasn't been extensively tested, so beware the alpha bug :) 
#   
# Brief: Uses 'diff' to identify the differences between two lines of text
#        $1 is a filename of a file which contains line pairs to be processed
#
#        If $1 is null "", then the sample pairs are processed (see below: Paris in the spring 
#          
# ║ = changed character
# ^ = exists if first line, but not in second 
# v = exists if second line, but not in first

bname="$(basename "$0")"
workd="/tmp/$USER/$bname"; [[ ! -d "$workd" ]] && mkdir -p "$workd"

# Use $1 as the input file-name, else use this Test-data
# Note: this test loop expands \t \n etc ...(my editor auto converts \t to spaces) 
if [[ "$1" == '' ]] ;then
  ifile="$workd/ifile"
{ while IFS= read -r line ;do echo -e "$line" ;done <<EOF
Paris in the spring 
Paris in the the spring
A cat on a hot tin roof.
a cant on a hot in roof
the quikc brown box jupps ober the laze dogs 
The\tquickbrown fox jumps over the lazy dogs
EOF
} >"$ifile"
else
  ifile="$1"
fi
#
[[ -f "$ifile" ]] || { echo "ERROR: Input file NOT found:" ;echo "$ifile" ;exit 1 ; }
#  
# Check for balanced pairs of lines
ilct=$(<"$ifile" wc -l)
((ilct%2==0)) || { echo "ERROR: Uneven number of lines ($ilct) in the input." ;exit 2 ; }
#
ifs="$IFS" ;IFS=$'\n' ;set -f
ix=0 ;left=0 ;right=1
while IFS= read -r line ;do
  pair[ix]="$line" ;((ix++))
  if ((ix%2==0)) ;then
    # Change \x20 to \x02 to simplify parsing diff's output,
    #+   then change \x02 back to \x20 for the final output. 
    # Change \x09 to \x01 to simplify parsing diff's output, 
    #+   then change \x01 into ☻ U+263B (BLACK SMILING FACE) 
    #+   to the keep the final display columns in line. 
    #+   '☻' is hopefully unique and obvious enough (otherwise change it) 
    diff --text -yt -W 19  \
         <(echo "${pair[0]}" |sed -e "s/\x09/\x01/g" -e "s/\x20/\x02/g" -e "s/\(.\)/\1\n/g") \
         <(echo "${pair[1]}" |sed -e "s/\x09/\x01/g" -e "s/\x20/\x02/g" -e "s/\(.\)/\1\n/g") \
     |sed -e "s/\x01/☻/g" -e "s/\x02/ /g" \
     |sed -e "s/^\(.\) *\x3C$/\1 \x3C  /g" \
     |sed -n "s/\(.\) *\(.\) \(.\)$/\1\2\3/p" \
     >"$workd/out"
     # (gedit "$workd/out" &)
     <"$workd/out" sed -e "s/^\(.\)..$/\1/" |tr -d '\n' ;echo
     <"$workd/out" sed -e "s/^..\(.\)$/\1/" |tr -d '\n' ;echo
     <"$workd/out" sed -e "s/^.\(.\).$/\1/" -e "s/|/║/" -e "s/</^/" -e "s/>/v/" |tr -d '\n' ;echo
    echo
    ((ix=0))
  fi
done <"$ifile"
IFS="$ifs" ;set +f
exit
#
4

wdiff is actually a very old method of comparing files word-by-word. It worked by reformatting files, then using diff to find differences and passing it back again. I myself suggested adding context, so that rather than word-by-word compare, it does it with each word surrounded by other 'context' words. That allows the diff to synchronise itself on common passages in files much better, especially when files are mostly different with only a few blocks of common words. For example when comparing text for for plagiarism, or re-use.

dwdiff was later created from wdiff. But dwdiff uses that text reformatting function to good effect in dwfilter. This is a great development – it means you can reformat one text to match another, and then compare them using any line-by-line graphical diff displayer. For example, using it with "diffuse" graphical diff....

dwfilter file1 file2 diffuse -w

This reformats file1 to the format of file2 and gives that to diffuse for a visual comparison. file2 is unmodified, so you can edit and merge word differences into it directly in diffuse. If you want to edit file1, you can add -r to reverse which file is reformatted. Try it and you will find it is extremely powerful!

My preference for the graphical diff (shown above) is diffuse as it feels far cleaner and more useful. Also it is a standalone python program, which means it is easy to install and distribute to other UNIX systems.

Other graphical diffs seem to have a lot of dependancies, but can also be used (you choice). These include kdiff3 or xxdiff.

4

Using @Peter.O's solution as a basis I rewrote it to make a number of changes.

enter image description here

  • It only prints every line once, using colour to show you the differences.
  • It doesn't write any temp files, piping everything instead.
  • You can provide two filenames and it'll compare the corresponding lines in each file. ./hairOfTheDiff.sh file1.txt file2.txt
  • Otherwise, if you use the original format (a single file with every second line needing to be compared to the one before) you may now simply pipe it in, no file needs to exist to be read. Take a look at demo in the source; this may open the door to fancy piping in order to not need files for two separate inputs too, using paste and multiple file-descriptors.

No highlight means the character was in both lines, highlight means it was in the first, and red means it was in the second.

The colours are changeable through variables at the top of the script and you can even forego colours entirely by using normal characters to express differences.

#!/bin/bash

same='-' #unchanged
up='△' #exists in first line, but not in second 
down='▽' #exists in second line, but not in first
reset=''

reset=$'\e[0m'
same=$reset
up=$reset$'\e[1m\e[7m'
down=$reset$'\e[1m\e[7m\e[31m'

timeout=1


if [[ "$1" != '' ]]
then
    paste -d'\n' "$1" "$2" | "$0"
    exit
fi

function demo {
    "$0" <<EOF
Paris in the spring 
Paris in the the spring
A cat on a hot tin roof.
a cant on a hot in roof
the quikc brown box jupps ober the laze dogs 
The quickbrown fox jumps over the lazy dogs
EOF
}

# Change \x20 to \x02 to simplify parsing diff's output,
#+   then change \x02 back to \x20 for the final output. 
# Change \x09 to \x01 to simplify parsing diff's output, 
#+   then change \x01 into → U+1F143 (Squared Latin Capital Letter T)
function input {
    sed \
        -e "s/\x09/\x01/g" \
        -e "s/\x20/\x02/g" \
        -e "s/\(.\)/\1\n/g"
}
function output {
    sed -n \
        -e "s/\x01/→/g" \
        -e "s/\x02/ /g" \
        -e "s/^\(.\) *\x3C$/\1 \x3C  /g" \
        -e "s/\(.\) *\(.\) \(.\)$/\1\2\3/p"
}

ifs="$IFS"
IFS=$'\n'
demo=true

while IFS= read -t "$timeout" -r a
do
    demo=false
    IFS= read -t "$timeout" -r b
    if [[ $? -ne 0 ]]
    then
        echo 'No corresponding line to compare with' > /dev/stderr
        exit 1
    fi

    diff --text -yt -W 19  \
        <(echo "$a" | input) \
        <(echo "$b" | input) \
    | \
    output | \
    {
        type=''
        buf=''
        while read -r line
        do
            if [[ "${line:1:1}" != "$type" ]]
            then
                if [[ "$type" = '|' ]]
                then
                    type='>'
                    echo -n "$down$buf"
                    buf=''
                fi

                if [[ "${line:1:1}" != "$type" ]]
                then
                    type="${line:1:1}"

                    echo -n "$type" \
                        | sed \
                            -e "s/[<|]/$up/" \
                            -e "s/>/$down/" \
                            -e "s/ /$same/"
                fi
            fi

            case "$type" in
            '|')
                buf="$buf${line:2:1}"
                echo -n "${line:0:1}"
                ;;
            '>')
                echo -n "${line:2:1}"
                ;;
            *)
                echo -n "${line:0:1}"
                ;;
            esac
        done

        if [[ "$type" = '|' ]]
        then
            echo -n "$down$buf"
        fi
    }

    echo -e "$reset"
done

IFS="$ifs"

if $demo
then
    demo
fi
3

Here's a simple one-liner:

diff -y <(cat a.txt | sed -e 's/,/\n/g') <(cat b.txt | sed -e 's/,/\n/g')

The idea is to replace commas (or whichever delimiter you wish to use) with newlines using sed. diff then takes care of the rest.

2

If I'm reading your question correctly, I use diff -y for this kind of thing.

It makes comparing a side by side comparison much simpler to find which lines are throwing the differences.

  • This does not highlight the difference within the line. If you have a long line, it's painfull to see the difference. wdiff, git diff --word-diff, vimgit, meld, kbdiff3, tkdiff all do this. – user2707671 Aug 17 '18 at 13:07
2
  • xxdiff: Another tool is xxdiff (GUI), which has to be installed, first.
  • spreadsheet: For database data, a spreadsheet from .csv is easily made, and a formula (A7==K7) ? "" : "diff" or similar inserted, and copy-pasted.
  • 1
    xxdiff looks like the 80's. Meld looks much better but it's extremely slow for CSV-like files. I've found Diffuse to be the fastest Linux diff tool. – Dan Dascalescu Jun 28 '17 at 22:01
  • @DanDascalescu: A tool which gets the job done looks always fine, no matter how old it looks. Another one, I used occasionally, but isn't installed to test it with long, column data, is tkdiff. – user unknown Jun 29 '17 at 3:08
  • Does xxdiff display moved lines? Or does it just show a missing line in one file and an added one in the other? (I tried building xxdiff but qmake failed and I see they don't bother to publish a Debian package). – Dan Dascalescu Jun 29 '17 at 3:25
  • @DanDascalescu: Today, I only have tkdiff installed. – user unknown Jun 30 '17 at 0:37
1

On the command line, I would make sure I add judicious new-lines before comparing files. You can use sed, awk, perl or anything really to add line breaks in some sort of systematic way - make sure not to add too many though.

But I find the best is to use vim as it highlights word differences. vim is good if there are not too many differences and the differences are simple.

  • Although not really an answer to the question this technique is rather efficient to learn about small differences in long lines. – jknappen Feb 4 '15 at 19:05
1

kdiff3 is becoming the standard GUI diff viewer on Linux. It is similar to xxdiff, but I think kdiff3 is better. It does many things well, including your request to show "exact character differences between two lines in certain files".

  • KDiff3 is extremely slow to highlight the inline differences in CSV files. I wouldn't recommend it. – Dan Dascalescu Jun 28 '17 at 22:00
1

I had the same problem and solved it with PHP Fine Diff, an online tool that allows you to specify granularity. I know it's not technically a *nix tool, but I didn't really want to download a program just to do a one-time, character level diff.

  • Some users can't upload sensitive or large files to a random online tool. There are plenty of tools that show line-level differences without compromising your privacy. – Dan Dascalescu Jun 28 '17 at 22:01
  • Yes, there are. But for diffs that don't contain sensitive information, online tools can be a good solution. – pillravi Jul 3 '17 at 14:01
  • Online diff tools also don't support command line integration. You can't use them from your version control flow. They're also far more cumbersome to use (select file 1, select file 2, upload) and can't do merging. – Dan Dascalescu Jul 4 '17 at 18:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.