diff within a line

Question

I have some sql dumps that I am looking at the differences between. diff can obviously show me the difference between two lines, but I'm driving myself nuts trying to find which values in the long list of comma-separated values are actually the ones causing the lines to be different.

What tool can I use to point out the exact character differences between two lines in certain files?

Answer 1

There's wdiff, the word-diff for that.

On desktop, meld can highlight the differences within a line for you.

Answer 2

Just another method using git-diff:

git diff -U0 --word-diff --no-index -- foo bar | grep -v ^@@

grep -v if not interested in positions of the diffs.

Answer 3

I've used vimdiff for this.

Here's a screenshot (not mine) showing minor one or two character differences that stands out pretty well. A quick tutorial too.

Answer 4

Here is a "..hair of the dog that bit you" method...
diff got you to this point; use it to take you further...

Here is the output from using the sample line pairs... ☻ indicates a TAB

Paris in the     spring 
Paris in the the spring 
             vvvv      ^

A ca t on a hot tin roof.
a cant on a hot  in roof 
║   v           ^       ^

the quikc brown box jupps ober the laze dogs 
The☻qui ckbrown fox jumps over the lazy dogs 
║  ║   ^ ║      ║     ║    ║          ║     ^

---

Here is the script.. You just need to ferret out the line pairs somehow.. (I've used diff only once (twice?) before today, so I don't know its many options, and sorting out the options for this script was enough for me, for one day :) .. I think it must be simple enough, but I'm due for a coffee break ....

#
# Name: hair-of-the-diff
# Note: This script hasn't been extensively tested, so beware the alpha bug :) 
#   
# Brief: Uses 'diff' to identify the differences between two lines of text
#        $1 is a filename of a file which contains line pairs to be processed
#
#        If $1 is null "", then the sample pairs are processed (see below: Paris in the spring 
#          
# ║ = changed character
# ^ = exists if first line, but not in second 
# v = exists if second line, but not in first

bname="$(basename "$0")"
workd="/tmp/$USER/$bname"; [[ ! -d "$workd" ]] && mkdir -p "$workd"

# Use $1 as the input file-name, else use this Test-data
# Note: this test loop expands \t \n etc ...(my editor auto converts \t to spaces) 
if [[ "$1" == '' ]] ;then
  ifile="$workd/ifile"
{ while IFS= read -r line ;do echo -e "$line" ;done <<EOF
Paris in the spring 
Paris in the the spring
A cat on a hot tin roof.
a cant on a hot in roof
the quikc brown box jupps ober the laze dogs 
The\tquickbrown fox jumps over the lazy dogs
EOF
} >"$ifile"
else
  ifile="$1"
fi
#
[[ -f "$ifile" ]] || { echo "ERROR: Input file NOT found:" ;echo "$ifile" ;exit 1 ; }
#  
# Check for balanced pairs of lines
ilct=$(<"$ifile" wc -l)
((ilct%2==0)) || { echo "ERROR: Uneven number of lines ($ilct) in the input." ;exit 2 ; }
#
ifs="$IFS" ;IFS=$'\n' ;set -f
ix=0 ;left=0 ;right=1
while IFS= read -r line ;do
  pair[ix]="$line" ;((ix++))
  if ((ix%2==0)) ;then
    # Change \x20 to \x02 to simplify parsing diff's output,
    #+   then change \x02 back to \x20 for the final output. 
    # Change \x09 to \x01 to simplify parsing diff's output, 
    #+   then change \x01 into ☻ U+263B (BLACK SMILING FACE) 
    #+   to the keep the final display columns in line. 
    #+   '☻' is hopefully unique and obvious enough (otherwise change it) 
    diff --text -yt -W 19  \
         <(echo "${pair[0]}" |sed -e "s/\x09/\x01/g" -e "s/\x20/\x02/g" -e "s/\(.\)/\1\n/g") \
         <(echo "${pair[1]}" |sed -e "s/\x09/\x01/g" -e "s/\x20/\x02/g" -e "s/\(.\)/\1\n/g") \
     |sed -e "s/\x01/☻/g" -e "s/\x02/ /g" \
     |sed -e "s/^\(.\) *\x3C$/\1 \x3C  /g" \
     |sed -n "s/\(.\) *\(.\) \(.\)$/\1\2\3/p" \
     >"$workd/out"
     # (gedit "$workd/out" &)
     <"$workd/out" sed -e "s/^\(.\)..$/\1/" |tr -d '\n' ;echo
     <"$workd/out" sed -e "s/^..\(.\)$/\1/" |tr -d '\n' ;echo
     <"$workd/out" sed -e "s/^.\(.\).$/\1/" -e "s/|/║/" -e "s/</^/" -e "s/>/v/" |tr -d '\n' ;echo
    echo
    ((ix=0))
  fi
done <"$ifile"
IFS="$ifs" ;set +f
exit
#

Answer 5

Using @Peter.O's solution as a basis I rewrote it to make a number of changes.

• It only prints every line once, using colour to show you the differences.
• It doesn't write any temp files, piping everything instead.
• You can provide two filenames and it'll compare the corresponding lines in each file. ./hairOfTheDiff.sh file1.txt file2.txt
• Otherwise, if you use the original format (a single file with every second line needing to be compared to the one before) you may now simply pipe it in, no file needs to exist to be read. Take a look at demo in the source; this may open the door to fancy piping in order to not need files for two separate inputs too, using paste and multiple file-descriptors.

No highlight means the character was in both lines, highlight means it was in the first, and red means it was in the second.

The colours are changeable through variables at the top of the script and you can even forego colours entirely by using normal characters to express differences.

#!/bin/bash

same='-' #unchanged
up='△' #exists in first line, but not in second 
down='▽' #exists in second line, but not in first
reset=''

reset=$'\e[0m'
same=$reset
up=$reset$'\e[1m\e[7m'
down=$reset$'\e[1m\e[7m\e[31m'

timeout=1


if [[ "$1" != '' ]]
then
    paste -d'\n' "$1" "$2" | "$0"
    exit
fi

function demo {
    "$0" <<EOF
Paris in the spring 
Paris in the the spring
A cat on a hot tin roof.
a cant on a hot in roof
the quikc brown box jupps ober the laze dogs 
The quickbrown fox jumps over the lazy dogs
EOF
}

# Change \x20 to \x02 to simplify parsing diff's output,
#+   then change \x02 back to \x20 for the final output. 
# Change \x09 to \x01 to simplify parsing diff's output, 
#+   then change \x01 into → U+1F143 (Squared Latin Capital Letter T)
function input {
    sed \
        -e "s/\x09/\x01/g" \
        -e "s/\x20/\x02/g" \
        -e "s/\(.\)/\1\n/g"
}
function output {
    sed -n \
        -e "s/\x01/→/g" \
        -e "s/\x02/ /g" \
        -e "s/^\(.\) *\x3C$/\1 \x3C  /g" \
        -e "s/\(.\) *\(.\) \(.\)$/\1\2\3/p"
}

ifs="$IFS"
IFS=$'\n'
demo=true

while IFS= read -t "$timeout" -r a
do
    demo=false
    IFS= read -t "$timeout" -r b
    if [[ $? -ne 0 ]]
    then
        echo 'No corresponding line to compare with' > /dev/stderr
        exit 1
    fi

    diff --text -yt -W 19  \
        <(echo "$a" | input) \
        <(echo "$b" | input) \
    | \
    output | \
    {
        type=''
        buf=''
        while read -r line
        do
            if [[ "${line:1:1}" != "$type" ]]
            then
                if [[ "$type" = '|' ]]
                then
                    type='>'
                    echo -n "$down$buf"
                    buf=''
                fi

                if [[ "${line:1:1}" != "$type" ]]
                then
                    type="${line:1:1}"

                    echo -n "$type" \
                        | sed \
                            -e "s/[<|]/$up/" \
                            -e "s/>/$down/" \
                            -e "s/ /$same/"
                fi
            fi

            case "$type" in
            '|')
                buf="$buf${line:2:1}"
                echo -n "${line:0:1}"
                ;;
            '>')
                echo -n "${line:2:1}"
                ;;
            *)
                echo -n "${line:0:1}"
                ;;
            esac
        done

        if [[ "$type" = '|' ]]
        then
            echo -n "$down$buf"
        fi
    }

    echo -e "$reset"
done

IFS="$ifs"

if $demo
then
    demo
fi

Answer 6

wdiff is actually a very old method of comparing files word-by-word. It worked by reformatting files, then using diff to find differences and passing it back again. I myself suggested adding context, so that rather than word-by-word compare, it does it with each word surrounded by other 'context' words. That allows the diff to synchronise itself on common passages in files much better, especially when files are mostly different with only a few blocks of common words. For example when comparing text for for plagiarism, or re-use.

dwdiff was later created from wdiff. But dwdiff uses that text reformatting function to good effect in dwfilter. This is a great development – it means you can reformat one text to match another, and then compare them using any line-by-line graphical diff displayer. For example, using it with "diffuse" graphical diff....

dwfilter file1 file2 diffuse -w

This reformats file1 to the format of file2 and gives that to diffuse for a visual comparison. file2 is unmodified, so you can edit and merge word differences into it directly in diffuse. If you want to edit file1, you can add -r to reverse which file is reformatted. Try it and you will find it is extremely powerful!

My preference for the graphical diff (shown above) is diffuse as it feels far cleaner and more useful. Also it is a standalone python program, which means it is easy to install and distribute to other UNIX systems.

Other graphical diffs seem to have a lot of dependancies, but can also be used (you choice). These include kdiff3 or xxdiff.

Answer 7

Here's a simple one-liner:

diff -y <(cat a.txt | sed -e 's/,/\n/g') <(cat b.txt | sed -e 's/,/\n/g')

The idea is to replace commas (or whichever delimiter you wish to use) with newlines using sed. diff then takes care of the rest.

Answer 8

GNU Emacs has the very good "ediff" mode. I use it and press a or b frequently.

And power users get even more fancy, e.g. : https://emacs.stackexchange.com/questions/16469/how-to-merge-git-conflicts-in-emacs

And it has lots of powerful features. Expanded help menu:

I highly recommend committing the capabilities shown in the menu to memory, so you use them when they'd be useful. (Wow, I haven't looked at the menu in so long it has features I forgot it had.)

Emacs is available for every *nix platform. And its as open source as it gets.

I use ediff so often that typing M-x ediff took too long, so I bound it to C-c C-e.

Answer 9

• xxdiff: Another tool is xxdiff (GUI), which has to be installed, first.
• spreadsheet: For database data, a spreadsheet from .csv is easily made, and a formula (A7==K7) ? "" : "diff" or similar inserted, and copy-pasted.

Answer 10

git diff --no-index --word-diff-regex=. --word-diff=porcelain ${site} ${site-prev} \
    | grep '^[-+]' \
    > ${site_diff}
if [[ $(wc -c < "${site_diff}") -gt 2000 ]]; then
    echo "Warning: ${url} has changed a lot:"
    # deal with it
fi

An overcrowded answer set already, but I wanted to share my solution from the automation perspective. To detect if a website is malfunctioning or has been hacked, I wanted an accurate, minimal diff between successive end-user snapshots. If there's huge changes, it may be reporting an error message, or may be blank, or have been maliciously changed, or accidentally drastically changed.

On the other hand, a CMS generates lots of trivial ID changes (such as cache timestamps) that do not indicate a real change, and the above is built to boil the issue down. If a few hundred IDs change in HTML tags, the net bytecount of difference will still be small, whereas if someone accidentally deletes several paragraphs, the byte count will be much larger. There's always a fuzzy middleground so I hand-wave at 2000 but probably even 500 is good.

It may also be a good idea to ignore whitespace changes depending on the application.

The essential options for me are:

• --word-diff-regex=. because words are arbitrarily long in code (SQL / HTML / CSS / whatever), so this is actually a character based diff
• --no-index because these files are not in a repository, but YAY GIT!
• --word-diff=porcelain to easily extract the actual differences, which are easily detected by the -/+ line prefixes, with no surrounding context.

The output of the top line without the further grep/wc processing is still human friendly and quite clear, possibly useful for the OP.

Answer 11

On the command line, I would make sure I add judicious new-lines before comparing files. You can use sed, awk, perl or anything really to add line breaks in some sort of systematic way - make sure not to add too many though.

But I find the best is to use vim as it highlights word differences. vim is good if there are not too many differences and the differences are simple.

Answer 12

I had the same problem and solved it with PHP Fine Diff, an online tool that allows you to specify granularity. I know it's not technically a *nix tool, but I didn't really want to download a program just to do a one-time, character level diff.

Answer 13

kdiff3 is becoming the standard GUI diff viewer on Linux. It is similar to xxdiff, but I think kdiff3 is better. It does many things well, including your request to show "exact character differences between two lines in certain files".

Answer 14

If I'm reading your question correctly, I use diff -y for this kind of thing.

It makes comparing a side by side comparison much simpler to find which lines are throwing the differences.

Answer 15

As noted in some comments above, you can install:

apt install wdiff colordiff

and then use

wdiff -n a b | colordiff

-n, --avoid-wraps do not extend fields through newlines