How can I compare two files line by line?

Question

I have two files A and B which are almost same with some lines different and with some lines jumbled. Since the two files are systemverilog files the lines also contain special characters like ; , = + and etc.

I want to loop through each line of fileA and check if there is a corresponding match in fileB. The comparison should follow the rules

1. Whitespaces at the beginning and end of the line can be ignored.
2. Multiple Space/Tab between words can be considered as single Space.
3. Empty lines can be ignored

The result should display the lines that are present in fileA but not in fileB.

I tried tkdiff but since some lines are jumbled it shows many differences.

Answer 1

I can't speak to how portable this is but I tried to cover all the bases. I did my best to replicate the two files in my testing based on your information. If you run into special character issues with sed they can be esacped in the second line of the cleanLine function.

#!/bin/bash

# compare two files and return lines in
# first file that are missing in second file

ProgName=${0##*/}
Pid=$$
CHK_FILE="$1"
REF_FILE="$2"
D_BUG="$3"
TMP_FILE="/tmp/REF_${Pid}.tmp"
declare -a MISSING='()'
m=0

scriptUsage() {
cat <<ENDUSE

    $ProgName  <file_to_check> <reference_file> [-d|--debug]

    Lines in 'file_to_check' not present in 'reference_file'
      are printed to standard output.

    file_to_check:     File being checked
    reference_file:    File to be checked against
    -d|--debug:        Run script in debug mode (Optional)
    -h|--help:         Print this help message

ENDUSE
}

# delete temp file on any exit
trap 'rm $TMP_FILE > /dev/null 2>&1' EXIT


#-- check args
  [[ $CHK_FILE == "-h" || $CHK_FILE == "--help" ]] && { scriptUsage; exit 0; }
  [[ -n $CHK_FILE && -n $REF_FILE ]] || { >&2 echo "Not enough arguments!"; scriptUsage; exit 1; }
  [[ $D_BUG == "-d" || $D_BUG == "--debug" ]] && set -x
  [[ -s $CHK_FILE ]] || { >&2 echo "File $CHK_FILE not found"; exit 1; }
  [[ -s $REF_FILE ]] || { >&2 echo "File $REF_FILE not found"; exit 1; }
#--


#== edit temp file to 3 match comparison rules
  # copy ref file to temp for editing
  cp "$REF_FILE" $TMP_FILE || { >&2 echo "Unable to create temporary file"; exit 1; }
  # rule 3 - ignore empty lines
  sed -i '/^\s*$/d' $TMP_FILE
  # rule 1 - ignore begin/end of line spaces
  sed -i 's/^[[:space:]][[:space:]]*//;s/[[:space:]][[:space:]]*$//' $TMP_FILE
  # rule 2 - multi space/tab as single space
  sed -i 's/[[:space:]][[:space:]]*/ /g' $TMP_FILE
#==


# function to clean LINE to match 3 rules
# & escape '/' and '.' for later sed command
cleanLine() {
  var=$(echo "$1" | sed 's/^[[:space:]][[:space:]]*//;s/[[:space:]][[:space:]]*$//;s/[[:space:]][[:space:]]*/ /g')
  echo $var | sed 's/\//\\\//g;s/\./\\\./g'
}


### parse check file
while IFS='' read -r LINE || [[ -n $LINE ]]
  do
    if [[ -z $LINE ]]
      then
        continue
      else
        CLN_LINE=$(cleanLine "$LINE")
        FOUND=$(sed -n "/$CLN_LINE/{p;q}" $TMP_FILE)
        [[ -z $FOUND ]] && MISSING[$m]="$LINE" && ((m++))
        FOUND=""
    fi
done < "$CHK_FILE"
###


#++ print missing line(s) (if any)
  if (( $m > 0 ))
    then
      printf "\n  Missing line(s) found:\n"
      #*SEE BELOW ON THIS
      for (( p=0; $p<$m; p++ ))
        do
          printf "    %s\n" "${MISSING[$p]}"
      done
      echo
    else
      printf "\n  **No missing lines found**\n\n"
  fi
#* using 'for p in ${MISSING[@]}' causes:
#* "SPACED LINES" to become:
#* "SPACED"
#* "LINES" when printed to stdout!
#++

Answer 2

A simple solution:

diff -bB fileA fileB | grep -v '^>'

-b (or --ignore-space-change) says “ignore changes in the amount of white space”.  -B (or --ignore-blank-lines) says “ignore changes whose lines are all blank”.  grep -v '>' strips out reports of lines that are in fileB but not fileA.

This doesn’t ignore leading spaces, but otherwise it’s close to what you seem to want.

If “The lines that are present in B but not in A is are also of interest”, why not just do diff -bB fileA fileB instead of doing half of a diff and doing it twice?

Answer 3

diff -w file1 file2

The -w flag to diff will make it ignore whitespace characters (this is an extension implemented by most diff implementations).

With the following input:

file1:

hello world

abc
123

this is line 2 (the last line)

file2:

        hello   world

abc
123

this is line 3 (the last line)

The command generates

6c6
< this is line 2 (the last line)
---
> this is line 3 (the last line)

To make it ignore empty lines, pre-process the input files by removing empty lines. Using a shell that understands process substitution (like bash or ksh93):

diff -w <( sed '/^[[:space:]]*$/d' file1 ) <( sed '/^[[:space:]]*$/d' file2 )

If your diff has an option for ignoring blank lines (look for -B in the manual if you're using GNU diff) then use that. Mine does not have such an option.

Answer 4

diff -wB file[12]

-w, --ignore-all-space 
    ignore all white space
-B, --ignore-blank-lines
    ignore changes whose lines are all blank

Please note that -w option is ignoring all whitespaces before diffing, so a line like thi s i s a line or this is a line, will compare as thisisaline and will not report difference.

Beside of above problem, we can use -b option as suggested by Scott, but -b option doesn't ignore whitespaces if come at the begging.

if the output is what you want by -w option, above is correct, otherwise we can use sed to ignoring those line starting with whitespaces and use diff -Bb.

diff -Bb <(sed 's/^[ \t]*//' file1) <(sed 's/^[ \t]*//' file2)

Answer 5

Here is the bash script. I didnt validated argument $1 and $2. You need to validate if two file exist. I didnt tested much but I guess your 3 conditions are met here. Here is the source code. Script returns 0 if two files are equal and 1 if they are not. You can run echo $? after running the the script.

#!/bin/bash

code=0;
n=1;

dstcount=`wc -l $2 | awk '{print $1}'`

while read line
do
    #remove spaces from the beginning of line and compress tab/spaces
    src=`echo $line | tr '\t' ' ' | tr -s ' '`
    dst=`echo $(sed -n "$n"p $2) | tr '\t' ' ' | tr -s ' '`

    if [ -z "$src" ]
    then
        continue
        #advance to next line in source file
    fi

    if [ -z "$dst" ]
    then
        #advance to next in destination file
        while [ $n -le $dstcount ]
        do
            dst=`echo $(sed -n "$n"p $2) | tr '\t' ' ' | tr -s ' '`
            if [ ! -z "$dst" ]
            then
                break;
            fi

            n=`expr $n + 1`

        done

        if [ $n -gt $dstcount ]
        then
            code=1
            break
        fi

    fi

    if [ ! "$src" == "$dst" ]
    then
        code=1
        break
    fi

    n=`expr $n + 1`

done < $1

exit $code;

Answer 6

Try this tool https://ricardojlrufino.github.io/bitdiff/

This compare individual characters