Compare an old file and new file, but ignore lines which only exist in new file?

Question

I have two files:

1. oldlist - This contains a list of files and a md5 hash for each file. This was generated one year ago.
2. newlist - This also contains a list of files and a md5 hash for each file. However, some files have been changed (e.g. their md5 hash is different) and some new files have been added.

I would like to see all differences between oldlist and newlist, but I want to ignore any files which don't exist in oldlist.

That is, I don't care about new files. I only want to compare the md5 hashes for each old file, so that I can see if any files have changed within the last year.

I have tried diff and comm, but have not found a solution yet.

Answer 1

Use join to combine matching lines from the two files. Assuming the file names come after the checksums (as in md5sum output) and don't contain whitespace, this will print all file names that are present in both lists, together with the old checksum and the new checksum:

join -1 2 -2 2 <(sort -k 2 oldlist) <(sort -k 2 newlist)

To also see new files, pass the -a option to join. A bit of output postprocessing will remove the file names for which the checksum has not changed.

join -a 2 -1 2 -2 2 <(sort -k 2 oldlist) <(sort -k 2 newlist) |
awk '$2 != $3'

Answer 2

You could do it with awk alone:

$ awk 'FNR==NR   { o[$2]=$1; next }       !o[$2] { print $0, "NEW"; next } 
       $1!=o[$2] { print $0, "CHANGED" }' newlist oldlist

(Note that the supposed format of the files is md5sum's output format: "md5 filename".)

Update: step-by-step explanation of how that awk one-liner works.

awk 'FNR==NR { # if current record number==overall record number (still processing the first file)
  o[$2]=$1     # store the record in array o: the key is the file name, the value is the md5
  next         # go to next record (do not execute the rest of the code)
}
# reaching this point means we are processing the second input file
!o[$2] {       # if array o not contains item with the current record`s file name
  print $0, "NEW" # print the current record and specify that it`s new
  next         # go to next record (do not execute the rest of the code)
}
# reaching this point means array o contains item with the current file name
$1!=o[$2] {    # if the current md5 is not equal with the md5 save for the current file name
  print $0, "CHANGED" # print the current record and specify it`s changed
}' newlist oldlist

Answer 3

if i'd understood your question correctly then comm can indeed do what you want. I'd suggest looking into comm --help

specificically

  -1              suppress column 1 (lines unique to FILE1)
  -2              suppress column 2 (lines unique to FILE2)
  -3              suppress column 3 (lines that appear in both files)

so comm newFile oldFile -1 -3 will do what you want.

Answer 4

Suppose the files looks like (space separated):

file1 md5sum1
file2 md5sum2

Straightforward solution:

# get only the files:
cut -f 1 -d " " oldlist > oldlist.files 

# from newlist, take only files which were also in the oldlist (updated files)
grep -w -F -f oldlist.files newlist > newlist.updated_files

And then you can just compare the two files (after sorting):

sort -u oldlist > oldlist.su
sort -u newlist.updated_files > newlist.updated_files.su
diff oldlist.su newlist.updated_files.su

Answer 5

Just for an alternative, I've always used "sdiff -s" for comparing lists of files or md5sums.

Assuming the files are normal md5sum output "md5hash filename". Then either:

sdiff -s oldfile newfile | grep -v ">"
# sorting on the md5hash should help align and pick up renamed files.
sdiff -s <(sort oldfile) <(sort newfile)

Breaking this down:
sdiff -s : supress common lines, so exact matches are ignored. Shows |,<,> for differences.
<(sort oldfile) : does the command sort before the sdiff.
grep -v ">" : Ignore newfile entries. Only works if you don't have > in thefilename, which is unlikely anyway.

The width of sdiff can be altered to show longer lines -w 100.