Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have two files:

  1. oldlist - This contains a list of files and a md5 hash for each file. This was generated one year ago.
  2. newlist - This also contains a list of files and a md5 hash for each file. However, some files have been changed (e.g. their md5 hash is different) and some new files have been added.

I would like to see all differences between oldlist and newlist, but I want to ignore any files which don't exist in oldlist.

That is, I don't care about new files. I only want to compare the md5 hashes for each old file, so that I can see if any files have changed within the last year.

I have tried diff and comm, but have not found a solution yet.

share|improve this question

Use join to combine matching lines from the two files. Assuming the file names come after the checksums (as in md5sum output) and don't contain whitespace, this will print all file names that are present in both lists, together with the old checksum and the new checksum:

join -1 2 -2 2 <(sort -k 2 oldlist) <(sort -k 2 newlist)

To also see new files, pass the -a option to join. A bit of output postprocessing will remove the file names for which the checksum has not changed.

join -a 2 -1 2 -2 2 <(sort -k 2 oldlist) <(sort -k 2 newlist) |
awk '$2 != $3'
share|improve this answer

You could do it with awk alone:

$ awk 'FNR==NR   { o[$2]=$1; next }       !o[$2] { print $0, "NEW"; next } 
       $1!=o[$2] { print $0, "CHANGED" }' newlist oldlist

(Note that the supposed format of the files is md5sum's output format: "md5 filename".)

Update: step-by-step explanation of how that awk one-liner works.

awk 'FNR==NR { # if current record number==overall record number (still processing the first file)
  o[$2]=$1     # store the record in array o: the key is the file name, the value is the md5
  next         # go to next record (do not execute the rest of the code)
# reaching this point means we are processing the second input file
!o[$2] {       # if array o not contains item with the current record`s file name
  print $0, "NEW" # print the current record and specify that it`s new
  next         # go to next record (do not execute the rest of the code)
# reaching this point means array o contains item with the current file name
$1!=o[$2] {    # if the current md5 is not equal with the md5 save for the current file name
  print $0, "CHANGED" # print the current record and specify it`s changed
}' newlist oldlist
share|improve this answer
nice answer, do you mind adding a small explanation of the awk program? For example that FNR==NR guards the action from applying to the 2nd file etc. – maxschlepzig Sep 8 '11 at 19:26
@maxschlepzig, I added the explanation. In case it not meets the SO standards and expectations I am open for suggestions to update it. – manatwork Sep 9 '11 at 6:57

if i'd understood your question correctly then comm can indeed do what you want. I'd suggest looking into comm --help


  -1              suppress column 1 (lines unique to FILE1)
  -2              suppress column 2 (lines unique to FILE2)
  -3              suppress column 3 (lines that appear in both files)

so comm newFile oldFile -1 -3 will do what you want.

share|improve this answer

Suppose the files looks like (space separated):

file1 md5sum1
file2 md5sum2

Straightforward solution:

# get only the files:
cut -f 1 -d " " oldlist > oldlist.files 

# from newlist, take only files which were also in the oldlist (updated files)
grep -w -F -f oldlist.files newlist > newlist.updated_files

And then you can just compare the two files (after sorting):

sort -u oldlist > oldlist.su
sort -u newlist.updated_files > newlist.updated_files.su
diff oldlist.su newlist.updated_files.su
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.