I have two input files

28a39a289906c01159f999a68996091a [email protected]
274d1d2c7e931fb55ac0c91dd41f2be7 [email protected]
44d25d3b1b70b240d5058f1be1cef576 pers[email protected]
2227a768f6d253b7bf81bb4ecb15b52d [email protected]



i'm trying to exclude all the rows in file 1 where file2 and file1 have a matching column 1.

md5 suppression.

What i could find was awk 'NR==FNR{a[$1]=$0;next}{print a[$2]}' $1 $2

which i know is pretty close just not sure what i did before to do what i want to do now.

  • awk 'FNR==NR{md5[$1];next}(!($2 in md5)){print $1}' $1 $2
    – Arian
    Feb 24, 2012 at 23:04
  • Any reason you need to do this in awk? It could be done easier other ways.
    – Kevin
    Feb 25, 2012 at 2:29

3 Answers 3


If order doesn't matter (i.e. just exclude all emails with an md5 in the exclude file) and you're not wedded to awk, use join:

join -v 1 -j 1 <(sort emails) <(sort excludes)

-v 1 tells it to print lines in the first file (emails) that don't have a corresponding line in the second file (excludes).
-j 1 tells it to only look at the first column of each.

If you want to use awk, I believe this should work:

awk 'NF==1{exclude[$1]++} NF==2&&!exclude[$1]' excludes emails

Or if the two files correspond line-by-line and you only want to exclude, e.g. line 2 if both have the same hash on that particular line, use this:

awk 'NF==1{hash[FNR]=$1} NF==2&&hash[FNR]!=$1' excludes emails
  • I can finally see what the last example is trying to do, if your FNL is meant to be FNR. I presume "lobbed up" means "not-in-order". Here, in Australia, "to lob up" means "to turn up unexpectedly (a person)", so figuring out how that and FNL were related was "interesting"... (+1) for a good answer...
    – Peter.O
    Feb 25, 2012 at 5:45
  • @Peter.O Yes, FNR; "lobbed up" was an input error on the tablet I was using. I've reworded it and tested now.
    – Kevin
    Feb 25, 2012 at 6:13

This has the benefit that the files can go in either order:

sort file1 file2 | uniq -u --check-chars 32 | grep -E '.{33}'

sort file1 file2 is combining and sorting the files in preparation for uniq.

uniq -u --check-chars 32 considers only the first 32 characters, and with -u, prints only the unique lines.

grep -E '.{33}' prints only the lines containing 33 or more characters.


As Peter.O points out, this does not handle the case where two or more entries in the files list have the same md5. In that case, those files will not be printed. I like the join solution the best.

However, this logic depends on all separate files having unique md5's, which is a bad assumption depending on the use of this output. Here is an example of a backup script using the md5sum logic. Though the join logic correctly identifies both fileA and fileB (unlike my solution), there is still a problem:

echo "hello world" > fileA
cp fileA fileB
cp fileA fileC

Did the op indend fileC to be backed up?

  • It is an interesting approach, but it does not print unmatched lines for which there is more than one with the same unmatched key
    – Peter.O
    Feb 25, 2012 at 7:56

This command should work:

awk '
  NR==FNR {
  { if ( a[$1] ) delete a[$1] } 
  END { 
    for ( md5 in a ) { 
      if ( a[md5] ) { print a[md5] } 
' file1 file2
  • This prints only the last unmatched line when there is more than one line with the same unmatched key.
    – Peter.O
    Feb 25, 2012 at 7:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .