Sign up ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

I have two input files




i'm trying to exclude all the rows in file 1 where file2 and file1 have a matching column 1.

md5 suppression.

What i could find was awk 'NR==FNR{a[$1]=$0;next}{print a[$2]}' $1 $2

which i know is pretty close just not sure what i did before to do what i want to do now.

share|improve this question
awk 'FNR==NR{md5[$1];next}(!($2 in md5)){print $1}' $1 $2 – Arian Feb 24 '12 at 23:04
Any reason you need to do this in awk? It could be done easier other ways. – Kevin Feb 25 '12 at 2:29

3 Answers 3

up vote 3 down vote accepted

If order doesn't matter (i.e. just exclude all emails with an md5 in the exclude file) and you're not wedded to awk, use join:

join -v 1 -j 1 <(sort emails) <(sort excludes)

-v 1 tells it to print lines in the first file (emails) that don't have a corresponding line in the second file (excludes).
-j 1 tells it to only look at the first column of each.

If you want to use awk, I believe this should work:

awk 'NF==1{exclude[$1]++} NF==2&&!exclude[$1]' excludes emails

Or if the two files correspond line-by-line and you only want to exclude, e.g. line 2 if both have the same hash on that particular line, use this:

awk 'NF==1{hash[FNR]=$1} NF==2&&hash[FNR]!=$1' excludes emails
share|improve this answer
I can finally see what the last example is trying to do, if your FNL is meant to be FNR. I presume "lobbed up" means "not-in-order". Here, in Australia, "to lob up" means "to turn up unexpectedly (a person)", so figuring out how that and FNL were related was "interesting"... (+1) for a good answer... – Peter.O Feb 25 '12 at 5:45
@Peter.O Yes, FNR; "lobbed up" was an input error on the tablet I was using. I've reworded it and tested now. – Kevin Feb 25 '12 at 6:13

This command should work:

awk '
  NR==FNR {
  { if ( a[$1] ) delete a[$1] } 
  END { 
    for ( md5 in a ) { 
      if ( a[md5] ) { print a[md5] } 
' file1 file2
share|improve this answer
This prints only the last unmatched line when there is more than one line with the same unmatched key. – Peter.O Feb 25 '12 at 7:52

This has the benefit that the files can go in either order:

sort file1 file2 | uniq -u --check-chars 32 | grep -E '.{33}'

sort file1 file2 is combining and sorting the files in preparation for uniq.

uniq -u --check-chars 32 considers only the first 32 characters, and with -u, prints only the unique lines.

grep -E '.{33}' prints only the lines containing 33 or more characters.


As Peter.O points out, this does not handle the case where two or more entries in the files list have the same md5. In that case, those files will not be printed. I like the join solution the best.

However, this logic depends on all separate files having unique md5's, which is a bad assumption depending on the use of this output. Here is an example of a backup script using the md5sum logic. Though the join logic correctly identifies both fileA and fileB (unlike my solution), there is still a problem:

echo "hello world" > fileA
cp fileA fileB
cp fileA fileC

Did the op indend fileC to be backed up?

share|improve this answer
It is an interesting approach, but it does not print unmatched lines for which there is more than one with the same unmatched key – Peter.O Feb 25 '12 at 7:56

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.