Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have two input files

28a39a289906c01159f999a68996091a tom@dong.com
274d1d2c7e931fb55ac0c91dd41f2be7 go@far.com
44d25d3b1b70b240d5058f1be1cef576 person@domain.com
2227a768f6d253b7bf81bb4ecb15b52d person@gmail.com

and

28a39a289906c01159f999a68996091a
274d1d2c7e931fb55ac0c91dd41f2be7
44d25d3b1b70b240d5058f1be1cef576
2227a768f6d253b7bf81bb4ecb15b52d

i'm trying to exclude all the rows in file 1 where file2 and file1 have a matching column 1.

md5 suppression.

What i could find was awk 'NR==FNR{a[$1]=$0;next}{print a[$2]}' $1 $2

which i know is pretty close just not sure what i did before to do what i want to do now.

share|improve this question
    
awk 'FNR==NR{md5[$1];next}(!($2 in md5)){print $1}' $1 $2 –  Arian Feb 24 '12 at 23:04
    
Any reason you need to do this in awk? It could be done easier other ways. –  Kevin Feb 25 '12 at 2:29
add comment

3 Answers

up vote 3 down vote accepted

If order doesn't matter (i.e. just exclude all emails with an md5 in the exclude file) and you're not wedded to awk, use join:

join -v 1 -j 1 <(sort emails) <(sort excludes)

-v 1 tells it to print lines in the first file (emails) that don't have a corresponding line in the second file (excludes).
-j 1 tells it to only look at the first column of each.


If you want to use awk, I believe this should work:

awk 'NF==1{exclude[$1]++} NF==2&&!exclude[$1]' excludes emails

Or if the two files correspond line-by-line and you only want to exclude, e.g. line 2 if both have the same hash on that particular line, use this:

awk 'NF==1{hash[FNR]=$1} NF==2&&hash[FNR]!=$1' excludes emails
share|improve this answer
    
I can finally see what the last example is trying to do, if your FNL is meant to be FNR. I presume "lobbed up" means "not-in-order". Here, in Australia, "to lob up" means "to turn up unexpectedly (a person)", so figuring out how that and FNL were related was "interesting"... (+1) for a good answer... –  Peter.O Feb 25 '12 at 5:45
    
@Peter.O Yes, FNR; "lobbed up" was an input error on the tablet I was using. I've reworded it and tested now. –  Kevin Feb 25 '12 at 6:13
add comment

This command should work:

awk '
  NR==FNR {
    a[$1]=$0;
    next
  }
  { if ( a[$1] ) delete a[$1] } 
  END { 
    for ( md5 in a ) { 
      if ( a[md5] ) { print a[md5] } 
    } 
  }
' file1 file2
share|improve this answer
    
This prints only the last unmatched line when there is more than one line with the same unmatched key. –  Peter.O Feb 25 '12 at 7:52
add comment

This has the benefit that the files can go in either order:

sort file1 file2 | uniq -u --check-chars 32 | grep -E '.{33}'

sort file1 file2 is combining and sorting the files in preparation for uniq.

uniq -u --check-chars 32 considers only the first 32 characters, and with -u, prints only the unique lines.

grep -E '.{33}' prints only the lines containing 33 or more characters.

EDIT

As Peter.O points out, this does not handle the case where two or more entries in the files list have the same md5. In that case, those files will not be printed. I like the join solution the best.

However, this logic depends on all separate files having unique md5's, which is a bad assumption depending on the use of this output. Here is an example of a backup script using the md5sum logic. Though the join logic correctly identifies both fileA and fileB (unlike my solution), there is still a problem:

echo "hello world" > fileA
cp fileA fileB
./backup_script.sh
cp fileA fileC
./backup_script.sh

Did the op indend fileC to be backed up?

share|improve this answer
    
It is an interesting approach, but it does not print unmatched lines for which there is more than one with the same unmatched key –  Peter.O Feb 25 '12 at 7:56
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.