Sorting columns from two files with awk

Question

I have two input files

28a39a289906c01159f999a68996091a [email protected]
274d1d2c7e931fb55ac0c91dd41f2be7 [email protected]
44d25d3b1b70b240d5058f1be1cef576 pers[email protected]
2227a768f6d253b7bf81bb4ecb15b52d [email protected]

and

28a39a289906c01159f999a68996091a
274d1d2c7e931fb55ac0c91dd41f2be7
44d25d3b1b70b240d5058f1be1cef576
2227a768f6d253b7bf81bb4ecb15b52d

i'm trying to exclude all the rows in file 1 where file2 and file1 have a matching column 1.

md5 suppression.

What i could find was awk 'NR==FNR{a[$1]=$0;next}{print a[$2]}' $1 $2

which i know is pretty close just not sure what i did before to do what i want to do now.

Answer 1

If order doesn't matter (i.e. just exclude all emails with an md5 in the exclude file) and you're not wedded to awk, use join:

join -v 1 -j 1 <(sort emails) <(sort excludes)

-v 1 tells it to print lines in the first file (emails) that don't have a corresponding line in the second file (excludes).
-j 1 tells it to only look at the first column of each.

---

If you want to use awk, I believe this should work:

awk 'NF==1{exclude[$1]++} NF==2&&!exclude[$1]' excludes emails

Or if the two files correspond line-by-line and you only want to exclude, e.g. line 2 if both have the same hash on that particular line, use this:

awk 'NF==1{hash[FNR]=$1} NF==2&&hash[FNR]!=$1' excludes emails

Answer 2

This has the benefit that the files can go in either order:

sort file1 file2 | uniq -u --check-chars 32 | grep -E '.{33}'

sort file1 file2 is combining and sorting the files in preparation for uniq.

uniq -u --check-chars 32 considers only the first 32 characters, and with -u, prints only the unique lines.

grep -E '.{33}' prints only the lines containing 33 or more characters.

EDIT

As Peter.O points out, this does not handle the case where two or more entries in the files list have the same md5. In that case, those files will not be printed. I like the join solution the best.

However, this logic depends on all separate files having unique md5's, which is a bad assumption depending on the use of this output. Here is an example of a backup script using the md5sum logic. Though the join logic correctly identifies both fileA and fileB (unlike my solution), there is still a problem:

echo "hello world" > fileA
cp fileA fileB
./backup_script.sh
cp fileA fileC
./backup_script.sh

Did the op indend fileC to be backed up?

Answer 3

This command should work:

awk '
  NR==FNR {
    a[$1]=$0;
    next
  }
  { if ( a[$1] ) delete a[$1] } 
  END { 
    for ( md5 in a ) { 
      if ( a[md5] ) { print a[md5] } 
    } 
  }
' file1 file2