0

I need to hash each line of second file (md5) and compare it with the first file line by line like that :

awk -F, 'FNR==NR {a[$1]; next}; !(md5($1) in a)' file1 file2 > file3

How can I do this using awk?

file1:

5ca4136a060a2574a936cdf6881f733c
8068123f8384b305d133add978a3c12c

file2:

aaa@aaa.com
ccc@ccc.com

and with this example :

5ca4136a060a2574a936cdf6881f733c = aaa@aaa.com

output:

ccc@ccc.com
  • Also, while you're there, explain the relevance of needing a comma as awk's field separator. – roaima Sep 29 '17 at 17:26
  • hey, thank you for eply please check i edit my post (y) – Hero Sep 29 '17 at 17:29
  • plz any one can help me do that ? – Hero Sep 29 '17 at 17:54
  • 2
    You asked the question half an hour ago. Please give us some time. – Kusalananda Sep 29 '17 at 17:55
  • So you want a file (or a line) in file2 which has no hash in file1? – αғsнιη Sep 29 '17 at 18:01
1

Try.

awk 'NR==FNR{seen[$0];next}{x="echo " $0 "|md5sum| cut -d- -f1|cut -f1 ";
     x|getline l;close(x);md5[l]=$0} 
END{for (M in md5) {chk=md5[M];gsub (" ", "", M);for (U in seen) if(M==U){print M, chk}}
}' file1 file2

Inputs:

==> file1 <==
5ca4136a060a2574a936cdf6881f733c
8068123f8384b305d133add978a3c12c
96653200bf87722738bc4abf7b3d5589
731a1f0a260d40c758aa18237de3fe6c
==> file2 <==
aaa@aaa.com
ccc@ccc.com
UNIX.stackexchange.com
STACKECHANGE.COM

Output (those mail addresses that their hash was seen in file1):

731a1f0a260d40c758aa18237de3fe6c UNIX.stackexchange.com
96653200bf87722738bc4abf7b3d5589 ccc@ccc.com

With using grep that's a quite easy.

grep -Ff file1 \
    <(awk '{x="echo " $0 "|md5sum| cut -d- -f1"; x|getline l;print l, $0}' file2)

96653200bf87722738bc4abf7b3d5589   ccc@ccc.com
731a1f0a260d40c758aa18237de3fe6c   UNIX.stackexchange.com
0

Using join and Perl for doing the MD5 computation, in bash or ksh93, or zsh:

$ join -v1 <( perl -MDigest::MD5 -ne 'chomp; printf("%s %s\n",Digest::MD5::md5_hex($_),$_)' text-file | sort ) \
           <( sort md5-file )
5a09ab8a3f03376046a6e6eec0a0d511 ccc@ccc.com

join is told to join the sorted MD5 hashes from the file with the mail addresses, with the ones found in the MD5 file, and to print every mismatched line from the address file.

The hash is printed too, but you may filter out that with cut -d ' ' -f 2-:

$ join -v1 <( perl -MDigest::MD5 -ne 'chomp; printf("%s %s\n",Digest::MD5::md5_hex($_),$_)' text-file | sort ) \
           <( sort md5-file ) | cut -d ' ' -f 2-
ccc@ccc.com

The output from the Perl code is on the form

5ca4136a060a2574a936cdf6881f733c aaa@aaa.com
5a09ab8a3f03376046a6e6eec0a0d511 ccc@ccc.com

and this is what's sorted and given to join. The join will happen on the first space-delimited field by default.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.