5

Given a file like:

a
b
c

How do I get an output like:

a       0cc175b9c0f1b6a831c399e269772661
b       92eb5ffee6ae2fec3ad71c777531578f
c       4a8a08f09d37b73795649038408b5f33

in an efficient way? (Input is 80 GB)

2

This could just be a oneliner in perl:

head 80gb | perl -MDigest::MD5=md5_hex -nlE'say"$_\t".md5_hex($_)'
a       0cc175b9c0f1b6a831c399e269772661
b       92eb5ffee6ae2fec3ad71c777531578f
c       4a8a08f09d37b73795649038408b5f33
d       8277e0910d750195b448797616e091ad
e       e1671797c52e15f763380b45e841ec32
f       8fa14cdd754f91cc6554c9e71929cce7
g       b2f5ff47436671b6e533d8dc3614845d
h       2510c39011c5be704182423e3a695e91
i       865c0c0b4ab0e063e5caa3387c1a8741
j       363b122c528f54df4a0446b6bab05515

If you need to store the output and want a nice progress bar while chewing this huge chunk:

sudo apt install pv          #ubuntu/debian
sudo yum install pv          #redhat/fedora
pv 80gb | perl -MDigest::MD5=md5_hex -nlE'say"$_\t".md5_hex($_)' | gzip -1 > 80gb-result.gz
  • 1
    Impressive. This is even faster: pv my80gb-file | parallel --pipe -Sworker1,worker2,worker3,: --round --block 1m -q perl -MDigest::MD5=md5_hex -nlE'say"$_\t".md5_hex($_)' – Ole Tange Aug 2 at 23:27
3

In addition to @Ole Tange's approach, here's an optimized solution (Python's part):

md5summer.py script:

#!/usr/bin/python

import sys
import hashlib

for r in sys.stdin:
    if r.strip():
        h = hashlib.md5()
        h.update(r.encode());
        print r, '\t', h.hexdigest()

Points of optimization:

  • hashlib - using actual library instead of deprecated one
  • for r in sys.stdin: - reading from generator-like object instead of list
  • if r.strip(): - check for empty record to avoid redundant hashlib methods invocations

Usage:

parallel --pipepart -a my80gb-file -S server1,server2 --block 1 /path_to/md5summer.py

Sample output:

a   0cc175b9c0f1b6a831c399e269772661
b   92eb5ffee6ae2fec3ad71c777531578f
c   4a8a08f09d37b73795649038408b5f33
d   8277e0910d750195b448797616e091ad
f   8fa14cdd754f91cc6554c9e71929cce7
e   e1671797c52e15f763380b45e841ec32
g   b2f5ff47436671b6e533d8dc3614845d
h   2510c39011c5be704182423e3a695e91
i   865c0c0b4ab0e063e5caa3387c1a8741
j   363b122c528f54df4a0446b6bab05515
k   8ce4b16b22b58894aa86c421e8759df3
l   2db95e8e1a9267b7a1188556b2013b33
m   6f8f57715090da2632453988d9a1501b
n   7b8b965ad4bca0e41ab51de7b31363a1
p   83878c91171338902e0fe0fb97a8c47a
o   d95679752134a2d9eb61dbd7b91c4bcc
q   7694f4a66316e53c8cdd9d9954bd611d
r   4b43b0aee35624cd95b910189b3dc231
s   03c7c0ace395d80182db07ae2c30f034
t   e358efa489f58062f10dd7316b65649e
u   7b774effe4a349c6dd82ad4f4f21d34c
v   9e3669d19b675bd57058fd4664205d2a
w   f1290186a5d0b1ceab27f4e77c0c5d68
x   9dd4e461268c8034f5c8564e155c67a6
y   415290769594460e2e485922904f345d
z   fbade9e36a3f36d3d676c1b808451dd7
...
  • It seems to fail on input: Asunción. UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128) – Ole Tange Jan 17 '18 at 0:57
  • On my system this solution is 50% slower than my solution. – Ole Tange Jan 17 '18 at 1:03
  • Actually is this equivalent to Ole's solution? This one appears to be generating the cumulative hash not hash per line – iruvar Jan 17 '18 at 1:35
  • I get the same output, so I would say: yes. – Ole Tange Jan 17 '18 at 3:57
  • @iruvar, what do meant with "the cumulative hash"? It'll generate a new hash object for each next record – RomanPerekhrest Jan 17 '18 at 21:24
2

First make a single threaded program (md5er), that can generate the correct output given the input:

#!/usr/bin/python

import sys
import hashlib

for r in sys.stdin:
  print r[:-1], '\t', hashlib.md5(r[:-1]).hexdigest()

Then use GNU Parallel to split the input into chunks that can be distributed to compute servers:

parallel --pipepart -a my80gb-file -Sworker1,worker2,worker3,: --block -10 md5er
  • Python's md5 module is deprecated. Use the hashlib module instead – RomanPerekhrest Jan 16 '18 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.