5

Given a file like:

a
b
c

How do I get an output like:

a       0cc175b9c0f1b6a831c399e269772661
b       92eb5ffee6ae2fec3ad71c777531578f
c       4a8a08f09d37b73795649038408b5f33

in an efficient way? (Input is 80 GB)

3 Answers 3

3

This could just be a oneliner in perl:

head 80gb | perl -MDigest::MD5=md5_hex -nlE'say"$_\t".md5_hex($_)'
a       0cc175b9c0f1b6a831c399e269772661
b       92eb5ffee6ae2fec3ad71c777531578f
c       4a8a08f09d37b73795649038408b5f33
d       8277e0910d750195b448797616e091ad
e       e1671797c52e15f763380b45e841ec32
f       8fa14cdd754f91cc6554c9e71929cce7
g       b2f5ff47436671b6e533d8dc3614845d
h       2510c39011c5be704182423e3a695e91
i       865c0c0b4ab0e063e5caa3387c1a8741
j       363b122c528f54df4a0446b6bab05515

If you need to store the output and want a nice progress bar while chewing this huge chunk:

sudo apt install pv          #ubuntu/debian
sudo yum install pv          #redhat/fedora
pv 80gb | perl -MDigest::MD5=md5_hex -nlE'say"$_\t".md5_hex($_)' | gzip -1 > 80gb-result.gz
1
  • 1
    Impressive. This is even faster: pv my80gb-file | parallel --pipe -Sworker1,worker2,worker3,: --round --block 1m -q perl -MDigest::MD5=md5_hex -nlE'say"$_\t".md5_hex($_)'
    – Ole Tange
    Commented Aug 2, 2019 at 23:27
3

In addition to @Ole Tange's approach, here's an optimized solution (Python's part):

md5summer.py script:

#!/usr/bin/python

import sys
import hashlib

for r in sys.stdin:
    if r.strip():
        h = hashlib.md5()
        h.update(r.encode());
        print r, '\t', h.hexdigest()

Points of optimization:

  • hashlib - using actual library instead of deprecated one
  • for r in sys.stdin: - reading from generator-like object instead of list
  • if r.strip(): - check for empty record to avoid redundant hashlib methods invocations

Usage:

parallel --pipepart -a my80gb-file -S server1,server2 --block 1 /path_to/md5summer.py

Sample output:

a   0cc175b9c0f1b6a831c399e269772661
b   92eb5ffee6ae2fec3ad71c777531578f
c   4a8a08f09d37b73795649038408b5f33
d   8277e0910d750195b448797616e091ad
f   8fa14cdd754f91cc6554c9e71929cce7
e   e1671797c52e15f763380b45e841ec32
g   b2f5ff47436671b6e533d8dc3614845d
h   2510c39011c5be704182423e3a695e91
i   865c0c0b4ab0e063e5caa3387c1a8741
j   363b122c528f54df4a0446b6bab05515
k   8ce4b16b22b58894aa86c421e8759df3
l   2db95e8e1a9267b7a1188556b2013b33
m   6f8f57715090da2632453988d9a1501b
n   7b8b965ad4bca0e41ab51de7b31363a1
p   83878c91171338902e0fe0fb97a8c47a
o   d95679752134a2d9eb61dbd7b91c4bcc
q   7694f4a66316e53c8cdd9d9954bd611d
r   4b43b0aee35624cd95b910189b3dc231
s   03c7c0ace395d80182db07ae2c30f034
t   e358efa489f58062f10dd7316b65649e
u   7b774effe4a349c6dd82ad4f4f21d34c
v   9e3669d19b675bd57058fd4664205d2a
w   f1290186a5d0b1ceab27f4e77c0c5d68
x   9dd4e461268c8034f5c8564e155c67a6
y   415290769594460e2e485922904f345d
z   fbade9e36a3f36d3d676c1b808451dd7
...
8
  • It seems to fail on input: Asunción. UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128)
    – Ole Tange
    Commented Jan 17, 2018 at 0:57
  • On my system this solution is 50% slower than my solution.
    – Ole Tange
    Commented Jan 17, 2018 at 1:03
  • Actually is this equivalent to Ole's solution? This one appears to be generating the cumulative hash not hash per line
    – iruvar
    Commented Jan 17, 2018 at 1:35
  • I get the same output, so I would say: yes.
    – Ole Tange
    Commented Jan 17, 2018 at 3:57
  • @iruvar, what do meant with "the cumulative hash"? It'll generate a new hash object for each next record Commented Jan 17, 2018 at 21:24
2

First make a single threaded program (md5er), that can generate the correct output given the input:

#!/usr/bin/python

import sys
import hashlib

for r in sys.stdin:
  print r[:-1], '\t', hashlib.md5(r[:-1]).hexdigest()

Then use GNU Parallel to split the input into chunks that can be distributed to compute servers:

parallel --pipepart -a my80gb-file -Sworker1,worker2,worker3,: --block -10 md5er
1
  • Python's md5 module is deprecated. Use the hashlib module instead Commented Jan 16, 2018 at 16:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .