3

I want to parallelize hash calculation process because I have a very large amount of file counts and sizes. When I see CPU usage of these commands, I get upset because they are only using one thread; how can I parallelize these?

sha256sum foo.mp4
OR
openssl -dgst sha256 foo.mp4
2
  • 2
    hashing algorithms are inherently serial, but for large file count you can of course just hash multiple files at a time by running multiple instances of the command
    – Fox
    May 11, 2021 at 16:26
  • @Fox Some hashing algorithms are inherently serial, but not all.
    – Ole Tange
    May 11, 2021 at 18:58

3 Answers 3

3

For parallelizing across files you can use GNU Parallel:

parallel sha256sum ::: *

Parallelizing hashing for a single file can be done with certain hash functions using a Merkle tree. b3sum is such a tool.

1

Alternatively you could upgrade to a CPU which supports hardware SHA256 calculations like newer Intel or AMD CPUs. My Ryzen 3700X is blazingly fast at calculating SHA256 hash sums, a lot faster than for calculating MD5 which is a much simpler algo. Check for sha_ni in /proc/cpuinfo.

Alternatively you could use Blake3 which is a lot (almost 15 times) faster even when implemented purely in software:

enter image description here

1
  • useful info, +1, but orthogonal to parallel processing. Even with faster HW and algorithm, there are still benefits to running one hashing process per CPU core/thread rather than just one total.
    – cas
    May 12, 2021 at 5:25
1

xargs has a -P option for running multiple jobs in parallel. It's nowhere near as flexible as Ole Tange's parallel program, but it's fine for most simple parallelisation tasks.

For example:

find . -name '*.mp4' -print0 | xargs -0r -n 1 -P 0 openssl dgst -sha256
  • -P 0 tells xargs to run as many jobs in parallel as possible (e.g. on my 32 thread Threadripper 1950x, it will run 32 openssl jobs in parallel)

  • -n 1 tells xargs to run each job with only one argument. While 1 job is sub-optimal for openssl dgst which can process many filenames on a command line, you will almost always want to use the -n (or -L) option with xargs -P.

    Otherwise it will try to fit as many args onto each command line as it can - usually resulting in just one job unless you have many tens of thousands of arguments. On Linux, the command line length limit is typically 2 million characters, 2097152 (check with getconf ARG_MAX). That allows for a lot of filenames.

    The optimum is to count how many args you have and divide that by the number of jobs you want to run in parallel. e.g.

    t=32
    numfiles=$(find .//. -name '*.mp4' | grep -c //)
    let n=numfiles/t
    find . -name '*.mp4' -print0 | xargs -0r -n "$n" -P "$t" openssl dgst -sha256
    

Note: -P is not a standard POSIX option for xargs. Requires GNU or *BSD xargs. Maybe some other versions too.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .