1

Commands like find and grep would benefit greatly if they were to run in parallel. Is there a utility or a program that takes a command as an argument and makes it run in parallel by running multiple threads of the command, each on a subset of the directory tree?

2

GNU xargs has a -P command for that.

find . ... -print0 | xargs -r0P4 -n10 some-command

Will run up to 4 some-commands at a time with up to 10 files as argument.

1

Actually commands like find and grep are almost always IO-bound: the disk is the bottleneck, not the CPU. In such cases, if you run several instances in parallel, they will compete for I/O bandwidth and cache, and so they will be slower.

With a command that is more CPU-intensive, or in rare cases (e.g. a very fast RAID-1 that isn't otherwise busy together with a slow CPU), you can use tools such as GNU parallel or the -P option of recent versions of GNU xargs.

Assuming that file names do not contain newlines, the following commands invoke cpu_intensive_command on each file in turn, with two parallel threads:

find -type f | xargs -I {} -P 2 cpu_intensive_command {}
find -type f | parallel -X --number-of-cpus=2 cpu_intensive_command {}

An alternative strategy is to divide the work in two parts that you estimate are approximately equal. This requires no fancy tools, but does not balance the jobs between the processors.

cpu_intensive_command [a-m]* &
cpu_intensive_command [!a-m]*
wait
0

Walk through GNU Parallel's tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html and watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.