Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

parallel from moreutils is a great tool for, among other things, distributing m independent tasks evenly over n CPUs. Does anybody know of a tool that accomplishes the same thing for multiple machines? Such a tool of course wouldn't have to know about the concept of multiple machines or networking or anyhting like that -- I'm just talking about distributing m tasks into N clusters, where in cluster i N_i tasks are run in parallel.

Today I use my own BASH scripts to accomplish the same thing, but a more streamlined and clean tool would be great. Does anyobdy know of any?

share|improve this question
Maybe ClusterIt – Gilles Feb 28 '12 at 19:52
up vote 7 down vote accepted

GNU Parallel does that and more (using ssh).

It can even deal with mixed speed of machines, as it simply has a queue of jobs, that are started on the list of machines (e.g. one per CPU core). When one jobs finishes another one is started.

So it does not divide the jobs into clusters before starting, but does it dynamically.

Watch the intro videos to learn more: http://pi.dk/1

share|improve this answer
Wow, for some reason I've been thinking that "moreutils parallel == GNU parallel", so I wrote it off. But no! Thanks! What you're describing is exactly what I want. – gspr Feb 29 '12 at 21:49

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.