Multi-machine tool in the spirit of moreutils' `parallel`?

Question

parallel from moreutils is a great tool for, among other things, distributing m independent tasks evenly over n CPUs. Does anybody know of a tool that accomplishes the same thing for multiple machines? Such a tool of course wouldn't have to know about the concept of multiple machines or networking or anyhting like that -- I'm just talking about distributing m tasks into N clusters, where in cluster i N_i tasks are run in parallel.

Today I use my own BASH scripts to accomplish the same thing, but a more streamlined and clean tool would be great. Does anyobdy know of any?

Answer 1

GNU Parallel does that and more (using ssh).

It can even deal with mixed speed of machines, as it simply has a queue of jobs, that are started on the list of machines (e.g. one per CPU core). When one jobs finishes another one is started.

So it does not divide the jobs into clusters before starting, but does it dynamically.

Watch the intro videos to learn more: http://pi.dk/1