Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

parallel from moreutils is a great tool for, among other things, distributing m independent tasks evenly over n CPUs. Does anybody know of a tool that accomplishes the same thing for multiple machines? Such a tool of course wouldn't have to know about the concept of multiple machines or networking or anyhting like that -- I'm just talking about distributing m tasks into N clusters, where in cluster i N_i tasks are run in parallel.

Today I use my own BASH scripts to accomplish the same thing, but a more streamlined and clean tool would be great. Does anyobdy know of any?

share|improve this question
    
Maybe ClusterIt –  Gilles Feb 28 '12 at 19:52
add comment

1 Answer 1

up vote 4 down vote accepted

GNU Parallel does that and more (using ssh).

It can even deal with mixed speed of machines, as it simply has a queue of jobs, that are started on the list of machines (e.g. one per CPU core). When one jobs finishes another one is started.

So it does not divide the jobs into clusters before starting, but does it dynamically.

Watch the intro videos to learn more: http://pi.dk/1

share|improve this answer
2  
Wow, for some reason I've been thinking that "moreutils parallel == GNU parallel", so I wrote it off. But no! Thanks! What you're describing is exactly what I want. –  gspr Feb 29 '12 at 21:49
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.