In case I have 8 powerful machines, what is the proper way to distribute and balance the load from user applications and scientific experiments?

Is virtualization right way to go, or is there any kind of load manager software to implement this?

  • 1
    What is your use case? How are these servers going to be used? What frameworks will your cluster need to support? – Jedi Jun 19 '16 at 15:17

Look into batch queueing software like TORQUE, PBS, Slurm etc. It lets you create a queue where users submit their tasks, and the software runs that on the machines available. Please note that this approach works well only with non-interactive stuff.

  • 1
    These are scheduler, not really a way to distribute the experiments. If you have infiniband, go for an MPI implementation (like OpenMPI): you should be able to declare your cluster and run your jobs on it. The schedulers are only useful if you need to share that cluster with somebody else. If you don't have infiniband, then any message passing system is good enough (MPI if it is already built-in your software) otherwise ZeroMQ, raw sockets, ssh, jupyter (Python), etc. – JohnW Oct 1 '16 at 16:09
  • I think the question can be read both ways and unfortunately, rkioji didn't provide further info when asked for it. Anyway, the point is moot now as this answer was accepted, hence it worked for the asker. – Alexander Batischev Oct 3 '16 at 10:37
  • I understand that: the point was not for you but for future readers that may understand the question the way I did (work distribution across nodes) and waste hours figuring out these schedulers won't help them at all. Therefore it is a complement rather than a complain. – JohnW Oct 3 '16 at 13:12
  1. What you want is kind of the opposite of virtualisation, a cluster.

    I'll try to explain the difference as simply as possible.

    Virtualisation is having multiple virtual machines running on one (or more) physical machines. This allows you to make more efficient use of your existing hardware, by running multiple completely separate virtual machines on the one physical computer.

    A cluster allows you to use one or more physical machines (called "nodes") to run your application. This allows you to use all of the CPU power in the cluster on computation jobs, co-ordinated by a batch scheduler.

    These two things can be combined - e.g. I used to work on a project called Nectar Research Cloud which used Openstack-based virtualisation to allow scientists and other researchers around Australia create anything from single-VMs to huge clusters of virtual machines on demand.

  2. As well as the batch queuing software like slurm, or torque, pbs mentioned by @Alexander Batischev, you probably also need MPI libraries or similar installed.

    If the kind of computation you need to run can, for example, be divided into smaller chunks that can processed independently, with the results combined later, you'll need MPI libraries. Your code will also need to be written to use them so that different instances of your process running on different nodes in the cluster can communicate with each other.

    Many of the common and popular applications in various scientific and other academic disciplines have MPI support (or similar, possibly proprietary, alternatives) built in because they are intended to be run on clusters.

  3. Alexander mentions that clusters work best with non-interactive software....that's true, but for many apps there are graphical front-ends for creating, viewing, and editing the data, creating and submitting the batch job files to the cluster (using slurm or torque etc), and viewing the results.

    The key point is that, for your main computation jobs, you don't just run them and interact with them directly. You (or your front-end software) create a batch script and submit that to the queue for execution. It will be queued and won't execute until the cluster has enough free resources to run your job (which might be almost immediately or, on a very busy cluster, not for days or even weeks). When the job finishes executing, it will typically save both a log file AND one or more results files which can be loaded into your desktop front-end program for viewing.

    The batch script can specify all sorts of things to let the batch scheduler know what to do with it - e.g. how much ram or disk space it needs, how many cpus or cpu cores and/or what kind (e.g. does it require 1 or more GPU nodes)?, whether it can run with MPI across multiple nodes or can only run on one node.

Here's an interesting blog post on setting up an HPC cluster from scratch - the focus is on Computational Chemistry but there's no reason the same kind of setup couldn't be used for other disciplines, the cluster described is built on Debian, and Debian has an enormous library of scientific software.

  • MPI is something very specific - to programming. I was looking for something such as slurm. I just want to balance the load of some users on demand on the machines I have. – rph Jun 20 '16 at 19:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.