Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I want to build a distributed computing system to run Matlab, C and other programming languages for scientific computing. Now I've several old Mac machines with Lion Mac OS installed acted as web servers or personal computers. I also have one latest 16-Xeon-core machine to be installed with Linux. I've not decided which Linux distribution should we use for the new machine, but we need to consider the following factors. Please help me to decide which Linux distribution I can use, what distributing computing software I can use and how to manage the data backups and queue assignments.

  1. All machines with either Mac or Linux OS's can be served as a cluster system for parallel or distributed computing. To be specific, we want to run programs crossing machines in a queue with multiple users and threads. In the case that all machines are not symmetric, but we don't want to lower the speed of the most powerful machine.

  2. The new machine is preferred to be used as a head node, but at least a secondary machine should also be able to act as a head node in case the head node was shut down.

  3. Backup process should be easy to setup, and can be controlled remotely. This is not as important as the first two factors. At least we can backup important data manually.

I searched Google already, but I haven't found decent solutions for my case. Thank you in advance for your suggestions!

share|improve this question
add comment

1 Answer 1

up vote 0 down vote accepted

Having set up a few HPC clusters in my time, I can tell you that what you want to do is going to cause you an enormous amount of trouble with compatibility issues between nodes in the cluster - which is probably why you can't find a direct answer via google.

These compatibility issues include differences in versions of software, system libraries, numerical and computation libraries, C & Fortran etc compilers (and their libraries), PATH & LD_LIBRARY_PATH etc variables, differences between GNU and non-GNU versions of shell utilities, possibly CUDA vs OPENCL (or versions of same) for GPGPU computation, and much much more.

You would run into many of these issues just using two different distros of linux (or even different versions of the same distro on different nodes of the cluster).

You may find that it's simply easier to set up two clusters - one with a single node (the Xeon running linux), and one with several nodes (old Macs running OS X Lion)

However, if that's not an option, the most important thing to consider is the scheduler, not the linux distribution.

I personally wouldn't want to set up what you want, but if I had to, I wouldn't consider using PBS or Torque, I'd use Slurm. Slurm has a lot more fine-grained control over what applications can be run on which nodes. Oracle's Grid Engine is another option that may do what you want, but I'm not familiar enough with it to do more than mention the fact that it exists.

share|improve this answer
    
How about the backup option? Make two separated backup server as well? –  Xiaodong Qi Jan 17 at 16:27
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.