1

I would like to buy a cluster (30-60 nodes) for work and install linux on it. The cluster should have a queue system like PBS, for instance. It should also be capable of doing parallel computation. I myself don't know too much about configuring clusters and maintaining them, but our IT guys know even less and I prefer to be the one in charge with these machines.

My question is: is there some simple (software) way to save power on these machines by turning off or putting nodes to sleep while no job is running? And could be that connected to the queue manager?

Sorry for not being more clear, but it's the first time I have to be sys admin on anything with more than two processors.

|improve this question
  • 1
    You may need to look at hiring someone with more experience. That is quite a lot of funds to shell out, with no staff to back it up. Clusters can be tricky, and if your IT team can't support them it's probably not the best way to go. BTW "... but our IT guys know even less and I prefer to be the one in charge with these machines..." is a very scary line. There are many considerations. like power, cooling, disaster, networking, et al that need consideration and 30 node cluster is a bit large to be a "hobby" or "learning" setup. – coteyr Apr 8 '15 at 17:45
  • I cannot do the hiring/firing in the IT department however much I wish that to be the case. And I really need a cluster of that size for a future project. There exists already a cluster at my work, but it was maintained and configured by one of my colleagues, not the IT people. He just doesn't have the time for this anymore. The problem is not to buy it, but to prove to the administration it is not too expensive to maintain and operate. I did find something: CLUES – Magicsowon Apr 8 '15 at 18:09
1

It's a tricky question and would likely require some code development and thought on your part and the part of the people you're building the cluster for:

  • When should a node be shut down?
  • When should it be brought up?
  • When the node comes up, are you sure it's synced with other nodes (does it need software updates?)

Aside from bringing in pros a good way to save a bit of electricity is to look at some of the power saving utilities available for Linux and start using those. PowerTOP is a good start to see where you can save. More specifically things like hdparm can spin the disks down for you (maybe). Lastly, you can take a look at cpufreq to set CPU speeds and throttle the speed down to save electricity when nothing else is running. Not all CPUs support this so you may not be able to scale all that much.

I generally kept my cluster up 24x7 because it was in such high use. If you're really that concerned that you won't be at 100% usage most of the time, you may want to look at on-demand cloud clusters like StarCluster that gives you a ready-made cluster of an arbitrary size for however long you want.

|improve this answer
  • I think the nodes should be shut down if for a period (10-30min) there is nothing waiting in the queue. As soon as someone submits a job to the queue, they should be turned on. As for syncing of the nodes, I guess that could be complicated, and most likely I can't think of all the possible problems. I would simply turn all cluster on and do updates manually when I would have to. – Magicsowon Apr 8 '15 at 19:34
  • Shutting down after a period of time is easy - just build a map that watches PBS nodes and tracks when the last job completed on that node. 30 minutes later, ssh over and run 'shutdown -h now'. Powering back up will be a bit more difficult. You will need to use some sort of out-of-band management (like IPMI) to have the remote system power itself back up. That's just keeping a map of what's been powered down compared to the number of processors in use and what's available. As soon as the number needed is higher than what's available, you power a node back up. – mkomarinski Apr 9 '15 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.