4

I have a time and resource consuming task I am running every once in a while (make), which can be run on any one of a servers pool.

Problem is, I am not the only user running make, and when two or more unaware users run make simultaneously, it takes forever, and sometimes the server even crashes. So we decided that whenever someone wants to run make, he or she must first ssh into one of the servers, make sure make is not already running on it by other user (using linux's w), and only then does he or she start make.

In my attempts to automate the process of choosing a server, I wrote a simple script, that loops on the servers pool, ssh into every one of them, and chooses the first server for which the output of w has no make in it, but this approach is too naive because it ignores the following:

  • Each server has different attributes (e.g. one with 12 CPUs, and the other with 80 CPUs)
  • make is not the only task that these servers run
  • w only shows processes of users that are logged through ssh, and while make is run using ssh most of the time, it may well be that someone is running make from the server itself.

I want to change the criteria for choosing a server, but not sure what it should be.
I looked online and found the top command, but again, I am not sure what should be considered.
For example, I thought about using the criteria: $(top -bn 1 | grep 'Cpu\(s\)' | gawk '{print $2+$3+$4+$6+$7+$8}') to determine which one is the least busiest right now, but this ignores the server's attributes. There could be a busier server with considerably more CPUs.

1

1 Answer 1

0

You should look for a distributed task/job scheduler, cluster management system, or cloud management. A lot of these already exist; cas pointed to several in his comments and Google will turn up plenty more.

I suspect you all will be much happier once you've deployed one, and no longer have to worry about stepping on each others' toes all the time. Also, you should fix your servers so that mistakes just cause the task to fail, not bring down the machine.

If you insist on building your own (something I confess I've done—though it was 15 years ago), tasks broadly speaking consume several different types of resources, and you probably want to consider which ones your tasks take:

  • memory (RAM) [from your goes really slow or crashes, I'd guess this is a big one for your make task]
  • disk I/O bandwidth
  • disk I/O operations per second (seeks)
  • disk space
  • CPU time
  • GPU time
  • network bandwidth

You can check memory usage via free, disk I/O via iostat, space with free, CPU usage via cat /proc/loadavg (on Linux) or uptime, top, ps, etc.

But of course, checking the current numbers has a problem—maybe your make job first does a few simple things taking a few minutes, then fires off the huge process that takes gigs of RAM. This could happen:

  1. Alice runs the script to start a "make" task.
  2. Script checks serverA, sees it has tons of free RAM, low CPU usage, starts task on serverA.
  3. Short while later, Bob runs the script to start some other RAM-intensive task.
  4. Alice's task hasn't gotten to the resource intensive phase yet. So when the script checks serverA, it still has tons of free RAM. Starts Bob's task on serverA as well.
  5. Bob's task eats up most of the free RAM on serverA
  6. Alice's task finally gets to the RAM-intensive part, but there is no RAM available now. Uh-oh! ServerA thrashes to death.

Yeah, the above comes from experience writing one (though for what I was using mine for, it was CPU time).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .