Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have to launch a total number, NUMTOT of a program (could be any, C, Mathematica, ns-2... mathematical environments). The problem is that the computer where these processes run only admits a maximum number of executions, e.g. MAX, (MAX < NUMTOT).

I run these programs with the following line in a terminal

./run.sh -d directory num

repeatedly, where run.sh is a script from which I invoke the math program; directory is the directory where my program is, and num indicates the number of new executions ( integer*num = NUM). The script 'run.sh' makes 'num' directories with a copy of 'directory' (the program) and launch 'num' number of instances of the program. In my case are Matlab instances.

I would like an script that automatically keeps track of the number of processes running and when the maximum number is reduced to (MAX-num), because 'num' has already finished, then launch the next 'num' simulations of Matlab until the total NUM of simulations is completed.

It would even better if the script could be able to maintain the MAX possible processes just invoking the next program when only one has just finished.

Actually, to know the number of running processes I run the following script

ps axu |grep plopez|grep simulacion|grep MATLAB|awk '{ $2} END{print NR}'

which gives the total Matlabs running in this moment.

How could I make the whole set of simulations just from a single script?

share|improve this question
    
An easier way to get the number of MATLAB instances is pgrep -cu plopez [MATLAB executable name], pgrep is designed for exactly this kind of thing. –  terdon Sep 18 '13 at 14:57

5 Answers 5

The usual way to enforce a maximum number of processes by user is via pam_limits(8). Check the limits.conf(5) manpage for the details, and examples to change your /etc/security/limits.conf accordingly.

Specifically you want to configure the nproc parameter:

 nproc
    maximum number of processes

An example from the manpage with this parameter in use:

       @faculty        soft    nproc           20
       @faculty        hard    nproc           50
share|improve this answer
    
My account don't have limited the number of process. I don't know why Matlab crashes when I start more than let say 32 instances. –  Pablo Sep 18 '13 at 15:25
    
Matlab dumps in a file with references to addresses of memory. First impression was some java problem so I run matlab without java with "nohup matlab -nodisplay -nodesktop -nosplash" options. But still has 32 as limit of running instances. –  Pablo Sep 18 '13 at 15:31
    
Is there anything useful in Matlab's logs? are you sure you have enough RAM/CPU to run that many processes? –  dawud Sep 18 '13 at 15:42
    
The machine has 48 CPUs and more than 100 GB of memory with SO Ubuntu. Matlab is v11a. The dump has many lines. I would upload to somewhere. Let me prepare a link to easily download it. –  Pablo Sep 18 '13 at 15:48
    

Instead of checking the number of running instances periodically, I would suggest writing a script that runs the program again and again until NUM is reached and start this script MAX times in parallel.

You could also trap the SIGCHLD signal which is emitted when a child process exits. This has to be enabled first with set -o monitor. A more detailed example and alternatives can be found in a thread on Stackoverflow.

share|improve this answer
    
The computer only admits a maximum number 32 instances of Matlab but I have to launch 1200. Now I have to wait for example that 16 has been finished and then launch a set of 16 new instances. Could you tell me something else about how to make the script? –  Pablo Sep 18 '13 at 14:53
    
My account don't have limited the number of process. I don't know why Matlab crashes when I start more than let say 32 instances. –  Pablo Sep 18 '13 at 15:25
    
Do you know if a while or for loops could be paused to wait some processes to finish? –  Pablo Sep 18 '13 at 15:35
    
Waiting for any one of multiple subprocesses is rather difficult in bash. In higher programming languages as c or python, it is much easier. The thread I linked has a good discussion. –  XZS Sep 18 '13 at 23:17

If your MATLAB jobs are non-interactive (i.e. running a batched script) then you may want to look into batch scheduling software like Torque or Slurm - this kind of resource management is exactly what they are designed to do.

These are normally used on HPC clusters, but in your case you'd have a cluster with only one compute node.

Slurm has the advantage of being a modern, well-designed system taking advantage of years of previous experience with other schedulers. Torque/PBS has the advantage of familiarity - lots of people have already used it on other clusters. Slurm has a set of compatibility wrappers so that people used to PBS or Torque can start using it immediately with familiar commands.

Either would give you many options to control scheduling of jobs - by cpu cores, memory, available licenses, and more. they also provide accounting if you need to keep track of, or limit, cumulative usage per user. Slurm, for example, has a fair-usage weighting option that can be used to prioritise jobs by people who haven't run many jobs recently over people who have used the system a lot - which is often better than a fixed hard quota of total cputime.

Another option to get around limited licenses is to write your MATLAB scripts so that they are compatible with GNU Octave as well as MATLAB. Then you can run as many test/development runs as you need, and do the final run in MATLAB if that is what is required to publish your results or collaborate with other researchers. Octave is mostly compatible with MATLAB, but there are some differences - here's a summary.

share|improve this answer

I believe that the ulimit command (assumming shell) or the limit command (assumming csh varient) has a limit for the number of processes.

share|improve this answer
    
ulimit returns me 'unlimited' –  Pablo Sep 18 '13 at 15:58

An answer in another thread dealing with limiting the number of subprocesses in bash shows the following snippet, which will wait until one of 4 jobs completed, checking every ten seconds.

while [ $(jobs | wc -l) -ge 4 ] ; do sleep 10 ; done

Keep in mind that such an active sleep strategy is rather inefficient, crude and imprecise.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.