I do a lot of statistical analysis with R, and heavily utilize large multicore instances on AWS. Mostly for hyperparameter searches, cross-validation, and bootstrapping.

Say I have an instance with c cores, and a job with r >= c replicates, which are farmed out to c cores at a time. Now, because of system process (such as my ssh client running htop), there are jobs besides my c replicates running.

This means, as far as I understand the working of the operating system, that there is some process that is shutting off my jobs so that htop (and whatever else) can access the processors. After giving these various processes some time in the sun, my jobs resume.

When I look at htop, I see a lot of red mixed with the green. Is it accurate to say that the green is my work, and the red is background stuff done to enable my work?

Intuitively, it seems that this sort of shuffling would be sub-optimal. So here is my direct question: if I have access to c cores, should I allocate my replicate jobs to all c of them, or perhaps c-1 or something?

I also imagine that there are a lot of details about how compute resources are assigned to jobs that I don't understand and am glossing over. What would be involved in having all of my jobs go to c-1 cores and all of the system processes go the the cth core? Would that turn all of my htop green, except for one bar? And would this make any sense?

I suppose I could do benchmarking experiments, but this would be tough with huge instances and datasets, and I'm not sure what I'd learn given how many things will be application-specific. So I want to understand better how things work.


It is hard to know the exact effect on a particular application without experimentation, BUT the general rule of thumb is that exceeding the number of cores by a small amount is beneficial (for instance most compiling guides suggest calling make with number of cores/threads + 1), but exceeding it by a large amount is likely counterproductive due to the extra overhead. The reason for this is if one (or a couple) of the tasks is sleeping waiting for I/O or timers or whatever, that the other threads can still proceed.

The work shuffling (OS scheduling) happens on all modern operating systems and is something we should work with, rather than fight it. If there seems to be something unrelated competing, you can drop the nice level of your process, but on a dedicated AWS instance... Its hard to imagine that being needed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.