I do a lot of statistical analysis with R, and heavily utilize large multicore instances on AWS. Mostly for hyperparameter searches, cross-validation, and bootstrapping.
Say I have an instance with
c cores, and a job with
r >= c replicates, which are farmed out to
c cores at a time. Now, because of system process (such as my ssh client running
htop), there are jobs besides my
c replicates running.
This means, as far as I understand the working of the operating system, that there is some process that is shutting off my jobs so that
htop (and whatever else) can access the processors. After giving these various processes some time in the sun, my jobs resume.
When I look at
htop, I see a lot of red mixed with the green. Is it accurate to say that the green is my work, and the red is background stuff done to enable my work?
Intuitively, it seems that this sort of shuffling would be sub-optimal. So here is my direct question: if I have access to
c cores, should I allocate my replicate jobs to all
c of them, or perhaps
c-1 or something?
I also imagine that there are a lot of details about how compute resources are assigned to jobs that I don't understand and am glossing over. What would be involved in having all of my jobs go to
c-1 cores and all of the system processes go the the
cth core? Would that turn all of my htop green, except for one bar? And would this make any sense?
I suppose I could do benchmarking experiments, but this would be tough with huge instances and datasets, and I'm not sure what I'd learn given how many things will be application-specific. So I want to understand better how things work.