0

Please excuse the complete ignorance of my question - I'm probably lacking the right terminology here...

I am trying to understand how a single process (R session) can spill over to all CPUs on our server (n=72). It seems to happen when the memory required for a given object or operation surpasses a given threshold.

This behavior can be seen in the following screenshot from htop, showing that all CPUs are being utilized:

enter image description here

During this point in time, both "marc" and "ismael" were running a single session on R, as can be seen from the output of top:

enter image description here

Ismael's process has exceeded some limit that results in multi-threading, where >5000% of CPU is being used. While this behavior might be desired for large calculations, I get the feeling that it is making a lot of things slow down on the entire server. It's not even that clear to me from occasions where this has happened to me that this multi-threading is improving the speed of the calculation.

Can someone please explain to me what is going on, and if there are any settings that we might adjust in order to improve the performance of job distributions on the server?

Many thanks in advance.

7
  • Have you tried asking ismael to limit how many CPU cores his R jobs use? sometimes it's both easy and effective to set guidelines on excessive use of a shared resource. BTW, are these jobs run directly by the user, or via a batch scheduling/resource-management system like slurm or torque?
    – cas
    Commented Sep 11, 2019 at 8:18
  • @cas - I have had the same behavior happen to me; where, while running a single R job, the processing spills over to many CPUs. As I mentioned, this is dependent upon the size of the job within a single R session. Yes, the jobs are run by a single user - calling R in the console opens the program. Commented Sep 11, 2019 at 8:23
  • yes, but that's a function of the code you're running. most, if not all, libraries that provide support for parallelising your code allow you to place limits on how many threads will be started. depending on what you're running, and the lib you're using, the default may be to use as many cores as are available.
    – cas
    Commented Sep 11, 2019 at 8:29
  • 1
    You may want to look into cgroups. I’ve got a couple questions/answers posted about them on this site. Also, ulimits, and renice. These are suggestions for areas to research, since I don’t know the best answer for your situation.
    – Wildcard
    Commented Sep 11, 2019 at 9:23
  • 1
    @Wildcard - cgroups does indeed seem to be a good possible solution here. Thanks for the suggestion - I will look into this further. Commented Sep 11, 2019 at 11:58

1 Answer 1

1

I don't know what's going on here or what's the question or how to help, but y'all should start using nice to launch your batched processes.

The nice program allows a nice user to declare a job low priority, so e.g. it would give priority to other people's shells (unless for some reason they are "nicer"), in order to make everyone else's life easier.

Run htop and watch this:

Run this:

stress --cpu 4

Open another tab, run this:

nice -n 19 stress --cpu 4

You'll see the latter only works when there's CPU time available left from other batches not so nicer.

Yes I'd say you want Ismael to launch is huge batch with "high niceness".

If he wouldn't, you can run htop with super user privileges, if you can, and make his batch nicer even from the very htop program.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .