11

enter image description here

I've read that the color red indicates "kernel processes." Does that mean little daemons that are regulating which task gets to use the CPU? And by extension, transaction costs in an oversubscribed system?

I'm running some large-scale geoprocessing jobs, and I've got two scripts running in parallel at the same time.

The first script does the actual processing, on all 96 cores. It is responsible for almost all of the memory use.

The second script uses curl to download the data to feed the first process, and it does so in parallel. I wrote it to download only until there are n_cores * 3 files downloaded. If that constraint isn't met, it waits for a minute or so and then check again. So, most of the time it isn't running -- or rather it is executing the Sys.sleep() command in R.

I've experimented with using fewer cores for the downloading process. When I do so, it can't keep up with the processing script (I'm DLing from S3).

TL;DR: Would my processes run faster if I could make htop less red? And are they red because there are more processes than cores?

5
  • What kind of processing does your script do? The "red" could indicate I/O time, if I'm not mistaken, so it would be interesting to know whether you consider it normal for your script to do lots of it. Either way, others may recommend better tools than htop to find out. (perf comes to mind.)
    – dhag
    Commented Feb 20, 2018 at 16:40
  • Does reading from netcdfs count as I/O? If so then yes. Will look into perf. Commented Feb 20, 2018 at 17:40
  • 3
    I found this post by looking up "htop" in Google Images and... can I just say, holy cow... I want your machine ._.
    – Jack Bauer
    Commented Apr 27, 2019 at 22:05
  • @Jack Bauer Amazon rents them Commented Apr 28, 2019 at 0:29
  • @generic_user I assumed it was some such scenario. Still makes me jealous :p
    – Jack Bauer
    Commented Apr 29, 2019 at 5:04

1 Answer 1

16

Red represents the time spent in the kernel, typically processing system calls on behalf of processes. This includes time spent on I/O. There’s no point in trying to reduce it just for the sake of reducing it, because it’s not time that’s wasted — it’s time that’s spent by the kernel doing useful stuff (as long as you’re not thrashing, so look at the number of context switches etc.).

I've experimented with using fewer cores for the downloading process. When I do so, it can't keep up with the processing script (I'm DLing from S3).

suggests that your current setup is evenly balanced between the I/O needed to feed the processing, and the processing itself, which is a rather nice result. If you suspect that you’ve got too many processes running, and that that’s causing waste (by thrashing), then you could try reducing the number of geoprocessing jobs, to see if your overall throughput increases. The usual benchmarking tips apply: identify what you’re going to tweak, determine what resulting variations could occur and what they mean, only tweak one thing at a time, and measure everything.

2
  • Thanks a lot! Indeed, my R script makes lots of calls to system("some bash command"). Until you answered my question, I assumed that those processes would come out green, instead of red, on htop. I bet that R's netcdf libraries also come out red. You also introduced me to the term "thrashing", which I hadn't come across in this context before. Commented Feb 20, 2018 at 18:12
  • 1
    @generic_user FYI, system calls in this context are functions exposed by the kernel to userspace. These are the fundamental operations that any program can perform, such as sys_read or sys_open. R's system() function doesn't make a system call, rather, executes another userspace process
    – Outurnate
    Commented Feb 20, 2018 at 21:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .