I've read that the color red indicates "kernel processes." Does that mean little daemons that are regulating which task gets to use the CPU? And by extension, transaction costs in an oversubscribed system?
I'm running some large-scale geoprocessing jobs, and I've got two scripts running in parallel at the same time.
The first script does the actual processing, on all 96 cores. It is responsible for almost all of the memory use.
The second script uses
curl to download the data to feed the first process, and it does so in parallel. I wrote it to download only until there are
n_cores * 3 files downloaded. If that constraint isn't met, it waits for a minute or so and then check again. So, most of the time it isn't running -- or rather it is executing the
Sys.sleep() command in
I've experimented with using fewer cores for the downloading process. When I do so, it can't keep up with the processing script (I'm DLing from S3).
TL;DR: Would my processes run faster if I could make
htop less red? And are they red because there are more processes than cores?