0

For right sizing Linux hosts, it's important to know the backlog of things that are pending. procs_running (particularly when divided by the number of cores) is a good indicator of process backlog, but does not provide insight into thread backlog. Everything can look honky-dory with load average, but there may be significant pressure on the cpu based on pending threads. This can occur, for example, with a single process that listens on a socket and handles multiple socket connections by spawning multiple threads, each of those threads placing a load on the cpu. In such a case, there is only one process running, therefore the load average and procs_running is low, yet the cpu is pegged at 100% and latency for client workload is high.

Is there any way to get insight into the size of such a thread queue - ie. threads that are waiting for runtime on the cpu?

2 Answers 2

0

The equivalent of procs_running for threads is procs_running itself. It contains the value returned by nr_running, which is the number of running threads.

(As a general rule, as far as the kernel is concerned threads are processes.)

3
  • no, this doesn't answer my question about determining the backlog of threads that have accumulated waiting for runtime. Oct 7, 2022 at 19:05
  • Could you elaborate? procs_running gives the number of runnable threads, which I thought was what you were looking for. Oct 7, 2022 at 19:08
  • Check out my answer which I'm posting shortly Oct 7, 2022 at 19:15
0

After poking around, I can basically answer my own question.

the -L option to ps includes all of the process's threads. Watching this on my host, I notice that when clients begin to connect to my service over the network, procs_running never goes above 4 (1 per core), but the number of threads in Sleep state associated with the process starts climbing up from 78 to 140, and doesn't return to 78 until all clients have finished their requests. This takes about an hour and during this time all cores are constantly pegged at 100% Util. Load average never climbs above 4.

So the -L option is, in fact, a good indicator of backlog for in this particular case where you have a single process that's serving network requests. In this case neither procs_running (nor load avg, which is directly related) indicates the size of the backlog. This is because due to the fact that there is but one parent process that spins off new threads according to the number of clients connecting, and procs_running only reflects the queue size of the parent process.

The only thing I don't understand is why the threads are marked as in Sleep state. I would figure that they should be Runnable since they are waiting for cpu time. If anyone knows the answer to this, let me know.

As I mentioned in the question, the reason it's important to be able to measure backlog is for right-sizing instances. It's not enough only to measure current utilization.

EDIT: This isn't satisfactory. Normally you'd expect each thread to contribute 1 to the backlog (procs_running) but that's not what's happening here. The only thing I can conclude is that the application is doing its own thing internally, like issuing sleeps() on its own threads. Other than the number of established sockets, I can't see any way for the Operating system to know the backlog of work that's pending.

By the way, the application in question is puppetserver.

2
  • If the threads are sleeping, then as far as the kernel is concerned they aren’t waiting for the CPU. The information given in your question and answer isn’t sufficient to determine why that is the case. What do you mean by “procs_running only reflects the queue size of the parent process”? Have you looked at the links in my answer? Oct 7, 2022 at 20:16
  • I'm looking for a way to quantify the backlog. If you've got a way, I'd like to hear it. Oct 8, 2022 at 1:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .