0

I've searched multiple answers here but couldn't find an answer that is related to this scenario, but if you think you do, kindly point me to it.

I am including the numbers here for the ease of my own comprehention.

I have a 96 core baremetal linux server with 256 GB RAM that's dedicated to run an in-house written distributed event-based asynchronous network service that acts as a caching server. This daemon runs with 32 worker threads. Apart from the main task of fetching and caching, this server also does a variety of related tasks in a couple of extra separate threads like polling other members' health-checks, writing metrics to a unix socket, etc. The worker threads value isn't bumped further because increasing this will increase the cache lock contention. There is not much disk activity from this server, as the metrics are tried to be written in batches and if the unix socket fails, it just ignores it and frees the memory.

This instance is a part of a 9 node cluster and the stats of this node speaks for the rest of the instances in this cluster.

With the recent surge in inbound traffic, I see the %CPU usage of the process has gone up considerably but the load average is still less than 1.

Please find the stats below.

:~$ nice top
top - 19:51:55 up 95 days,  7:27,  1 user,  load average: 0.33, 0.28, 0.32
PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
587486 cacher   20   0  107.4g  93.0g  76912 S  17.2  37.0   5038:13 cacher

The %CPU goes up till 80% some times but the load average is considerably less even then and doesn't go beyond 1.5, and this happens mostly when there's a cache miss and the cacher has to fetch it from an upstream, so it is mostly a set of network activities. As far as I understood, the compute heavy operation this service does at runtime is when it has to compute the hash of the item to be cached when it has to store it into the appropriate distributed buckets. There are no systemd limits set on this service for any parameters, and it is also tuned to disable the kernel's oomkiller for this process, although it is nowhere near the upper limit. The systemd sockets to which this binds has been tuned to accommodate more tx and rx buffer.

  • why is the load average less than 1 on a 96 core server when the %CPU for the service that uses 32 threads fluctuates between 20% and 80% consistently?
  • On a 96 core server, how much is considered a safe value for %CPU to go safely? Does it have a relation with how many threads it is used? If the number of threads are bumped, is a higher %CPU usage theoretically accepted?

Thank you.

1 Answer 1

5

There are a couple of good answers to this question on other SE sites here and here. Essentially, load average shows how many processes are waiting for a cpu core while %CPU indicates how busy the cores are.

On a 96 core server you can run all 96 cores at 100% with few issues. You probably don't want your application to take all of the resources like this since the OS and other processes need some amount or resources.

Load average is often not a good metric. If you have a large number of processes doing I/O you can have a very high load average even when the cpu utilization is low and response time for applications is very good.

1
  • Thank you. This is a dedicated server and we can afford to push it to the safe limits for %CPU utilization. I've read through the links that you've kindly provided. From my understanding, on a 96 core server with the application using 32 cores, can I assume that the maximum theoretical limit for %CPU is 3200% because it is computed over the total CPU time, or did I get it wrong? (I assume it is total because this metric is on par with the process_cpu_seconds_total metrics that we expose ) Thank you.
    – init
    Jan 8, 2023 at 6:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .