I am trying to understand if setting cpu.cpu_quota_us in cpu cgroup subsytem has any impact on application performance. Essentially by reducing the CPU quota, but increasing the number of CPUs such that "effective" CPUs are still the same, would it impact the application? For example, is 4 CPU 100% quota configuration same as 8 CPU 50% quota configuration?

I know this depends a lot on the application design and whether its cpu or io bound. Here I am only concerned about CPU intensive applications.

My effort:

I wrote a simple C application available here https://github.com/ashu-mehra/cpu-quota-test.

This program creates 'N' threads. Each thread starts computing prime numbers starting between number 'n' and 1000000. The starting number 'n' is different for each thread. After computing 100 prime numbers, the thread sleeps for fixed duration of time. Once the thread reaches 1000000, it starts over from 2. At the end, the main thread displays the cumulative number of primes calculated by each thread. I treat this as the "throughput" of this sample application.

I ran this program under following configurations:

  1. In a cgroup which has 4 cpus and no limit on quota.
  2. In a cgroup which has 8 cpus and 50% quota.

I disabled the hyperthreads by setting /sys/devices/system/cpu/cpu/online` to 0.

For each configuration I varied the number of threads from 4 to 32. Following are the results for the "throughput" generated by the sample program. Numbers are average of 10 iterations.

threads    cpu4quota100    cpu8quota50
4          66229.5         66079.4
8          128129          129768
16         189247          134882
24         188238          98917.8
32         176236          87252.5

Notice there is big difference in throughput between two cases from thread 16 onwards. For 24 and 32 threads, throughput dropped considerably for "cpu8quota50" case.

I have the perf stat results for these runs as well. I noticed cpu-migrations reported by perf vary a lot between these two configurations. Here is the comparison

threads    cpu4quota100    cpu8quota50
4          9.6             11.2
8          3252.2          37.9
16         2956.2          4490.5
24         472.6           2347
32         118.3           1727.2

Numbers for threads 4, 8 and 16 make sense but I can't comprehend the numbers for thread 24 and 32 for "cpu4quota100" case which are way less than thread 16 case.

Can some one provide explanation for these results? Also, does "cpu-migration" have any impact on application performance?

Sorry for the long post!

Edit 1:

I updated my script for running above mentioned sample program to time the execution using time command to see if there is any difference between "cpu4quota100" and "cpu8quota50" cases. I did the run for 32 threads only, and these are the results:

time    cpu4quota100    cpu8quota50
user    119.956 secs    120.076 secs
sys     0.001 secs      0.009 secs
CPU     386.2%          386.5%

So not much difference in user and sys time in the two cases, but the "throughput" is twice in cpu4quota100 case compared to cpu8quota50 case.

Edit 2:

It seems changing kernel governor for CPU frequency helped in improving the throughput of cpu8quota50 case. Earlier numbers were obtained when frequency governor "powersave" was in use. With "powersave" CPU frequency of the cores in case of cpu4quota100 shot to maximum but for cpu8quota50 it was much lower. However, after changing frequency governor to "performance", CPU frequency in case of cpu8quota50 was also close to max. For 32 threads running with "performance" as frequency governor I get following numbers:

threads    cpu4quota100    cpu8quota50
32         175804          163831

So the difference has now come down from nearly 50% to 6.8% only.

But its interesting to note the difference in behavior of "powersave" governor in the two cases as mentioned above. I am not sure if it is working as expected in cpu8quota50 case.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.