Why would a process limited to a portion of one cpu run slower when the Cpu is partially idle?

Question

A task must receive the same resource allocation, and no more, regardless of whether the rest of the system is idle. The aim is to achieve a predictable time to complete the task irrespective of other load on the system.

Experimenting with systemd, using CPUAllocation and CPUQuota, with a simple single-threaded test program shows that it takes longer when there are fewer running.

The process, when unrestricted, takes 15 seconds to run. When it's restricted to 20% of one CPU, running five instances each take around 75 seconds in line with expectations (because 5*15=75).

This leads to the expection that each instance should take 75 seconds to run when restricted to 20% of one CPU regardless of the number of instances (to maximum of five for one CPU).

However, running increasingly fewer instances (leaving some of the CPU capacity unused) takes increasingly longer than the expected 75 seconds, and running a single instance (with no other system load) takes 250 seconds.

This does not make sense: a single process should take 75 seconds, the same time that running five takes.

If another process is run to "soak up" the unused CPU capacity the the fewer number of instances complete in the expected 75 seconds.

Why does this happen?

and

Is there a more appropriate way to achieve the desired effect of ensuring that a task takes a similar time to execute regardless of other system load?

(systemd was used whilst experimenting but a direct usage of cgroups may be more appropriate, perhaps more so).

---

systemd service unit file used for testing

$ cat /etc/systemd/system/cputest\@.service

[Service]
User=nobody
SyslogIdentifier=cputest
ExecStart=/usr/local/bin/cputest.rb
CPUAffinity=1
CPUQuota=20%

Test program is a deliberately inefficient algorithm for Pi:

$ cat cat /usr/local/bin/cputest.rb

#!/usr/bin/ruby
PRECISION = 10 ** 8 # decimal places
PI_TARGET = (Math::PI * PRECISION).to_i
d  = neg = 1
pi_test = pi = 0
START = Time.now.to_f
def elapsed() Time.now.to_f - START end
while pi_test != PI_TARGET do
  pi += neg * (4.0/d)
  pi_test = (pi * PRECISION).to_i
  d+=2
  neg*=-1
end
puts elapsed()

Run with

for i in {1..4}; do sudo systemctl start cputest@$i; done

Collect output with

sudo journalctl -xft cputest

Answer 1

It would appear that the slowness of an under-utilized CPU is due to power saving techniques applied by the CPU. When the CPU is maxed out, it is allowed to turbo boost and run faster. When capped at 20% it isn't maxed out and therefore runs at a slower speed.

I was developing on a laptop which had one of these CPUs. I got hold of a server and ran the tests again and got the results that I was expecting.

I haven't done any more testing beyond trying the "performance" governor but this made no difference. I haven't gone into BIOS settings or anything like that. I think this behaviour is something to be aware of when limiting the cpu of processes on a system designed to save power.