2

A task must receive the same resource allocation, and no more, regardless of whether the rest of the system is idle. The aim is to achieve a predictable time to complete the task irrespective of other load on the system.

Experimenting with systemd, using CPUAllocation and CPUQuota, with a simple single-threaded test program shows that it takes longer when there are fewer running.

The process, when unrestricted, takes 15 seconds to run. When it's restricted to 20% of one CPU, running five instances each take around 75 seconds in line with expectations (because 5*15=75).

This leads to the expection that each instance should take 75 seconds to run when restricted to 20% of one CPU regardless of the number of instances (to maximum of five for one CPU).

However, running increasingly fewer instances (leaving some of the CPU capacity unused) takes increasingly longer than the expected 75 seconds, and running a single instance (with no other system load) takes 250 seconds.

This does not make sense: a single process should take 75 seconds, the same time that running five takes.

If another process is run to "soak up" the unused CPU capacity the the fewer number of instances complete in the expected 75 seconds.

Why does this happen?

and

Is there a more appropriate way to achieve the desired effect of ensuring that a task takes a similar time to execute regardless of other system load?

(systemd was used whilst experimenting but a direct usage of cgroups may be more appropriate, perhaps more so).


systemd service unit file used for testing

$ cat /etc/systemd/system/cputest\@.service

[Service]
User=nobody
SyslogIdentifier=cputest
ExecStart=/usr/local/bin/cputest.rb
CPUAffinity=1
CPUQuota=20%

Test program is a deliberately inefficient algorithm for Pi:

$ cat cat /usr/local/bin/cputest.rb

#!/usr/bin/ruby
PRECISION = 10 ** 8 # decimal places
PI_TARGET = (Math::PI * PRECISION).to_i
d  = neg = 1
pi_test = pi = 0
START = Time.now.to_f
def elapsed() Time.now.to_f - START end
while pi_test != PI_TARGET do
  pi += neg * (4.0/d)
  pi_test = (pi * PRECISION).to_i
  d+=2
  neg*=-1
end
puts elapsed()

Run with

for i in {1..4}; do sudo systemctl start cputest@$i; done

Collect output with

sudo journalctl -xft cputest
  • 1
    Interesting....however from having a look at the results from a search on red hat docs (sorry, on iPad - this will look messy), “Setting CPUQuota to 20% ensures that the executed processes never get more than 20% CPU time on a single CPU.” it says maximum, not precise. – Mylo Mylo Jun 11 at 12:53
  • 1
    Can you share a reproducer for this issue? You might want to consider reporting an issue in the systemd bug tracker, you might get better support there than here... Yes, it's possible the issue is with underlying cgroups, but reporting it to systemd is still appropriate, since it's possible a problem exists there. – filbranden Jun 11 at 13:55
  • 1
    Is there a chance that this has something to do with CPU throttling? It sounds a little like the CPU is idle so something is telling it to run slower. Maybe investigate with linux.die.net/man/1/cpufreq-info – Philip Couling Jun 11 at 15:01
  • 2
    @filbranden I've added what I used to perform the test. I'll leave this open for a little while to see if others here have any insight and then ask the systemd folks for any ideas. – starfry Jun 11 at 16:45
  • 2
    This might be affected by TimerSlackNSec=(50usecs?) (see what man systemd-system.conf says about it (and man prctl). For alternatives to systemd you might look at man 7 sched and the Linux-specific SCHED_DEADLINE policy to do your own scheduling. Or realtime processes using SCHED_FIFO and timer_create() for an interval timer counting cpu time (CLOCK_THREAD_CPUTIME_ID) and signalling when it is over. – meuh Jun 11 at 17:43
1

It would appear that the slowness of an under-utilized CPU is due to power saving techniques applied by the CPU. When the CPU is maxed out, it is allowed to turbo boost and run faster. When capped at 20% it isn't maxed out and therefore runs at a slower speed.

I was developing on a laptop which had one of these CPUs. I got hold of a server and ran the tests again and got the results that I was expecting.

I haven't done any more testing beyond trying the "performance" governor but this made no difference. I haven't gone into BIOS settings or anything like that. I think this behaviour is something to be aware of when limiting the cpu of processes on a system designed to save power.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.