5

To quote the 2013 systemd announcement of the new control group interface (with emphasis added):

Note that the number of cgroup attributes currently exposed as unit properties is limited. This will be extended later on, as their kernel interfaces are cleaned up. For example cpuset or freezer are currently not exposed at all due to the broken inheritance semantics of the kernel logic. Also, migrating units to a different slice at runtime is not supported (i.e. altering the Slice= property for running units) as the kernel currently lacks atomic cgroup subtree moves.

So, what's broken about the inheritance semantics of the kernel logic for cpuset (and how does this brokenness not apply to other cgroup controllers such as cpu)?

There is an article on RedHat's website giving an unverified solution for how to use cgroup cpusets in RHEL 7 despite their lack of support as easy-to-manage systemd unit properties...but is this even a good idea? The bolded quotation above is concerning.

To put it another way, what are the "gotchas" (pitfalls) that could apply to using cgroup v1 cpuset which are being referenced here?


I'm starting a bounty on this.

Possible sources of information to answer this question (in no special order) include:

  1. cgroup v1 documentation;
  2. kernel source code;
  3. test results;
  4. real-world experience.

One possible meaning of the bolded line in the quote above would be that when a new process is forked it does not stay in the same cpuset cgroup as its parent, or that it is in the same cgroup but in some sort of "unenforced" status whereby it may actually be running on a different CPU than the cgroup allows. However, this is pure speculation on my part and I need a definitive answer.

  • A statement like "X is broken" often translates too "I've a personal opinion about it, and according to that, it should work differently, and I don't care what the reasons were for making it the way it is". So you should ask whoever wrote this... – dirkt Sep 5 '18 at 17:43
  • @dirkt, to be fair, I can find no mention of cpusets in the kernel's cgroup v2 documentation, but there is quite a long section at the end under the header "Issues with v1 and Rationales for v2." cgroup v1 was poorly abstracted and complicated; no one will argue against that. So it's probably correct that there are/were issues with cgroup v1 cpusets. I just would like to understand them more fully and learn what alternatives exist. – Wildcard Sep 5 '18 at 18:59
  • 2
    See the Frequently Given Answer Linux control groups are not jobs. for more on the freezer. – JdeBP Sep 6 '18 at 10:21
4
+75

I'm not nearly well-versed enough with cgroups to give a definitive answer (and I certainly don't have experience with cgroups going back to 2013!) but on a vanilla Ubuntu 16.04 cgroups v1 seems to have it's act together:

I devised a small test that forces forking as a different user using a child sudo /bin/bash spun off with & - the -H flag is extra paranoia to force sudo to execute with root's home environment.

cat <(whoami) /proc/self/cgroup >me.cgroup && \
sudo -H /bin/bash -c 'cat <(whoami) /proc/self/cgroup >you.cgroup' & \
sleep 2 && diff me.cgroup you.cgroup

This yields:

1c1
< admlocal
---
> root

For reference, this is the structure of cgroup mounts on my system:

$ mount | grep group
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
$
  • 2
    Welcome to Unix StackExchange! Please note, that the editor here does not use Markdown syntax for code blocks, they can be created by indenting every line by 4 spaces. Use the {} button to indent selected text accodingly. This will allow you to include the diff. – crater2150 Sep 5 '18 at 9:01
3

At least one definite and unresolved issue with cpusets is documented in the kernel bug tracker here:

Bug 42789 - cpuset cgroup: when a CPU goes offline, it is removed from all cgroup's cpuset.cpus, but when it comes online, it is only restored to the root cpuset.cpus

To quote one comment from the ticket (I'm adding the hyperlinks to the actual commits, and removing the IBM email address in case of spambots):

This was independently reported by Prashanth Nageshappa...and fixed in commit 8f2f748b0656257153bcf0941df8d6060acc5ca6, but subsequently reverted by Linus as commit 4293f20c19f44ca66e5ac836b411d25e14b9f185. According to his commit, the fix caused regressions elsewhere.

The fix commit (that was later reverted) describes the issue well:

Currently, during CPU hotplug, the cpuset callbacks modify the cpusets to reflect the state of the system, and this handling is asymmetric. That is, upon CPU offline, that CPU is removed from all cpusets. However when it comes back online, it is put back only to the root cpuset.

This gives rise to a significant problem during suspend/resume. During suspend, we offline all non-boot cpus and during resume we online them back. Which means, after a resume, all cpusets (except the root cpuset) will be restricted to just one single CPU (the boot cpu). But the whole point of suspend/resume is to restore the system to a state which is as close as possible to how it was before suspend.


The same asymmetric hotplug issue is described with further insight into how it relates to inheritance, in:

Bug 188101 - process scheduling in cpuset of cgroup is not working properly.

Quoting that ticket:

When cpuset of a container (docker/lxc both uses underlying cgroup) becomes empty (due to hotplug/hotunplug) then processes running in that container can be scheduled on any cpus in cpuset of its nearest non-empty ancestor.

But, when cpuset of a running container (docker/lxc) becomes non-empty from an empty state (adding cpu to the empty cpuset) by updating the cpuset of the running container (by using echo method), the processes running in that container still uses the same cpuset as that of its nearest non-empty ancestor.


While there may be other issues with cpuset, the above is enough to understand and make sense of the statement that systemd doesn't expose or utilize cpuset "due to the broken inheritance semantics of the kernel logic."

From these two bug reports, not only do CPUs not get added back to a cpuset after resume, but even when they are (manually) added, the processes in that cgroup will still be left running on CPUs potentially disallowed by the cpuset.


I found a message from Lennart Poettering that directly confirms this as the reason (bolding added):

On Wed, 2016-08-03 at 16:56 +0200, Lennart Poettering wrote:

On Wed, 03.08.16 14:46, Dr. Werner Fink (werner at suse.de) wrote:

problem with v228 (and I guess this is also later AFAICS from logs of current git) that repeating CPU hotplug events (offline/online). The root cause is that cpuset.cpus become not restored by machined. Please note that libvirt can not do this as it is not allowed to do so.

This is a limitation of the kernel cpuset interface, and it's one of the reasons we do not expose cpusets at all in systemd right now. Thankfully, there's an alternative to cpusets, which is the CPU affinity controls exposed via CPUAffinity= in systemd, which do much of the same, but have less borked semantics.

We'd like to support cpusets directly in systemd, but we don't do this as long as the kernel interfaces are as borked as they are. For example, cpusets are flushed out entirely currently when the system goes through a suspend/resume cycle.

1

What's “broken” about cpuset cgroup inheritance semantics in the Linux kernel?

"Note that the number of cgroup attributes currently exposed as unit properties is limited. This will be extended later on, as their kernel interfaces are cleaned up. For example cpuset or freezer are currently not exposed at all due to the broken inheritance semantics of the kernel logic. Also, migrating units to a different slice at runtime is not supported (i.e. altering the Slice= property for running units) as the kernel currently lacks atomic cgroup subtree moves."

So, what's broken about the inheritance semantics of the kernel logic for cpuset (and how does this brokenness not apply to other cgroup controllers such as cpu)?

The bolded quotation above is concerning. To put it another way, what are the "gotchas" (pitfalls) that could apply to using cgroup v1 cpuset which are being referenced here?

Really short answer: The code doesn't multi-process well, different processes use and free PIDs returning them to the pool before their children's PIDs have terminated - leaving upstream to believe that the PID's children are active so skip that PID, but that PID ought not to have been reissued before terminating the children. In short, poor locks.

Services, scopes and slices may be created freely by the administrator or dynamically by programs. That can interfere with the default slices setup by the OS during startup.

With Cgroups a process and all its children draw resources from the containing group.

And so much more ... leading to a lengthy answer ...

A number of people have expressed their concerns:

  1. "Linux control groups are not jobs" (2016) by Jonathan de Boyne Pollard:

    An operating system kernel that provides a "job" abstraction provides a way of cancelling/killing an entire "job". Witness the Win32 TerminateJobObject() mechanism, for example.

    When systemd terminates all of the processes in a cgroup, it doesn't issue a single "terminate job" system call. There isn't such a thing. It instead sits in a loop in application-mode code repeatedly scanning all of the process IDs in the cgroup (by re-reading a file full of PID numbers) and sending signals to new processes that it hasn't seen before. There are several problems with this.

    • systemd can be slower than whatever is reaping child processes within the process group, leading to the termination signals being sent to completely the wrong process: one that just happened to re-use the same process ID in between systemd reading the cgroup's process list file and it actually getting around to sending signals to the list of processes. ...

    ...

    • A program that forks new processes quickly enough within the cgroup can keep systemd spinning for a long time, in theory indefinitely as long as suitable "weather" prevails, as at each loop iteration there will be one more process to kill. Note that this does not have to be a fork bomb. It only has to fork enough that systemd sees at least one more new process ID in the cgroup every time that it runs its loop.

    • systemd keeps process IDs of processes that it has already signalled in a set, to know which ones it won't try to send signals to again. It's possible that a process with ID N could be signalled, terminate, and be cleaned out of the process table by a reaper/parent; and then something within the cgroup fork a new process that is allocated the same process ID N once again. systemd will re-read the cgroup's process ID list, think that it has already signalled the new process, and not signal it at all.

     

    These are addressed by a true "job" mechanism. But cgroups are not such. cgroups were intended as an improvement upon the traditional Unix resource limit mechanisms, addressing some of their long-standing and well-known design flaws. They weren't designed to be the equivalent of a VMS or a Windows NT Job Object.

    No, the freezer is not the answer. Not only does systemd not use the freezer, but the systemd people explicitly describe it as having "broken inheritance semantics of the kernel logic". You'll have to ask them what they mean by that, but the freezer does not, for them, magically turn cgroups into a job mechanism either.

    Moreover: This is not to mention that Docker and others will manipulate the freeze status of control groups for their own purposes, and there is no real race-free mechanism for sharing this setting amongst multiple owners, such as an atomic read-and-update for it.
    Permission is hereby granted to copy and to distribute this web page in its original, unmodified form as long as its last modification datestamp is preserved.

    • TerminateJobObject() function

      Terminates all processes currently associated with the job. If the  
      job is nested, this function terminates all processes currently  
      associated with the job and all of its child jobs in the hierarchy. 
      
    • Windows NT Job Objects

      A job object allows groups of processes to be managed as a unit.  
      Job objects are namable, securable, sharable objects that control  
      attributes of the processes associated with them. Operations  
      performed on a job object affect all processes associated with the  
      job object. Examples include enforcing limits such as working set   
      size and process priority or terminating all processes associated 
      with a job.
      

    The answer offered in Jonathan's explanation is:

    systemd's Resource Control Concepts

    ...

    Service, scope and slice units directly map to objects in the cgroup tree. When these units are activated they each map to directly (modulo some character escaping) to cgroup paths built from the unit names. For example, a service quux.service in a slice foobar-waldo.slice is found in the cgroup foobar.slice/foobar-waldo.slice/quux.service/.

    Services, scopes and slices may be created freely by the administrator or dynamically by programs. However by default the OS defines a number of built-in services that are necessary to start-up the system. Also, there are four slices defined by default: first of all the root slice -.slice (as mentioned above), but also system.slice, machine.slice, user.slice. By default all system services are placed in the first slice, all virtual machines and containers in the second, and user sessions in the third. However, this is just a default, and the administrator my freely define new slices and assign services and scopes to them. Also note that all login sessions automatically are placed in an individual scope unit, as are VM and container processes. Finally, all users logging in will also get an implicit slice of their own where all the session scopes are placed.

    ...

    As you can see, services and scopes contain process and are placed in slices, and slices do not contain processes of their own. Also note that the special "-.slice" is not shown as it is implicitly identified with the root of the entire tree.

    Resource limits may be set on services, scopes and slices the same way. ...

Follow the links above for the unabridged explanation.

  1. "Cgroups v2: resource management done even worse the second time around" (Oct 14 2016), by davmac:

    ...

    You can create nested hierarchy such that there are groups within other groups, and the nested groups share the resources of their parent group (and may be further limited). You move a process into a group by writing its PID into one of the group’s control files. A group therefore potentially contains both processes and subgroups.

    The two obvious resources you might want to limit are memory and CPU time, and each of these has a “controller”, but there are potentially others (such as I/O bandwidth), and some Cgroup controllers don’t really manage resource utilisation as such (eg the “freezer” controller/subsystem). The Cgroups v1 interface allowed creating multiple hierarchies with different controllers attached to them (the value of this is dubious, but the possibility is there).

    Importantly, processes inherit their cgroup membership from their parent process, and cannot move themselves out of (or into) a cgroup unless they have appropriate privileges, which means that a process cannot escape its any limitations that have been imposed on it by forking. Compare this with the use of setrlimit, where a process’s use of memory (for example) can be limited using an RLIMIT_AS (address space) limitation, but the process can fork and its children can consume additional memory without drawing from the resources of the original process. With Cgroups on the other hand, a process and all its children draw resources from the containing group.

    ...

    cgroup controllers implemented a number of knobs which would never be accepted as public APIs because they were just adding control knobs to system-management pseudo filesystem. cgroup ended up with interface knobs which were not properly abstracted or refined and directly revealed kernel internal details.

    These knobs got exposed to individual applications through the ill-defined delegation mechanism effectively abusing cgroup as a shortcut to implementing public APIs without going through the required scrutiny.

    ...

    cgroup v1 allowed threads to be in any cgroups which created an interesting problem where threads belonging to a parent cgroup and its children cgroups competed for resources. This was nasty as two different types of entities competed and there was no obvious way to settle it. Different controllers did different things.

  2. Also see the cgroup v2 docs: "Issues with v1 and Rationales for v2":

    Multiple Hierarchies

    cgroup v1 allowed an arbitrary number of hierarchies and each hierarchy could host any number of controllers. While this seemed to provide a high level of flexibility, it wasn’t useful in practice.

    For example, as there is only one instance of each controller, utility type controllers such as freezer which can be useful in all hierarchies could only be used in one. The issue is exacerbated by the fact that controllers couldn’t be moved to another hierarchy once hierarchies were populated. Another issue was that all controllers bound to a hierarchy were forced to have exactly the same view of the hierarchy. It wasn’t possible to vary the granularity depending on the specific controller.

    In practice, these issues heavily limited which controllers could be put on the same hierarchy and most configurations resorted to putting each controller on its own hierarchy. Only closely related ones, such as the cpu and cpuacct controllers, made sense to be put on the same hierarchy. This often meant that userland ended up managing multiple similar hierarchies repeating the same steps on each hierarchy whenever a hierarchy management operation was necessary.

    Furthermore, support for multiple hierarchies came at a steep cost. It greatly complicated cgroup core implementation but more importantly the support for multiple hierarchies restricted how cgroup could be used in general and what controllers was able to do.

    There was no limit on how many hierarchies there might be, which meant that a thread’s cgroup membership couldn’t be described in finite length. The key might contain any number of entries and was unlimited in length, which made it highly awkward to manipulate and led to addition of controllers which existed only to identify membership, which in turn exacerbated the original problem of proliferating number of hierarchies.

    Also, as a controller couldn’t have any expectation regarding the topologies of hierarchies other controllers might be on, each controller had to assume that all other controllers were attached to completely orthogonal hierarchies. This made it impossible, or at least very cumbersome, for controllers to cooperate with each other.

    In most use cases, putting controllers on hierarchies which are completely orthogonal to each other isn’t necessary. What usually is called for is the ability to have differing levels of granularity depending on the specific controller. In other words, hierarchy may be collapsed from leaf towards root when viewed from specific controllers. For example, a given configuration might not care about how memory is distributed beyond a certain level while still wanting to control how CPU cycles are distributed.

Please see section 3's link for more information.

  1. Communication between Lennart Poettering (systemd Developer) and Daniel P. Berrange (Redhat) on Wed, 20.07.16 12:53 retrieved from the systemd-devel archives titled: "[systemd-devel] Confining ALL processes to a CPUs/RAM via cpuset controller":

    On Wed, 20.07.16 12:53, Daniel P. Berrange (berrange at redhat.com) wrote:

    For virtualized hosts it is quite common to want to confine all host OS processes to a subset of CPUs/RAM nodes, leaving the rest available for exclusive use by QEMU/KVM. Historically people have used the "isolcpus" kernel arg todo this, but last year that had its semantics changed, so that any CPUs listed there also get excluded from load balancing by the schedular making it quite useless in general non-real-time use cases where you still want QEMU threads load-balanced across CPUs.

    So the only option is to use the cpuset cgroup controller to confine procosses. AFAIK, systemd does not have an explicit support for the cpuset controller at this time, so I'm trying to work out the "optimal" way to achieve this behind systemd's back while minimising the risk that future systemd releases will break things.

    On Wed, Jul 20, 2016 at 03:29:30PM +0200, Lennart Poettering replied:

    Yes, we don't support this as of now, but we'd like to. The thing though is that the kernel interface for it is pretty borked as it is right now, and until that's not fixed we are unlikely going to support this in systemd. (And as I understood Tejun the mem vs. cpu thing in cpuset is probably not going to stay the way it is either).

    Next message

    On Wed, 20.07.16 14:49, Daniel P. Berrange (berrange at redhat.com) wrote:

    cgroupsv2 is likely to break many things once distros switch over, so I assume that wouldn't be done in a minor update - only a major new distro release so, not so concerning.

I hope that clarifies things.

  • Thanks, it’s good to have this noted here. I already read almost all of this in my research. However, it doesn’t answer the question as nothing here is specific to cpusets or freezer; also systemd works with both cgroup v1 and cgroup v2, but does NOT interact with the cpuset controller in any way that I can find. (Also cgroup v2 doesn’t have any cpuset controller.) Again I emphasize: since all of this is pertinent to ALL cgroups (some of it just to all v1 cgroups), NONE of it answers why systemd devs shunned cpuset cgroups specifically, while happily integrating with other cgroups. – Wildcard Sep 7 '18 at 9:55
  • "CPU" is mentioned twice in my answer. I'll be able to rereview this in a few hours. – Rob Sep 7 '18 at 11:48
  • @Wildcard - Added section #4. – Rob Sep 7 '18 at 16:45
  • @Rob There is an excellent cgroup series on lwn.net/Articles/604609 . The final part of the series (lwn.net/Articles/608425 ) has a discussion about unification and mentions an inconsistency with the cpuset controller. – Anon Sep 8 '18 at 6:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.