As part of my current research, I've been doing tests where I disable/re-enable cores by echoing
0 or 1 to /sys/devices/system/cpu/cpuN/online to simulate vertical scaling.
With Kernel v4.4.0 (Ubuntu 16.04 LTS), this works perfectly on multiple Xeon E3 machines. Recently, I had the chance to test dual-socket EPYC 7281 machines, also on Ubuntu 16.04, with v4.4.0. Disabling cores on these systems worked just fine. When re-enabling them, however, they were shown as online by lscpu and htop, but no work was ever scheduled on them again until I rebooted the machine, i.e. when I stressed the machine, htop showed that only cores that had not been disabled and re-enabled before were being used.
In other words: On a Xeon machine, i first disable all cores but core #0, then re-enable them all again. Afterwards, everything runs normally as it did before disabling cores. On the EPYC machines, cores that had been disabled were never used again, even after re-enabling them and even though they were shown as being available by htop and lscpu.
I can no longer access the EPYC machines to test this further myself (only had them for 2 weeks and wasn't allowed to upgrade to newer kernel), but we're thinking about buying new hardware and I need to find out whether Threadripper/EPYC (Zen in general) can generally not disable/re-enable cores via the kernel, or whether this was just some weird isolated problem.
My question(s) is (are):
Could cores not coming online properly be a problem of the older 4.4. kernel, so would this issue be fixed if I were to upgrade to a newer kernel? Unfortunately, I can't test this, as I no longer have access to the EPYC machines (otherwise, this post would not be here :)). I would, however, have full control over our new hardware. So if this issue is fixed by a newer kernel, I'd be happy.
If not due to the kernel version, why would EPYCs behave in this way, and what could I do to remedy this?
I tried searching for answers, but so far haven't found anything conclusive, other than the fact that apparently there have been lots of patches for Zen-based CPUs since kernel v4.10 and later.
I looked at this link, which gives a few details about how the process works for Intel CPUs, and even links to the actual code responsible for disabling cores. Unfortunately, that didn't help a lot, at least I didn't understand enough to answer my questions.
This didn't help either, as I need to vertically scale cores during runtime, without rebooting.
I also found some references to a new power management system in newer kernels, but that went way over my head :-/
Or maybe instead of digging and finding the root of the problem, it would be enough if anyone who currently has a Threadripper or EPYC system with a recent kernel could maybe run a quick test by disabling/re-enabling cores and then loading them to check whether they are properly being utilized?
I would really appreciate any insight and help with this problem!
I'll be here to give further details if necessary, I don't know what else could be needed to explain this behavior. Also, is this the right exchange to ask this question, or would you recommend any other means of asking for help?
