33 of 53 Expanded explanation of C1/C1E.

As I accumulated more information about Core 2 CPU power states ("C-states"), it started to become apparent that the C-states supported in the Core 2 line of processors are far more complex than those in both earlier and later processors. At the time the intel_idle driver was released, the code was not particularly mature and several Core 2 processors had been released that had conflicting C-state support.

Some compelling information on Core 2 Solo/Duo C-state support was found in this article from 2006. This is in relation to support on Windows, however it does indicate the robust hardware C-state support on these processors:

...the quad-core Intel Core 2 Extreme (Kentsfield) processor supports all five performance and power saving technologies — Enhanced Intel SpeedStep (EIST), Thermal Monitor 1 (TM1) and Thermal Monitor 2 (TM2), old On-Demand Clock Modulation (ODCM), as well as Enhanced C States (CxE). Compared to Intel Pentium 4 and Pentium D 600, 800, and 900 processors, which are characterized only by Enhanced Halt (C1) State, this function has been expanded in Intel Core 2 processors (as well as Intel Core Solo/Duo processors) for all possible idle states of a processor, including Stop Grant (C2), Deep Sleep (C3), and Deeper Sleep (C4).

This article from 2008 outlines support for per-core C-states on multi-core Intel processors, including Core 2 Duo and Core 2 Quad (additional helpful background reading was found in this white paper from Dell):

A core C-state is a hardware C-state. There are several core idle states, e.g. CC1 and CC3. As we know, a modern state of the art processor has multiple cores, such as the recently released Core Duo T5000/T7000 mobile processors, known as Penryn in some circles. What we used to think of as a CPU / processor, actually has multiple general purpose CPUs in side of it. The Intel Core Duo has 2 cores in the processor chip. The Intel Core-2 Quad has 4 such cores per processor chip. Each of these cores has its own idle state. This makes sense as one core might be idle while another is hard at work on a thread. So a core C-state is the idle state of one of those cores.

I found a 2010 presentation from Intel that provides some additional background about the intel_idle driver, but unfortunately does not explain the lack of support for Core 2:

This EXPERIMENTAL driver supersedes acpi_idle on Intel Atom Processors, Intel Core i3/i5/i7 Processors and associated Intel Xeon processors. It does not support the Intel Core2 processor or earlier.

The above presentation does indicate that the intel_idle driver is an implementation of the "menu" CPU governor, which has an impact on Linux kernel configuration (i.e., CONFIG_CPU_IDLE_GOV_LADDER vs. CONFIG_CPU_IDLE_GOV_MENU). The differences between the ladder and menu governors are succinctly described in this answer.

I also found this article, which conveniently summarizes the differences in C-state support for a variety of Intel processors (up to Core 2 Duo):

Mode   Name                   CPUs
C0     Operating State        All CPUs
C1     Halt                   486DX4 and above
C1E    Enhanced Halt          All socket LGA775 CPUs
C1E    —                      Turion 64, 65-nm Athlon X2 and Phenom CPUs
C2     Stop Grant             486DX4 and above
C2     Stop Clock             Only 486DX4, Pentium, Pentium MMX, K5, K6, K6-2, K6-III
C2E    Extended Stop Grant    Core 2 Duo and above (Intel only)
C3     Sleep                  Pentium II, Athlon and above, but not on Core 2 Duo E4000 and E6000
C3     Deep Sleep             Pentium II and above, but not on Core 2 Duo E4000 and E6000; Turion 64
C3     AltVID                 AMD Turion 64
C4     Deeper Sleep           Pentium M and above, but not on Core 2 Duo E4000 and E6000 series; AMD Turion 64
C4E/C5 Enhanced Deeper Sleep  Core Solo, Core Duo and 45-nm mobile Core 2 Duo only
C6     Deep Power Down        45-nm mobile Core 2 Duo only

From this table, it appears that there were a variety of differences in C-state support with the Core 2 processors (Note that nearly all Core 2 processors are Socket LGA775, except for Core 2 Solo SU3500, which is Socket BGA956 and Merom/Penryn processors. "Intel Core" Solo/Duo processors are one of Socket PBGA479 or PPGA478).

An additional exception to the table was found in this article:

Intel’s Core 2 Duo E8500 supports C-states C2 and C4, while the Core 2 Extreme QX9650 does not.

Interestingly, the QX9650 is a Yorkfield processor (Intel family 6, model 23, stepping 6). For reference, my Q9550S is Intel family 6, model 23 (0x17), stepping 10, which supposedly supports C-state C4 (confirmed through experimentation). Additionally, the Core 2 Solo U3500 has an identical CPUID (family, model, stepping) to the Q9550S but is available in a non-LGA775 socket, which confounds interpretation of the above table.

Clearly, the CPUID must be used at least down to the stepping in order to identify C-state support for this model of processor, and in some cases that may be insufficient (undetermined at this time).

The method signature for assigning CPU idle information is:

#define ICPU(model, cpu) \
{ X86_VENDOR_INTEL, 6, model, X86_FEATURE_ANY, (unsigned long)&cpu }

Where model is enumerated in asm/intel-family.h. Examining this header file, I see that Intel CPUs are assigned 8-bit identifiers that appear to match the Intel family 6 model numbers:

#define INTEL_FAM6_CORE2_PENRYN 0x17

From the above, we have Intel Family 6, Model 23 (0x17) defined as INTEL_FAM6_CORE2_PENRYN. This should be sufficient for defining idle states for most of the Model 23 processors, but could potentially cause issues with QX9650 as noted above.

So, minimally, each group of processors that has a distinct C-state set would need to be defined in this list.

Zagacki and Ponnala, Intel Technology Journal 12(3):219-227, 2008 indicate that Yorkfield processors do indeed support C2 and C4. They also seem to indicate that the ACPI 3.0a specification supports transitions only between C-states C0, C1, C2 and C3, which I presume may also limit the Linux acpi_idle driver to transitions between that limited set of C-states. However, this article indicates that may not always be the case:

Bear in mind that is the ACPI C state, not the processor one, so ACPI C3 might be HW C6, etc.

Also of note:

Beyond the processor itself, since C4 is a synchronized effort between major silicon components in the platform, the Intel Q45 Express Chipset achieves a 28-percent power improvement.

The chipset I'm using is indeed an Intel Q45 Express Chipset.

My interpretation of the above table (combined with a table from Wikipedia, asm/intel-family.h and the above articles) is:

Model 9 0x09 (Pentium M and Celeron M):

  • Banias: C0, C1, C2, C3, C4

Model 13 0x0D (Pentium M and Celeron M):

  • Dothan, Stealey: C0, C1, C2, C3, C4

Model 14 0x0E INTEL_FAM6_CORE_YONAH (Enhanced Pentium M, Enhanced Celeron M or Intel Core):

  • Yonah (Core Solo, Core Duo): C0, C1, C2, C3, C4, C4E/C5

Model 15 0x0F INTEL_FAM6_CORE2_MEROM (some Core 2 and Pentium Dual-Core):

  • Kentsfield, Merom, Conroe, Allendale (E2xxx/E4xxx and Core 2 Duo E6xxx, T7xxxx/T8xxxx, Core 2 Extreme QX6xxx, Core 2 Quad Q6xxx): C0, C1, C1E, C2, C2E

Model 23 0x17 INTEL_FAM6_CORE2_PENRYN (Core 2):

  • Merom-L/Penryn-L: ?
  • Penryn (Core 2 Duo 45-nm mobile): C0, C1, C1E, C2, C2E, C3, C4, C4E/C5, C6
  • Yorkfield (Core 2 Extreme QX9650): C0, C1, C1E, C2E?, C3
  • Wolfdale/Yorkfield (Core 2 Quad, C2Q Xeon, Core 2 Duo E5xxx/E7xxx/E8xxx, Pentium Dual-Core E6xxx, Celeron Dual-Core): C0, C1, C1E, C2, C2E, C3, C4

From the amount of diversity in C-state support within just the Core 2 line of processors, it appears that a lack of consistent support for C-states may have been the reason for not attempting to fully support them via the intel_idle driver. I would like to fully complete the above list for the entire Core 2 line.

This is not really a satisfying answer, because it makes me wonder how much unnecessary power is used and excess heat has been (and still is) generated by not fully utilizing the robust power-saving states on these processors.

As a test, I inserted the following at intel_idle.c line 127:

static struct cpuidle_state core2_cstates[] = {
    {
        .name = "C1",
        .desc = "MWAIT 0x00",
        .flags = MWAIT2flg(0x00),
        .exit_latency = 3,
        .target_residency = 6,
        .enter = &intel_idle,
        .enter_s2idle = intel_idle_s2idle, },
    {
        .name = "C1E",
        .desc = "MWAIT 0x01",
        .flags = MWAIT2flg(0x01),
        .exit_latency = 10,
        .target_residency = 20,
        .enter = &intel_idle,
        .enter_s2idle = intel_idle_s2idle, },
//  {
//      .name = "C2",
//      .desc = "MWAIT 0x10",
//      .flags = MWAIT2flg(0x10),
//      .exit_latency = 20,
//      .target_residency = 40,
//      .enter = &intel_idle,
//      .enter_s2idle = intel_idle_s2idle, },
    {
        .name = "C2E",
        .desc = "MWAIT 0x11",
        .flags = MWAIT2flg(0x11),
        .exit_latency = 40,
        .target_residency = 100,
        .enter = &intel_idle,
        .enter_s2idle = intel_idle_s2idle, },
    {
        .name = "C3",
        .desc = "MWAIT 0x20",
        .flags = MWAIT2flg(0x20) | CPUIDLE_FLAG_TLB_FLUSHED,
        .exit_latency = 85,
        .target_residency = 200,
        .enter = &intel_idle,
        .enter_s2idle = intel_idle_s2idle, },
    {
        .name = "C4",
        .desc = "MWAIT 0x30",
        .flags = MWAIT2flg(0x30) | CPUIDLE_FLAG_TLB_FLUSHED,
        .exit_latency = 100,
        .target_residency = 400,
        .enter = &intel_idle,
        .enter_s2idle = intel_idle_s2idle, },
    {
        .enter = NULL }
};

at intel_idle.c line 983:

static const struct idle_cpu idle_cpu_core2 = {
    .state_table = core2_cstates,
    .disable_promotion_to_c1e = false,
};

at intel_idle.c line 1073:

ICPU(INTEL_FAM6_CORE2_PENRYN, idle_cpu_core2),

After a quick compile and reboot of my PXE nodes, dmesg now shows:

[    0.019845] cpuidle: using governor menu
[    0.515785] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.543404] intel_idle: MWAIT substates: 0x22220
[    0.543405] intel_idle: v0.4.1 model 0x17
[    0.543413] tsc: Marking TSC unstable due to TSC halts in idle states deeper than C2
[    0.543680] intel_idle: lapic_timer_reliable_states 0x2

And now PowerTOP is showing:

          Package   |            CPU 0
POLL        2.5%    | POLL        0.0%    0.0 ms
C1E         2.9%    | C1E         5.0%   22.4 ms
C2E         0.4%    | C2E         0.2%    0.2 ms
C3          2.1%    | C3          1.9%    0.5 ms
C4         89.9%    | C4         92.6%   66.5 ms

                    |            CPU 1
                    | POLL       10.0%  400.8 ms
                    | C1E         5.1%    6.4 ms
                    | C2E         0.3%    0.1 ms
                    | C3          1.4%    0.6 ms
                    | C4         76.8%   73.6 ms

                    |            CPU 2
                    | POLL        0.0%    0.2 ms
                    | C1E         1.1%    3.7 ms
                    | C2E         0.2%    0.2 ms
                    | C3          3.9%    1.3 ms
                    | C4         93.1%   26.4 ms

                    |            CPU 3
                    | POLL        0.0%    0.7 ms
                    | C1E         0.3%    0.3 ms
                    | C2E         1.1%    0.4 ms
                    | C3          1.1%    0.5 ms
                    | C4         97.0%   45.2 ms

I've finally accessed the more advanced Core 2 C-states, and now that my cluster is rebooted the fan noise has dropped to almost nothing in this room. And it looks like there is a measurable drop in power consumption - my meter on 8 nodes appears to be averaging at least 5% lower (with one node still running the old kernel), but I'll try swapping the kernels out again as a test.

The C1 state appears to be ignored in lieu of C1E, and I'm not sure what to make of that, so I commented out C2 as well. I'm uncertain if C1/C1E and C2/C2E can be used together or if they are redundant. I found a note in this presentation that states:

C1E is only used when all the cores are in C1E

Also, the flags, latency and residency values could probably stand to be fine-tuned, but just taking educated guesses based on the nehalem idle values seems to work fine. More reading will be required to make any improvements.

I tested this on a Dual Core Pentium E5300 (Wolfdale) and a Core 2 Duo E7400 (Wolfdale) and Core 2 Quad Q9550S (Yorkfield), and I have found no issues.

Not covered by this driver modification:

  • The original Core Solo/Core Duo (Yonah, non Core 2) are family 6, model 14. This is good because they supported the C4E/C5 (Enhanced Deep Sleep) C-states but not the C1E/C2E states and would need their own idle definition.
  • Core 2 Duo E4000 (Conroe) and Core 2 Duo E6000 (Allendale, essentially a sabotaged Conroe) are family 6, model 15. Also good, because these processors lack support for C-state C4.
  • Only partial support for the Core 2 Duo Mobile 45nm (Penryn), which should work but this won't use the additional C4E/C5 and C6 states supported by that processor (ironic because Linux incorrectly defines model 23 as "PENRYN" when a more suitable name would be "CORE2"). However, this driver modification may still be better than using the acpi_idle driver, which can only support three states (I'm uncertain which hardware states would be selected - this would depend on the BIOS).

The only issues that I can think of are:

  • Core 2 Solo SU3300/SU3500 (Penryn-L) are family 6, model 23 and will be detected by this driver. However, they are not Socket LGA775 so they may not support the C1E Enhanced Halt C-state. Likewise for the Core 2 Solo ULV U2100/U2200 (Merom-L). This may well be the case with all Merom/Penryn processors, but I can't find any helpful documentation on these and I don't have one of these to test.
  • Core 2 Extreme QX9650 (Yorkfield) reportedly does not support C-state C2 or C4. I'm skeptical of this because this is contrary to all other documentation I've found.

In conclusion, it turns out that there was no real reason for the lack of Core 2 support in the intel_idle driver. It is clear now that the original stub code for "Core 2 Duo" only handled C-states C1 and C2, which would have been far less efficient than the acpi_idle function which also handles C-state C3. Once I knew where to look, implementing support was easy. The helpful comments and other answers were much appreciated, and if Amazon is listening, you know where to send the check.

This update has been committed to github. I will e-mail a patch to the LKML soon.