1

I have a machine with 768 GiB of ECC DDR4 ram. One of the modules is flaky and needs replacing: dmesg fills up with entries like:

    [Mar25 08:07] mce: [Hardware Error]: Machine check events logged
[  +0.000014] EDAC skx MC1: HANDLING MCE MEMORY ERROR
[  +0.000003] EDAC skx MC1: CPU 0: Machine Check Event: 0x0 Bank 8: 0xdc00008001010092
[  +0.000002] EDAC skx MC1: TSC 0x116da2b2e75b2
[  +0.000001] EDAC skx MC1: ADDR 0x2e9282d900
[  +0.000001] EDAC skx MC1: MISC 0x200801c091001086
[  +0.000004] EDAC skx MC1: PROCESSOR 0:0x50657 TIME 1616659657 SOCKET 0 APIC 0x0
[  +0.000013] EDAC MC1: 2 CE memory read error on CPU_SrcID#0_MC#1_Chan#2_DIMM#0 (channel:2 slot:0 page:0x2e9282d offset:0x900 grain:32 syndrome:0x0 -  OVERFLOW err_code:0x0101:0x0092 socket:0 imc:1 rank:1 bg:2 ba:2 row:0xfc0b col:0x40)

and there are correctable recovered ECC errors:

$ edac-util --report=simple
mc0: Correctable errors:   0
mc0: Uncorrectable errors: 0
mc1: Correctable errors:   270
mc1: Uncorrectable errors: 0
mc2: Correctable errors:   0
mc2: Uncorrectable errors: 0
mc3: Correctable errors:   0
mc3: Uncorrectable errors: 0
Total CE: 270
Total UE: 0

Whilst dmidecode shows me lots of wonderful information about the system and its ram, I don't understand the relationship between the (logical) addresses shown in dmesg's output and that of dmidecode('s physical addresses).

Specifically, whilst it looks like the kernel is telling me the error occurred at the address 0x289282d900, this must be a logical address because the upper end of physical addresses is below that. The highest memory mapped physical address my system has is:

Handle 0x004F, DMI type 19, 31 bytes
Memory Array Mapped Address
        Starting Address: 0x09000000000
        Ending Address: 0x0BFFFFFFFFF
        Range Size: 192 GB
        Physical Array Handle: 0x004E
        Partition Width: 3

Similarly, the helpful information of channel:2 slot:0 does help me narrow down which dimm needs replacing, but dmidecode doesn't list any channels at all -- but its full output does helpfully state which physical address range corresponds to the (labelled) socket on the motherboard:

$ sudo dmidecode -t memory|grep -i channel
$

Similarly the old trick of grep -i 'System RAM' /proc/iomem doesn't work:

00000000-00000000 : System RAM
00000000-00000000 : System RAM
00000000-00000000 : System RAM
00000000-00000000 : System RAM
00000000-00000000 : System RAM
00000000-00000000 : System RAM

Irritatingly, my motherboard's manual doesn't actually specify how the detail of how this (and indeed numa nodes!) are laid out.

Can I find remotely out which physical stick of ram is dodgy?

2 Answers 2

1

This line should be all that you need:

[  +0.000013] EDAC MC1: 2 CE memory read error on CPU_SrcID#0_MC#1_Chan#2_DIMM#0 (channel:2 slot:0

To read this one:

CPU_SrcID#0=(CPU0)MC#1=(Memory Controller 1

your edac-util list posted shows

0-3)Chan#2_DIMM#0=(Would be C0 DIMM).

Memory Channels should start at 0(A0/A1), 1(B0/B1), 2(C0/C1), etc. DIMM0 would be primary slot in that channel.

Should be able to find the same info with:

edac-util -v

which reports on UE/CE.

1

Due to memory bank interleaving and mapping it is very difficult to find out which module is used by an operation.

How about removing half of the modules, then see whether problems persists? Keep bisecting until you have the culprit. In order to speed up the search, run (== boot into) memtest86+ when testing to force the error to appear. Many Linux installation DVDs (e.g. Ubuntu) have a memtest86+ image you can easily boot.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .