ESXi host hangs or reports ADDDC lockstep when DIMMs emit correctable ECC memory storms

Products

• VMware vSphere ESXi

Issue/Introduction

An ESXi host may suddenly stop responding to vCenter Server, showing as "Not Responding" or "Disconnected," due to physical hardware degradation in memory cells triggering a continuous stream of Corrected Memory Errors (CE).

On platforms like Cisco UCS, where the BIOS handles these errors via System Management Interrupts (SMI), high error rates saturate the CPU, causing the host to hang. The ESXi VMkernel is blinded to these errors because platform firmware does not route them to the OS. 

Symptoms

• The ESXi host appears "hung," the DCUI local console freezes, and it does not accept keyboard input.

• The host cannot be rebooted from the management UI and recovers only after a full cold power cycle through the BMC.

• Hardware Management (e.g., Cisco Intersight) may report: "Post Package Repair will be performed on this DIMM during the next system reboot" or indicate ADDDC lockstep activation.

• A Manual NMI sent from the hardware interface reports success but fails to produce a diagnostic memory dump on the ESXi side.

• In /var/run/log/vmkernel.log, no memory error or panic entries appear prior to the hang.

• The boot-time ApeiHEST line in /var/run/log/vmkernel.log shows 0 CMC, confirming firmware is not routing errors to the OS:
  
  ApeiHEST: 984: GESB lists: 1 NMI, 0 MCE, 0 CMC, 0 DMC, 1 SCI, 0 VMWPR

• In /var/run/log/vmksummary.log, hourly heartbeat lines abruptly stop and resume only after a manual power cycle: 
  
  YYYY-MM-DD HH:MM:SSZ In(14) heartbeat[#######]: up ##d##h##m##s, # VMs; ...
YYYY-MM-DD HH:MM:SSZ No(13) bootstop[#######]: Host has booted

• In /var/run/log/vmkernel.log, multiple Machine Check Architecture (MCA) errors are recorded: MCA: 202: CE Intr G0 B8 S9c00004001010091... Memory Controller Read Error on Channel 1.

• Alternatively, in /var/run/log/vmkernel.log, you see no Machine Check Exception (MCE), Corrected Machine Check Interrupt (CMCI), memory error, or panic entries in the period leading up to the hang.

• In /var/run/log/hostd.log and other userspace service logs, the last entries from the previous boot are routine activity with no error spike preceding the silence.

• The command localcli hardware ipmi sdr list shows a Memory Module sensor with a status of error.

• In the server BMC log, you see a burst of correctable ECC events attributed to a single DIMM in the minutes preceding the host hang:
  
  YYYY-MM-DD HH:MM:SS | Memory <slot-identifier> | read 50 correctable ECC errors on CPU<N> DIMM <slot> 
YYYY-MM-DD HH:MM:SS | Memory <slot-identifier> | read 925 correctable ECC errors on CPU<N> DIMM <slot>

• In the server vendor's per-DIMM error counter database, you see a non-zero correctable ECC count on exactly one DIMM slot, with every other populated slot reporting zero.

• The server BMC shows the host as healthy or "OK" in the hardware management UI even while the ESXi host is unresponsive in vCenter Server.

Environment

• VMware ESXi 7.0.x

• VMware ESXi 8.0.x

• ESXi hosts on server platforms where corrected memory errors are handled by the platform BIOS through System Management Interrupts (SMI) rather than surfaced to the operating system as Corrected Machine Check Interrupts (CMCI). This routing policy is indicated by 0 CMC in the boot-time ApeiHEST line in vmkernel.log.

Cause

• A single DIMM begins to emit single-bit memory errors at a sustained high rate. Each error is corrected by ECC at the hardware level, so no data is lost, but each correction generates a notification that the platform BIOS handles inside a System Management Interrupt (SMI). On platforms where corrected memory errors are not surfaced to the operating system as a CMCI event, the ESXi VMkernel has no visibility into the error stream and does not log it.

• When the error rate is small, the SMI overhead is negligible. When the rate grows into the hundreds of errors per second, the affected CPU spends nearly all of its cycles inside the SMI handler, leaving the ESXi VMkernel and userspace services unable to make forward progress. The host appears hung to vCenter Server, the console freezes, and management agents stop responding.

• Because the CPU is held in System Management Mode, it also cannot dispatch an NMI handler from the operating system. This is why a Manual NMI sent from the BMC does not produce a diagnostic dump even on hosts that are otherwise configured to panic on NMI.

• The condition does not appear in vmkernel.log because the corrected errors never reach the kernel. It does appear in the server BMC logs, which is where the diagnostic evidence is collected from.

Resolution

1. Collect the server vendor's hardware support log bundle from the affected host. The BMC is able to collect this bundle even when the ESXi host operating system is hung.

   • Common examples include Cisco Intersight server tech-support, Dell iDRAC SupportAssist Collection, HPE Active Health System (AHS) log, and Lenovo XClarity Controller First Failure Data Capture (FFDC). On platforms without a vendor-specific collector, a raw IPMI System Event Log (SEL) export through ipmitool sel list or the vendor's equivalent is acceptable.

2. Open the SEL or memory-error log inside the vendor bundle. Look for events of type "correctable ECC error," "corrected memory error," or the vendor's equivalent phrasing, attributed to a memory device. Record the slot designation (for example P1_F1, DIMM_A1, Proc 1 DIMM 1, depending on vendor naming convention), the timestamps, and the per-event error counts.

3. Open the vendor's per-DIMM error counter database, if available. A single-DIMM failure pattern shows a non-zero correctable error count on one slot with every other populated slot at zero. If multiple slots show non-zero error counts, this article does not apply, and a memory-controller or motherboard fault should be suspected instead.

4. Map the failing slot to its physical part using the ESXi host's SMBIOS dump (commands/smbiosDump.txt inside the ESXi support bundle). Locate the Memory Device (Type 17) entry where Location matches the failing slot, and record the Part Number, Serial, and Asset Tag fields. These identify the physical DIMM for the hardware replacement.

5. If the host is still hung, perform a cold reboot of the ESXi host through the BMC to return it to service.

   • Note: The reboot is mandatory to allow the hardware-level Post Package Repair (PPR) mechanism to map out and bypass the degraded memory cells. It also resets DIMM training and BIOS error counters, which typically allows the host to come back up cleanly even with the failing DIMM still installed.

6. Monitor the boot process to ensure the PPR sequence completes successfully.

7. Open a hardware service case with the server vendor's support organization (e.g., Cisco) to replace the failing DIMM. Provide the SEL or memory-error log entries, the per-DIMM error counter summary (if available), and the DIMM part/serial number.

8. Until the DIMM has been replaced, do not place production workload on the affected host. The failing module is still installed, and the same failure mode can recur at any time.

9. After the DIMM has been replaced, set the advanced setting Misc.NMILint1IntAction to 1 (Panic) on the affected host. With the default value of 0, ESXi takes no action on receipt of an NMI. With the value 1, ESXi panics and writes a memory dump when an NMI is received.

10. Review the server vendor's automatic DIMM isolation feature with the vendor support organization to confirm the current policy state. Current-generation server platforms expose a policy allowing the BMC to disable a DIMM with a high correctable error count at the next boot, preventing recurrence on the same module without operator intervention.

    • Feature names vary by vendor (for example, Cisco UCS exposes "DIMM block-listing" through the service profile; other vendors use "Memory Page Retire," "Bank Page Retire," "Reliable Memory," or "Advanced ECC with Page Retire"). The current policy state can be confirmed with the server vendor.

Additional Information

• For an ESXi host hang with the same surface symptoms but a different software-side cause, see ESXi host becomes unresponsive and unrebootable - requires cold boot. That article applies when the trigger is the vmsyslogd signal-handler race. The article above applies when the trigger is a hardware memory-error storm visible in the serverBMC logs.

• For instructions on collecting an ESXi host support bundle for log review, see Collecting diagnostic information for VMware products.

• For instructions on uploading collected logs to a Broadcom support case, see Uploading files to cases.

• To improve diagnostic coverage in advance of a similar incident, set Misc.NMILint1IntAction = 1 on each ESXi host so that any future NMI from the BMC has the opportunity to produce a memory dump. Consider applying the setting through a host profile for fleet consistency.

• The presence of 0 CMC in the boot-time ApeiHEST line in vmkernel.log is the platform-level signal that corrected memory errors are not visible to ESXi on the server in question.

• To improve diagnostic coverage for similar incidents, set Misc.NMILint1IntAction = 1 on each ESXi host to ensure future NMIs from the BMC can produce a memory dump. Apply the setting through a host profile for fleet consistency.

• The presence of 0 CMC in the boot-time ApeiHEST line in /var/run/log/vmkernel.log signals that corrected memory errors are not visible to ESXi on the affected server. On these platforms, review the server vendor's BMC logs as the authoritative source for corrected memory error data during any unexplained ESXi hang investigation.

• If the value is non-zero (for example, 1 CMC), the OS receives corrected error notifications, and /var/run/log/vmkernel.log normally contains MCE or CMCI entries. This article does not apply to platforms with non-zero values.