ESXi Host becomes unresponsive or crashes due to Memory Controller Errors

Products

• VMware vSphere ESXi

Issue/Introduction

In a scenario where hardware encounters a large number of corrected memory errors, the ESXi host may exhibit the following symptoms:

• The host becomes unresponsive for an extended period or reboots.
• The system presents a Purple Screen of Death (PSOD) with a backtrace including Panic_WithBacktrace and no heartbeat.
• Storage latency increases, leading to intermittent access issues or Guest OS freezes.
• System Management Interrupts (SMIs) trigger at a high rate, flooding the CPU.

Memory Error Classifications

• Frequent Correctable Errors: A high volume of correctable errors occurs rapidly, flooding the CPU with Machine Check Exceptions (MCE), Corrected Machine Check Interrupts (CMCI), and System Management Interrupts (SMI). This overwhelms the processor and traps the BIOS in System Management Mode (SMM), causing a CPU lockup and a subsequent PSOD.
• Infrequent Correctable Errors: Intermittent, isolated correctable memory errors occur at a low frequency. The system actively logs and handles these errors in the background without experiencing CPU lockup or impacting overall host stability.
• Uncorrectable Errors: A single, fatal memory error that cannot be resolved by the hardware's error-correcting code. This will immediately trigger a PSOD to protect data integrity.

In scenarios where a PSOD is not triggered, the host may experience temporary performance degradation but can ultimately resume normal operation without manual intervention. Self-recovery typically occurs under the following conditions:

• Successful Error Correction: The memory controller successfully completes the error-correction phase for Correctable Errors (CE) without encountering a secondary, uncorrectable fault.
• Transient CPU Consumption: The elevated CPU cycle consumption required to process and clear the interrupts subsides before reaching the threshold of a definitive kernel panic or permanent CPU lockup.

Once the hardware handles the overhead, the host automatically stabilizes and resumes standard operation.

Sample log messages associated with this issue:

  1. Memory Controller Errors found in /var/run/log/vmkernel.log, such as:
      YYYY-MM-DDTHH:MM:SS cpu##:##)MCA: ##: CE Poll G0 B8 S9c0000####0091 A6619##### M200401c0898##### P6619f37#####/40 Memory Controller Read Error on Channel 1.

  2. hostd Service Hang found in /var/run/log/vmkernel.log, such as:
      YYYY-MM-DDTHH:MM:SS cpu#:####)ALERT: hostd detected to be non-responsive

  3. Storage I/O issues found in /var/run/log/vmkernel.log, such as:
    YYYY-MM-DDTHH:MM:SS cpu#:####)WARNING: ScsiDeviceIO: 1513: Device naa.############### performance has deteriorated. I/O latency increased from average value of 1234 microseconds to 123456 microseconds.
    YYYY-MM-DDTHH:MM:SS cpu#:####)ScsiDeviceIO: 4176: Cmd(0x45b96d402508) 0x89, CmdSN 0x1578666 from world 3361402 to dev "naa.#################" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0

  4. Storage I/O Issues found in /var/run/log/hostd.log, such as:
      YYYY-MM-DDTHH:MM:SS warning hostd[####] [Originator@6876 sub=IoTracker] In thread 2099376, fopen("/vmfs/volumes/#####-#####-####-###########/vm-name/vm-name.vmx") took over 1236 sec.

  5. EFI + VMB messages soon after the controller read error:

   YYYY-MM-DDTHH:MM:SS ************** vmkernel: cpu53:***** )MCA: 202: CE Intr G0 B19 S************ Ad********** M************ P**********/40 Memory Controller Read Error on Channel 0.
   YYYY-MM-DDTHH:MM:SS ************** vmkernel: VMB: 65: Reserved * MPNs starting @ ****
   YYYY-MM-DDTHH:MM:SS ************** vmkernel: EFI: 250: 64-bit EFI v2.80 revision ******
   YYYY-MM-DDTHH:MM:SS ************** vmkernel: VMB_SERIAL: 170: Serial port configuration obtained from firmware.
   YYYY-MM-DDTHH:MM:SS ************** vmkernel: VMB: 79: TDX: Unsupported on CPU (MSR_MTRRCAP = ******)

These log entries demonstrate the sequence of events: memory controller errors, followed by storage I/O issues, leading to the hostd service becoming unresponsive, and VMFS volume access problems.

Environment

• VMware vSphere ESXi 7.0.x
• VMware vSphere ESXi 8.0.x
• HPE ProLiant Gen10 servers (and other vendors supporting ECC memory)

Cause

Frequent correctable memory errors flood the CPU with interrupts (CMCI/MCE) and System Management Interrupts (SMIs). This flood overwhelms the processor and forces the BIOS into System Management Mode (SMM) for error correction. If the interrupt rate is high enough, it prevents the vmkernel from updating its heartbeat, leading to a CPU lockup and subsequent PSOD.

Resolution

1. Identify the faulty DIMM by reviewing Out-of-Band Management logs (e.g., HPE iLO, Dell iDRAC, or Cisco UCS Manager) to pinpoint the memory slot reporting ECC errors.
2. Replace the defective DIMM(s) to stabilize the ESXi host environment.
3. Contact the hardware vendor to run comprehensive system diagnostics and initiate a replacement for the failing hardware.
4. Update system firmware, including BIOS and motherboard firmware, to the vendor's latest recommended versions to ensure optimal hardware error handling.