Diagnostics for VMware Cloud Foundation: ESX Health: Memory Health

Products

VCF Operations

Issue/Introduction

Memory Health is monitored through two primary lenses: the physical integrity of the hardware (Hardware State) and the availability of memory resources (Reservation State).

1. Hardware State

The hardware status is determined by the Exponentially Weighted Moving Average (EWMA) of corrected errors and the cumulative count of uncorrected errors.

A Machine Check Exception (MCE) is a hardware error detected by the CPU. These typically signal significant issues such as ECC memory errors, processor cache failures, system bus faults, or thermal issues.

Status	Condition Logic	Default Thresholds
Green	MCE counts are within healthy operational limits.	N/A
Yellow	EWMA of corrected errors exceeds MCERateYellow OR total uncorrected errors exceed MCEUncorrYellow.	20 - MCERateYellow and 1 - MCEUncorrYellow
Red	EWMA of corrected errors exceeds MCERateRed OR total uncorrected errors exceed MCEUncorrRed.	1200 - MCERateRed and 10 -MCEUncorrRed

Reservation State

This state monitors the host's ability to fulfil memory guarantees for virtual machines and system processes.

Green: Sufficient memory is available to satisfy all current and pending reservations.
Yellow: Available reservable memory is approaching the critical threshold; the host may soon struggle to meet new reservation requests.
Red: The host is at risk of exhausting all reservable memory. This is a critical state that can lead to admission control failures or system instability.

Description:

The Memory Health service aggregates various counters to provide a holistic view of the subsystem. Anomalies in any of the following may trigger a Warning or Critical status:

Resource Allocation: Monitoring of memory shares, reservations, and limits.
vSAN Health: Tracking memory pools (slabs) specifically allocated for vSAN operations.
Memory Heaps: Monitoring kernel memory allocation to prevent heap exhaustion.
Congestion: Identifying bottlenecks in memory access or swap subsystem pressure.
General Utilization: Analyzing the delta between Consumed memory (physical RAM mapped) and Active memory (RAM recently touched by the CPU).

Environment

VCF Operations 9.x

Resolution

1. Addressing Hardware State (MCEs)

If the Hardware State is triggered, the physical integrity of the DIMMs or CPU may be compromised.

Identify the Faulty Component: Check the Hardware Status tab in the vSphere Client or examine the IPMI/System Event Log (SEL) via the out-of-band management controller (iDRAC, ILO, or IMM).
Review VMkernel Logs: Use the following command via SSH to search for Machine Check events:

cat /var/log/vmkernel.log | grep -i "MCE"

Remediation: *

Correctable Errors: Monitor the rate. If it continues to climb toward the Red threshold (1200), schedule a maintenance window to reseat or replace the affected DIMM.
Uncorrectable Errors: These often cause an immediate Purple Screen of Death (PSOD). If the host is still up, evacuate VMs immediately and replace the hardware.

2. Addressing Reservation State

A Reservation State warning indicates that the host's "admission control" is reaching its limit.

Check Over-Reservation: Identify VMs with high memory reservations.

Path: Host > VM tab > Monitor > Resource Allocation > Memory.

Calculate Available Capacity: Ensure that the sum of all VM reservations + ESXi overhead does not exceed the physical RAM minus the "Userworld" reserved capacity.
Remediation: Reduce or remove reservations on non-critical VMs.

vMotion VMs to hosts with higher available capacity.
Check for Memory Ballooning or Swapping, which are indicators that the host is actively struggling to reclaim memory.