Understanding NVMe Memory Tiering Activation Thresholds and Monitoring in vSphere 9.x

Products

• VMware vSphere ESXi

Issue/Introduction

During the qualification of NVMe Memory Tiering, you may observe that Tier 1 (NVMe) resources are not being consumed even when Tier 0 (DRAM) resource consumption exceeds 70%.

Environment

VMware vSphere ESXi 8.0 Update 3 and later

Cause

This behavior is by design. The ESXi memory management tiering algorithm is selective and governed by specific operational triggers:

1. Threshold Activation: The VMkernel initiates memory tiering only when total host physical DRAM (Tier 0) consumption reaches approximately the 80% threshold. If consumption is below this trigger (e.g., at 70-75%), the system will not migrate pages to Tier 1.
2. Page Classification: ESXi monitors page activity based on recency and frequency. Only memory pages strictly classified as "cold" (inactive) are migrated to the NVMe Tier 1 device. The active working set ("hot" pages) is deliberately retained in Tier 0 to prevent performance penalties.
3. Scan Rate: The page scan and migration rate scale with memory pressure. At lower consumption levels (just crossing the 80% mark), the rate is highly conservative.

Resolution

If Tier 1 resources are not being consumed, verify the current host memory footprint to see if it has crossed the 80% trigger point. Use the following diagnostic methods to validate behavior:

CLI Verification

Log in to the ESXi host via SSH and run the following command to isolate Tier 1 consumption on a per-VM basis:

memstats -r vmtier-stats -u mb -s name:memSize:active:tier1Target:tier1Consumed

vSphere Client Monitoring

1. Navigate to the affected host or VM.
2. Select the Monitor tab, then click Performance > Advanced.
3. Click Chart Options and select the Memory view.
4. Select the following counters to chart NVMe tier performance:
   • Memory type read bandwidth
   • Memory type read latency
   • Memory type write bandwidth
   • Memory type write latency