Frequent "Virtual machine memory usage" warning message on virtual machines.

Products

• VMware vSphere ESXi

Issue/Introduction

Virtual machines residing on a cluster frequently trigger the Virtual machine memory usage warning alert. The alert status fluctuates between Green and Yellow (and back to Green) over short intervals. 


Sample of default Alarm Definition:

Sample of agressive Alarm Definition:

Environment

VMware vSphere ESXi

VMware vCenter Server

Cause

This behavior typically occurs due to one of the following:

• Aggressive Alarm Thresholds: The vCenter alarm definition for "Virtual machine memory usage" may be set to a very low threshold (e.g., "VM Memory Usage is above 10% for 30 Seconds"). This causes frequent triggers based on minor spikes in memory activity and is considered expected behavior by design.
• High Page-In Rate: The Guest Operating System may be experiencing a high pageInRate, indicating active memory management within the OS itself
• In a Windows VM, the alert is triggered when high cache memory is held within the guest OS. To validate this, navigate to the Task manager > Memory of the VM within the guest OS.

Resolution

To validate and resolve the frequent alarms, perform the following checks:

1. Validate Alarm Definitions

• Navigate to vCenter > Configure > Alarm Definitions.
• Search for and expand Virtual machine memory usage.
• Identify the Alarm Rules and specifically the Warning condition threshold and duration.

2. Host-Level Performance Analysis

Connect to the ESXi host where the affected VM is residing via SSH and execute the following commands to gather physical memory metrics:

Command 1: Evaluate VM Resource Allocation Review this command before running it.

memstats -r vm-stats -s name:memSize:allocTgt:mapped:consumed:touched:min:max:memNeeded:touched:memLimit -u mb

• Calculation: Calculate the percentage of active usage using the formula: (touched / memSize * 100).
• Validation: Compare this result against the trigger threshold identified in Step 1.

Command 2: Evaluate Guest Memory Activity Review this command before running it

memstats -r guest-stats -u mb -s name:memTotal:lpageTotal:memNeeded:memNeededResv:memFree:pageSize:pageInRate

• Analysis: If the usage calculated in Command 1 exceeds 85%, validate the pageInRate value in this output.
• High pageInRate: A high value indicates the Operating System is actively utilizing memory. In this case, consult with the OS vendor for internal memory optimization.

3. Recommended Mitigation

If the metrics show high activity but no performance impact, or if the current alarm definition is set too low for the environment's baseline:

• Modify the Alarm Definition to the default value of 85% for 10 minutes to reduce false-positive notifications.