Memory usage alarm triggers for certain types of Virtual Machines in ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

• 'Virtual machine memory usage' alarm is displayed for the virtual machine.
• The Virtual Machine's Memory Usage/Active performance metric is continually reported as 100%.
• On vSphere 6.0, the Virtual Machine is configured with Latency Sensitivity set to High.
• On vSphere 6.5 and later the Virtual Machine is configured with at least one of these options:

• PCI passthrough devices
• Fault Tolerance (FT) enabled
• Latency Sensitivity set to High
• vGPU enabled
• Manually configured "sched.mem.prealloc.pinnedMainMem = True"

• On vSphere 6.7 and later the Virtual Machine is configured with either of the above or:
• 1GB Pages

 

Note: If Latency Sensitivity was set to High while the Virtual Machine was powered on, the pre-allocation will only apply at the next power cycle.

 

Environment

VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.0
VMware vSphere ESXi 7.0.1

Cause

The behavior of showing 100% "Memory Active", depending on the ESXi version, is by design for Virtual Machines in the specified configurations.

 

ESXi's active memory metric, despite being called "Memory Utilization" or "Memory Usage" in different parts of the UI, is in no way related to the in-guest memory metrics. It doesn't show how much guest OS memory is available nor how much guest memory is in an "active" working set or "resident". It is only used for making memory reclamation decisions in addition to other resource controls like shares, limits and reservation

"Memory Active" is a heuristic utilizing a weighted moving average based on reads and writes to a small, moving subset of pages of the Virtual Machine's memory. The sampling of this subset of 100 random pages over one minute also incurs a minimal overhead and preallocating memory is designed to remove most possibilities of jitter. The Virtual Machine configurations mentioned in the Symptoms section force a full memory reservation and are not subjected to reclamation techniques, hence memory sampling is disabled and active memory will default to a display value of 100%.

Resolution

This is a display artifact only and has no negative performance impact on the virtual machine or the host it is running on. To limit false positives, the memory activity for VMs with pre-allocated memory has been reduced to 75% in ESXi 7.0 U2

 

Workaround:

Disable the Virtual machine memory usage alarm to avoid false positives.

 

To disable the alarm definition using the vSphere Web Client:

1. Select the vCenter object in the navigation pane of the vSphere Web Client.
2. Click the Monitor tab > Issues sub-tab.
3. Select Alarm Definitions and search for Virtual machine memory usage.
4. Highlight the alarm and click Edit.
5. Deselect Enable this alarm.
6. Click Finish.

To disable the alarm definition using the HTML5 vSphere Client:

1. Select the vCenter object in the navigation pane of the vSphere Client
2. Click Configure > Alarm Definitions
3. In the Alarm Name column header, click the filter icon and search for Virtual machine memory usage
4. Highlight the alarm
5. Click Disable

Note: These steps will disable the alarm for all virtual machines. If you require the memory usage alarm to be functional for other than the VMs affected by the behavior described in this article, you will need to create a custom alarm with the same settings as the default one, on an object these VMs are nested under (such as a VM folder for example).

Additional Information

VM memory usage heuristic over-reporting on ESXi 6.5