Virtual Machines using large pages can temporarily become unresponsive after vMotion

Products

• VMware vSphere ESXi

Issue/Introduction

• A virtual machine (VM) configured with large virtual hardware resources and under memory-intensive load becomes unresponsive after the vMotion has completed.
• Analysis of the vMotion process through the logs does not show high execution switch-over times, and the vMotion appears to complete normally.
  
  Note: After the brief sluggishness period, the guest OS will continue to function normally.

Environment

• VMware vSphere ESXi 6.0
• VMware vSphere ESXi 5.5

Cause

vMotion transfers pages at a 4 KB (small) granularity. Before the virtual machine resumes on the destination, it attempts to remap those to 2 MB (large) pages. This remapping is attempted in a near sequential manner for all of the virtual machine's allocated memory. Depending on the memory size and in guest memory activity, this may lead to noticeable workload interruptions.

Resolution

This is a known issue affecting VMware ESXi 5.5 and 6.0.

 

Currently, there is no resolution.

To work around this issue, temporarily disable the large page promotion path on destination host during vMotion, and enable the same after vMotion.

1. Run this command to disable large page promotion path on destination host prior to vMotion.
   
   vsish -e set /config/Mem/intOpts/AllocGuestLargePage 0
   
   
2. Run this command to enable large page promotion path on destination host after vMotion.
   
   vsish -e set /config/Mem/intOpts/AllocGuestLargePage 1

Note: After the vMotion has completed, and the large pages have been re-enabled on the destination Host, the small memory pages will be remapped to large pages as the guest touches the memory. How fast this conversion takes place is dependent on the guest workload. But every time the guest touches the page, vmkernel/monitor will opportunistically convert these small pages to large page mappings. This is all transparent to the guest.