Guest OS CPU Spikes and Pod Restarts During vMotion of Large Memory VMs

Products

• VMware vSphere ESXi
• VMware Cloud Foundation

Issue/Introduction

Large virtual machines (VMs) configured with high vCPU counts (e.g., 48 vCPU) and large memory footprints (e.g., 384GB RAM) may experience significant compute utilization spikes during vMotion.

Hypervisor observations: 

• Multiple concurrent large vMotions triggered simultaneously due to placing the host in Maintenance Mode (MM) or executing bulk manual migrations.
• Disabling SDPS causes longer switchover times and further performance degradation.
• esxtop performance data captured during the vMotion indicates the vMotion network is fully saturated, consuming all available bandwidth.
• The /var/run/log/vmkernel.log on the source ESXi host, filtered by vMotion ID, reports SDPS activation before switchover due to insufficient bandwidth: 

YYYY:MM:DDTHH:MM:SS.Z cpu8:8168209)Migrate: 102: 329181981589220830 S: MigrateState: Precopying memory
YYYY:MM:DDTHH:MM:SS.Z cpu107:8168212)VMotion: 4851: 329181981589220830 S: Another pre-copy iteration needed with 2948231 pages left to send (prev2 100663296, prev 100663296, passthru device dirty pages 0, network bandwidth ~583.544 MB/s, 28719% t2d)
YYYY:MM:DDTHH:MM:SS.Z cpu96:8168212)VMotion: 4851: 329181981589220830 S: Another pre-copy iteration needed with 560993 pages left to send (prev2 100663296, prev 2948231, passthru device dirty pages 0, network bandwidth ~380.437 MB/s, 456% t2d)
YYYY:MM:DDTHH:MM:SS.Z cpu99:8168212)VMotion: 4851: 329181981589220830 S: Another pre-copy iteration needed with 247296 pages left to send (prev2 2948231, prev 560993, passthru device dirty pages 0, network bandwidth ~171.261 MB/s, 183% t2d)
YYYY:MM:DDTHH:MM:SS.Z cpu97:8168212)VMotion: 4779: 329181981589220830 S: Not enough forward progress, enabling SDPS (Pages left to send: prev2 560993, prev 247296, cur 169711, network bandwidth ~136.222 MB/s, 128% t2d)
YYYY:MM:DDTHH:MM:SS.Z cpu111:8168212)VMotion: 4746: 329181981589220830 S: Stopping pre-copy: only 26208 pages left to send, which can be sent within the switchover time goal of 0.500 seconds (network bandwidth ~214.944 MB/s, 499% t2d)
YYYY:MM:DDTHH:MM:SS.Z cpu111:8169847)Migrate: 102: 329181981589220830 S: MigrateState: Quiesing VM

Guest Symptoms include:

• Increased Guest OS system wait time and overall resource utilization.
• Application failovers and service instability.
• Brief ping drops or increased network latency.
• In Kubernetes or containerized environments, liveness probe failures leading to Pod restarts.

 

Environment

vSphere  7.x

vSphere 8.x 

VCF 9.x 

 

Cause

Cause: The issue was identified as a performance impact caused by the Stun During Page Send (SDPS) mechanism, rather than a product defect.

When migrating large, resource-intensive VMs , if the VM's memory "dirty rate" (the speed at which data is changed) exceeds the available vMotion network bandwidth, the migration cannot "converge" (finish copying memory). To force completion, vMotion triggers SDPS to throttle the guest VM's memory write rate. This throttling manifests as a significant CPU spike within the guest OS, which in this case caused Kubernetes liveness probes to fail, leading to Pod/Applications restarts.

Resolution

To mitigate the impact of SDPS and prevent application or Pod restarts during migration, implement one or a combination of the following environmental optimizations:

1. Reduce vMotion Concurrency Lower the default vCenter Server limit for simultaneous migrations per host. Reducing this value ensures more dedicated bandwidth is available for individual large VMs during Maintenance Mode evacuations.

   • Action: Navigate to vCenter Server Advanced Settings and modify config.vpxd.ResourceManager.costPerVmotionESX6x.

   • Recommendation: Change the value to 4 (reducing concurrent migrations from 8 to 4) or 2 for extremely latency-sensitive environments.

2. Increase vMotion Network Bandwidth Ensure the physical network architecture can accommodate the memory dirty rate of large VMs.

   • Action: Dedicate multiple physical NICs specifically for vMotion traffic by configuring a Multi-NIC vMotion topology. This aggregates bandwidth across multiple interfaces for faster memory page transfers.

3. Implement vMotion Application Notifications For latency-sensitive applications (e.g., Envoy, clustered databases, Java-based financial services), utilize the vMotion Application Notification feature introduced in vSphere 8.0.

   • Action: Enable Application Notification for the VM. This allows the Guest OS to receive a signal before a migration starts, enabling the application to gracefully handle the temporary latency, pause clustered application health checks, or temporarily disable liveness probes.

4. Avoid Host Overcommitment Maintain lower CPU and memory contention on ESXi hosts running large, critical VMs.

   • Action: Balance cluster utilization to ensure adequate headroom. High host resource utilization exacerbates the time required for vMotion to quiesce and resume worlds.

Additional Information

• vMotion Application Notification
• Virtual machine performance degrades while a vMotion is being performed 
• Multiple-NIC vMotion in vSphere 
• Collecting esxtop batch data for ESXi performance troubleshooting 
  • Note: For this scenario, start the esxtop batch mode collection after initiating the vmotions to make sure the new vmotions worlds are captured on the esxtop.