Load balancer VM failover due to hypervisor scheduling delays on VMware ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

• A load balancer virtual machine (VM) unexpectedly fails over to its High Availability (HA) peer. While the ESXi host appears healthy, underlying resource contention causes heartbeats to timeout.
  
  
• Load balancer cluster triggers a failover to the standby node.
  
  
• vSphere performance charts show high CPU Ready Time or Co-Stop on the host/cluster.

Environment

• VMware ESXi 8.0U3
  
  
• Load Balancer Virtual Appliances

Cause

Transient CPU scheduling starvation at the hypervisor level. When the ESXi scheduler cannot provide CPU cycles to the VM within the application's heartbeat window, the guest perceives a hang and triggers a failover.

Resolution

Engage the internal application team or vendor to extract and analyze the internal load-balancer cluster logs (such as heartbeat metrics and configuration status) to determine the exact criteria that triggered the software-level failover.

To prevent host resource contention from impacting virtual machine heartbeat processing and causing software-level failovers, apply the following workaround:

1. Relocate VM to Lower-Utilized Host:

   Create a vSphere DRS VM-Host affinity rule to ensure the load-balancer virtual machine resides on an ESXi host with lower historical resource utilization.

2. Set 100% Guest Memory Reservation:

   a. Navigate to the virtual machine in the vSphere Client inventory.

   b. Right-click the virtual machine and select Edit Settings.

   c. Expand Memory.

   d. Select the check box for Reserve all guest memory.

   e. Click OK to save settings.

3. Set 100% CPU Reservation:

   Since CPU does not have a "Reserve all" toggle, the value must be entered manually based on the host's physical clock speed.

   1. Right-click the VM and select Edit Settings.
      
      
   2. Expand the CPU section.
      
      
   3. Locate the Reservation field (MHz).
      
      
   4. Enter the required value (e.g., 19,200 MHz).
      
      
   • Calculation: (Number of vCPUs) x (Clock Speed of Host Physical Core).
     
     
   • Example: 8 vCPUs x 2400 MHz (2.4 GHz) = 19,200 MHz.
     
     
   5. Click OK.

   Note: While these changes can be applied while the VM is powered on, some applications may require a guest OS reboot to acknowledge the resource change and clear the alarm immediately.