VCF cluster Upgrade Precheck fails at ESXI host Maintenance Mode dry run due to insufficient resources to satisfy HA Admission control policy

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• VMware Cloud Foundation (VCF) cluster upgrade from SDDC Manager fails during the precheck phase due to an ESXi host failing to enter maintenance mode with below error on the UI:
  
  
  
  
• Upon reviewing the related task from the SDDC Manager UI, the below error message is encountered:
  
  Message: There are one or more exceptions that would prevent the host from entering maintenance mode. Check for Anti-Affinity rule violations, Insufficient Memory/Resources faults, DRS vMotion incompatible faults, etc. If Domain has Horizon VDI deployment then EMM might be failing due to Horizon instant clones. Please follow https://kb.vmware.com/s/article/79720 to remove Horizon instant clones.
Remediation Message: Check for errors in the lcm log files located on SDDC Manager under /var/log/vmware/vcf/lcm. Please retry the upgrade once the upgrade is available again.
Reference Token: <TOKEN>
  
  
• On the /var/log/vmware/vcf/lcm/lcm.log the following entries are observed:
  
  YYYY-MM-DDThh:mm:s ERROR [vct_Icm, 6####ffb4ae, 4##0] [c.v. v.1.b. 1.TranslationMessage, http-n1o-127.0.0.1-400-exec-4] Can't find resource for bundle java.util.PropertyResourceBundle, key com.vmware.vcf.error.environment.esx.mm.enter.check.failed.remedy
YYYY-MM-DDThh:mm:ss ERROR [vcf_lcm, 6####ffb4ae, 4##0] [c.v.evo.sddc.lcm.model.task.SubTask, http-nio-127.0.0.1-7400-exec-4] Upgrade error occured: There are one or more exceptions that would prevent the host from entering maintenance mode. Check for Anti-Affinity rule violations, Insufficient Memory/Resources faults, DRS vMotion incompatible faults, etc. If Domain has Horizon VDI deployment then EMM might be failing due to Horizon instant clones. Please follow https://kb.vmware.com/s/article/79720 to remove Horizon instant clones. Reference token <TOKEN>
  
  
• Additionally, attempting to manually place the affected ESXi host into Maintenance Mode from the vSphere Client stalls while waiting for virtual machines to evacuate, eventually timing out even though virtual machine reservations do not exist and there are no Anti-Affinity rule violations.

Cause

As part of the the cluster Host upgrade precheck, when the SDDC manager attempts to perform a dry run to check if the ESXi host is able to move to maintenance mode, vSphere HA Admission Control prevents the ESXi host from entering maintenance mode because active virtual machine workloads exceed the remaining cluster resources' capacity required by the configured failover policy. For example, in a 4-host cluster, evacuating one host fails if the remaining 3 hosts cannot support the active workloads while satisfying the configured failover tolerance.

Resolution

To resolve this issue and allow the VCF cluster upgrade precheck to proceed, make sure that the failover capacity is being met by either increasing the Cluster Resource/Capacity or reducing the active/powered-on workload:

1. Ensure there are sufficient resources so that the Admission Control and HA failover capacity is met before the upgrade begins.

2. Add additional ESXi host resources to the cluster or power off non-essential virtual machines to free up failover capacity.