In SDDC, Upgrading the ESXI fails at prechecks.

Products

• VMware SDDC Manager

Issue/Introduction

• In SDDC Manager, the operation fails, and an error message is displayed in the UI indicating a failure during the ESX host prechecks.

• Alarms & Warnings :

 

lcm.log :

YYYY-MM-DDTHH.947+0000 ERROR [vcf_1cm, 0000000000000000, 0000, upgradeId=fc51e4a8-813b-4b13-94f1-d51b347c8834, resourceType=ESX_CLUSTER, resourceId=b2f9c021-2ec0-4d6e-83c0
-14baaec91be3,bundleElementId=4e5d3fc2-a467-4a9f-a5a7-de4de361c585] [c.v.e.s.l.p.i.c. VlcmUpdateStageRunnerImpl, Upgrade
java. lang. RuntimeException: Retriable operation 'Checking if VLCM cluster config status is green' failed to complete after 3 retries.
at com. vmware.evo.sddc.common. util.RetriableCallable.call (RetriableCallable.java: 222)
at com. vmware.evo.sddc.lcm.primitive. impl. cluster. VlcmUtils. checkClusterHealthWithRetries (Vlcmutils. java: 363)
at com. vmware.evo.sddc.lcm.primitive.impl.cluster. VlcmUpdateStageRunnerImpl.doPrecheck (VlcmUpdateStageRunnerImpl.java: 643)
at com. vmware.evo.sddc.lcm.primitive.impl.cluster. VlcmUpdateStageRunner.dostage (VlcmUpdateStageRunner. java: 161)
at com. vmware. evo.sddc.lcm.primitive.impl. cluster. VlcmPrimitiveImpl. runUpgrade (VlcmPrimitiveImpl. java:516)
at com. vmware.evo.sddc.lcm.primitive. impl. cluster. VlcmPrimitiveImpl. startOrResumeUpgrade (VlcmPrimitiveImpl. java: 262)
at com. vmware. evo.sddc. lcm.primitive. impl. cluster. VlcmPrimitiveImpl. postUpgrade (VlcmPrimitiveImpl. java: 205)
at com.vmware.evo.sddc.lcm.orch. PrimitiveServiceImpl.postUpgradeAsync (PrimitiveServiceImpl.java: 323)
at com. vmware.evo.sddc.lcm.orch. PrimitiveServiceImpl.lambda$postUpgrade$0 (PrimitiveServiceImpl. java: 165)
at java.base/java.util.concurrent.ThreadPoolExecutor. runWorker (ThreadPoolExecutor. java: 1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java: 635)
Caused by: com. vmware.evo.sddc.lcm.model.error.LcmException: Cluster is unhealthy. Reason: vSphere HA failover operation in progress in cluster [Cluster name] in datacenter
Workload-01-Domain-DC: 0 VMs being restarted, 1 VMs waiting for a retry, 0 VMs waiting for resources, 0 inaccessible vSAN VMs
at com. vmware.evo.sddc.lcm.primitive. impl. cluster. VlcmUtils. checkClusterHealth (VlcmUtils. java: 413)
at com. vmware. evo.sddc. lcm.primitive.impl.cluster. VlcmUtils. lambda$checkClusterHealthWithRetries$2(Vlcmutils.java: 358)
at com. vmware.evo.sddc.common.util.RetriableCallable.call(RetriableCallable.java:182)
... 11 common frames omitted
YYYY-MM-DDTHH.949+0000 ERROR [vcf_1cm, [REDACTED] 0000, upgradeId=fc51e4a8-813b-4b13-94f1-d51b347c8834, resourceType=ESX_CLUSTER, resourceId=b2f9c021-2ec0-4d6e-83c0
com. vmware. evo.sddc. lcm.model. error.LcmException: Prechecks failed
at com. vmware.evo.sddc.lcm.primitive. impl.cluster. VlcmUpdateStageRunnerImpl.doPrecheck
YYYY-MM-DDTHH.949+0000 DEBUG [vcf_lcm,[REDACTED] 0000, upgradeId=fc51e4a8-813b-4b13-94f1-d51b347c8834, resourceType=ESX_CLUSTER, resourceId=b2f9c021-2ec0-4d6e-83c0
-14baaec91be3,bundleElementId=4e5d3fc2-a467-4a9f-a5a7-de4de361c585] [c.v.e.s.l.p.i.c.VlcmPrimitiveImpl, Upgrade-1] overallUpgradeStatus = COMPLETED_WITH_FAILURE.

 

Environment

 

• VMware cloud foundation 4.X 
• VMware cloud foundation 5.X

Cause

 There was older alarms and warnings at the cluster level for HA and cluster health which was resolved but not reset it to green & due to alarms upgrade failed at prechecks to complete in SDDC UI.

Example :

vSphere HA falover in progress

vSphere HA falover operation in progress in cluster [Cluster name] in datacenter Worklcad-01-Domain-DC: 0 VMs being restarted, 1 VMs waiting for a retry, O VMs waiting for resources, O inaccessible vSAN VMS

 

Resolution

• Ensure cluster is healthy.
• Try to fix the alerts, warnings and alarms at cluster level if it is already fixed then reset it to Green.
• Try to disabled/enable the HA to fix the HA alarms and warnings.
• Once the alarms/warnings cleared at cluster level & cluster become healthy proceed with prechecks.