vCenter upgrade failed on the SDDC manager at the pre-validation precheck stage
search cancel

vCenter upgrade failed on the SDDC manager at the pre-validation precheck stage

book

Article ID: 390255

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

  • It would be observed that the vCenter upgrade will fail on the SDDC manager at the first level pre-validation check with the following error on the UI. 

Vcenter Upgrade and interop compatibility checks failed.","metadata":"VCENTER upgrade bundle is not compatible with your existing version 8.0.2.00400-23929136. Refer to the https://kb.vmware.com/s/article/91728 to address incompatibilities and proceed with the upgrade.
Please retry the upgrade once the upgrade is available again.\nContact GSS for assistance. Reference Token:

 

Environment

VCF 5.x 

Cause

  • As part of the precheck the, the SDDC manager queries VC and checks the cluster and fetches the version of all the hosts. 
  • It was found that the one of the host was on Not-Responding state. 
  • This was reported on /var/log/vmware/vc/lcm/lcm-debug.log:

YYYY-MM-DDTHH:MM:SS.234+0000 INFO  [vcf_lcm,0000000000000000,0000,upgradeId=d124f037-997f-4470-xxxx-693df4e2025f,resourceType=VCENTER,resourceId=e13c1725-0dea-4f2e-bae3-2234337424c8,bundleElementId=3eab1127-xxxx-yyyy-a2fa-df3163f8be1a] [c.v.e.s.c.c.v.vsphere.VsphereClient,Upgrade-2] Successfully logged in to https://vCenter_FQDN.net:443/sdk
YYYY-MM-DDTHH:MM:SS.595+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=d124f037-997f-4470-xxxx-693df4e2025f,resourceType=VCENTER,resourceId=e13c1725-0dea-4f2e-bae3-2234337424c8,bundleElementId=3eab1127-xxxx-yyyy-a2fa-df3163f8be1a] [c.v.e.s.l.p.impl.esx.EsxUtils,Upgrade-2] Exception while fetching the current version for hosts belonging to vcenter with cluster MOID: domain-cx
YYYY-MM-DDTHH:MM:SS.599+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=d124f037-997f-4470-xxxx-693df4e2025f,resourceType=VCENTER,resourceId=e13c1725-0dea-4f2e-bae3-2234337424c8,bundleElementId=3eab1127-xxxx-yyyy-a2fa-df3163f8be1a] [c.v.e.s.l.p.i.v.VcenterUpgradeCompatibilityPrimitiveHelper,Upgrade-2] Failed to calculate vcenter upgrade compatibility com.vmware.evo.sddc.lcm.model.error.LcmException: null
        at com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxUtils.getEsxVersion(EsxUtils.java:244)
        at com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxUtils.getEsxVersionsOfCluster(EsxUtils.java:211)
        at com.vmware.evo.sddc.lcm.primitive.util.CompatibilityVersionHelper.getEsxVersionsForVcenterAndCluster(CompatibilityVersionHelper.java:108)
        at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VcenterUpgradeCompatibilityHelper.getDistinctInteropCombinationForPostUpgradeInteropVerification(VcenterUpgradeCompatibilityHelper.java:163)
        at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VcenterUpgradeCompatibilityHelper.getInterOpCompatibility(VcenterUpgradeCompatibilityHelper.java:96)
        at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VcenterUpgradeCompatibilityHelper.isVcenterUpgradeVvsCompatible(VcenterUpgradeCompatibilityHelper.java:51)
        at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VcenterUpgradeCompatibilityPrimitiveHelper.verifyUpgradeVvsCompatibility(VcenterUpgradeCompatibilityPrimitiveHelper.java:58)
        at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterUpdater.update(VCenterUpdater.java:183)
        at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterPatchUpgradeImpl.postUpgradeHelper(VCenterPatchUpgradeImpl.java:240)
        at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterPatchUpgradeImpl.postUpgradeWithState(VCenterPatchUpgradeImpl.java:149)
        at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterPatchUpgradeImpl.postUpgrade(VCenterPatchUpgradeImpl.java:141)
        at com.vmware.evo.sddc.lcm.orch.PrimitiveServiceImpl.postUpgradeAsync(PrimitiveServiceImpl.java:289)
        at com.vmware.evo.sddc.lcm.orch.PrimitiveServiceImpl.lambda$postUpgrade$0(PrimitiveServiceImpl.java:165)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.NullPointerException: Cannot invoke "com.vmware.vim.binding.vim.host.ConfigInfo.getProduct()" because the return value of "com.vmware.vim.binding.vim.HostSystem.getConfig()" is null
        at com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxUtils.lambda$getEsxVersion$0(EsxUtils.java:238)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
        at com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxUtils.getEsxVersion(EsxUtils.java:233)
        ... 15 common frames omitted

Resolution

To resolve this issue: 

  • Identify the cluster that is pointing to the domain value as per line: Exception while fetching the current version for hosts belonging to vcenter with cluster MOID: domain-cx
    • The cluster represented by the domain, can be identified on the VC, by selecting a cluster and checking it on the browser url.  
  • Once the cluster is identified, look for any unhealthy hosts that are either in not responding state or a state where the hosts resources information is not fetched. 
  • Once identified, resolve the issue with the host connection with the VC. 
  • If the host issue cannot be fixed at the moment: 
    • Then perform a disconnect activity on the host and remove it from the cluster and place it under the Datacenter object.
    • In this way the validation will skip the host .

Note: Do not remove the host out of inventory, if you are intending to reuse the host after fix/repair. If the host is removed, then it would need to be cleaned up from the SDDC manager inventory and rebuild for re-commissioned.  

After the above steps, retry the activity back on the SDDC manager and now there should be progress to install stage for the upgrade. 

Powered by Wolken Software