vCenter upgrade from SDDC Manager fails with error "getDnsServerForVM failed due to exception for the VC"

Products

• VMware SDDC Manager
• VMware Cloud Foundation 5.x

Issue/Introduction

• vCenter upgrade from SDDC Manager failed at "VCENTER UPGRADE CREATE INPUT SPEC" stage
• /var/log/vmware/vcf/lcm/lcm-debug.log shows the below error

  yyy-mm-ddThh:mm:ss DEBUG [vcf_lcm,0000000000000000,0000,upgradeId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx,resourceType=VCENTER,resourceId=zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzz,bundleElementId=yyyyyyy-yyyy-yyyy-yyyyyyyyyy] [c.v.e.s.l.c.v.vsphere.VsphereUtils,Upgrade-45] Exception when trying to get dns server for vm vcenter.example.com: {}
  java.lang.NullPointerException: Cannot load from object array because the return value of "com.vmware.vim.binding.vim.vm.GuestInfo.getIpStack()" is null
   at com.vmware.evo.sddc.lcm.client.vmware.vsphere.VsphereUtils.lambda$getDnsServerForVM$1(VsphereUtils.java:1090)
   at com.vmware.evo.sddc.lcm.client.vmware.vsphere.VsphereUtils.resolve(VsphereUtils.java:1111)
   at com.vmware.evo.sddc.lcm.client.vmware.vsphere.VsphereUtils.getDnsServerForVM(VsphereUtils.java:1090)
   at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterCommon.createInputSpecV2130ForVC(VCenterCommon.java:2066)
   at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterUpgrader.createInputSpec(VCenterUpgrader.java:1165)
   at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterUpgrader.doUpdateStep(VCenterUpgrader.java:280)
   at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterUpgradeImpl.postUpgradeHelper(VCenterUpgradeImpl.java:225)
   at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterUpgradeImpl.postUpgradeWithState(VCenterUpgradeImpl.java:148)
   at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterUpgradeImpl.postUpgrade(VCenterUpgradeImpl.java:136)
   at com.vmware.evo.sddc.lcm.orch.PrimitiveServiceImpl.postUpgradeAsync(PrimitiveServiceImpl.java:306)
   at com.vmware.evo.sddc.lcm.orch.PrimitiveServiceImpl.lambda$postUpgrade$0(PrimitiveServiceImpl.java:176)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
   at java.base/java.lang.Thread.run(Thread.java:840)
  2024-08-05T03:44:23.352+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx,resourceType=VCENTER,resourceId=zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzz,bundleElementId=yyyyyyy-yyyy-yyyy-yyyyyyyyyy] [c.v.e.s.l.p.i.vcenter.VCenterCommon,Upgrade-45] In vc upgrade create input spec,  getDnsServerForVM failed due to exception for the VC vcenter.example.com
  java.lang.NullPointerException: Cannot load from object array because the return value of 

Environment

VMware Cloud Foundation 5.1

Cause

Every workload vCenter VM has a corresponding powered-off VM (from previous upgrades) with the same hostname. Even though the old vCenter VMs are powered off, the hostname attribute in guestinfo data object is not cleared.

Due to this, during the vCenter upgrade workflow, the 'vm inventory' search query is returning the MOID of the powered off VM instead of the correct powered on VM. This is causing the upgrade to fail.

Resolution

Issue will be fixed in VCF future release

Workaround:

1. For the failing workload vCenter, *unregister* the corresponding powered-off VM
   1. Login to the vSphere Client
   2. Locate the Powered off Workload vCenter VM (as identified in logs)
   3. Right click and unregister
2. Retry the upgrade from SDDC Manager