SDDC upgrade from 4.5.2.0 to 5.1.0.0 - vCenter upgrade post install failure (going from 7.0.3 to 8.0.2)

Products

• VMware SDDC Manager

Issue/Introduction

Issue can potentially be pre-empted using these KBs -

https://knowledge.broadcom.com/external/article/314655

https://knowledge.broadcom.com/external/article/325778

 

However, an issue can still occur whereby the source VC UUID doesn't get saved, triggering a post install failure.

Symptoms:

In /var/log/vmware/vcf/lcm/lcm.log, we see error messages like this:

• "upgradeError":{"errorType":"RECOVERABLE","stage":"VCENTER_UPGRADE_POST_INSTALL","errorCode":"SAVE_SOURCE_VC_UUID_FAILED","errorDescription":"VC upgrade failed while attempting to save source VC UUID.","errorDescriptionCodes":["GENERIC_REASON"],"errorDescriptionAttributes":{"0":"VC upgrade failed while attempting to save source VC UUID."},

 

• Select the resource pool null where the source VC vcenter-upgrade-target-appliance_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx is associated with and click \u0027ok\u0027.\n

 

• 2024-05-06T01:38:00.342+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId= xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,resourceType=VCENTER,resourceId= xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,bundleElementId= xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] [c.v.e.s.l.p.i.v.VCenterUpgrader,Upgrade-7] Can't obtain Resource pool for the source vm null in post install

 

• com.vmware.evo.sddc.lcm.model.error.LcmException: Failure occurred during vCenter upgrade. Unable to find the VirtualMachine Managed Object Reference of the vCenter being upgraded.

 

Environment

VMware Cloud Foundation 5.1.0.0

 

Cause

The property collector interface in the vSphere API provides a way to monitor and retrieve properties of managed objects in the vSphere inventory.

It is used during an upgrade, such as was being performed.

An issue has been observed previously, whereby the property collector response is truncated to 100 objects/entries while the actual response would have been expected to yield all the objects (with several VMs managed by the vCenter Server).

If the required vCenter Server VM is absent in the truncated response (in this case the source vCenter VM), the SDDC Manager is unable to find the vCenter VM, causing the upgrade to be marked as a failure (hence the null entries in the log snippet above).

This article captures a workaround, which could be implemented in advance of a similar upgrade.

Note - the failure happens at a different step, but the resolution described is still applicable to avoid the issue encountered in this case.

vCenter upgrade in VCF 5.1.x fails at stage SET_DRS_AUTOMATION_LEVEL_TO_MANUAL_FAILED

https://knowledge.broadcom.com/external/article/314655/vcenter-upgrade-in-vcf-51x-fails-at-stag.html

 

 

Resolution

It is intended that this behavior will be resolved in future VCF releases.

Please contact Broadcom Support to assist with implementing the work around.