"Getting Upgradable API call failed" and "Retrieving update patches bundles failed" Errors in SDDC Manager Updates tab after upgrade

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

After upgrading/updating SDDC Manager, users may encounter errors when attempting to access the updates tab in the SDDC Manager UI. These errors prevent the system from properly retrieving and displaying available updates and patches.
One or more of the following errors are observed in the SDDC Manager UI under the Updates tab:
- Getting Upgradable API call failed. Something went wrong. Please retry or contact the service provider and provide the reference token.
- Retrieving update patches bundles failed. Unable to retrieve aggregated LCM bundles. Error in getDomainBomVersion(): Encountered error requesting http://127.0.0.1/v1/system/upgradables api. Something went wrong. Please retry or contact the service provider and provide the reference token.

- Getting Upgradable API call failed. Target VCF Version 5.2.1.0 is lower than source VCF Version 9.0.2.0.
- Retrieving available bundles failed. Unable to retrieve aggregated domains upgrade status: Encountered fetching http://127.0.0.1/lcm/inventory/upgrades api - Target VCF version 5.2.1.0 is lower than source VCF version 5.2.2.0.

References from /var/log/vmware/vcf/lcm/lcm.log:
YYYY-MM-DDTHH:MM:SSZ ERROR [vcf_lcm,0000000000000000,0000] o.a.c.c.C.[.[.[.[dispatcherServlet],http-nio-127.0.0.1-7400-exec-6] Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: java.lang.RuntimeException: Target VCF version 5.2.1.0 is lower than source VCF Version 9.0.2.0] with root cause
java.lang.RuntimeException: Target VCF version 5.2.1.0 is lower than source VCF Version 9.0.2.0
YYYY-MM-DDTHH:MM:SSZ ERROR [vcf_lcm,0000000000000000,0000] [c.v.e.s.e.h.VcfFallbackErrorController,http-nio-127.0.0.1-7400-exec-6] [GRKSK5] - Error attributes: {timestamp=<Day-Date-Time>, status=500, error=Internal Server Error, exception=java.lang.RuntimeException, trace=java.lang.RuntimeException: Target VCF version 5.2.1.0 is lower than source VCF Version 9.0.2.0
at com.vmware.evo.sddc.lcm.services.impl.LcmManifestServiceImpl.getBundleComponentsPair(LcmManifestServiceImpl.java:1534)
at com.vmware.evo.sddc.lcm.services.impl.LcmManifestServiceImpl.getBundleComponentsPair(LcmManifestServiceImpl.java:1513)
at com.vmware.evo.sddc.lcm.services.impl.LcmManifestServiceImpl.getVmwBundlesForSkipUpgrade(LcmManifestServiceImpl.java:1082)
at com.vmware.evo.sddc.lcm.services.vvs_availabilty.impl.BomBundlesAvailabilityServiceImpl.findDomainBundles(BomBundlesAvailabilityServiceImpl.java:197)
at com.vmware.evo.sddc.lcm.services.vvs_availabilty.impl.BomBundlesAvailabilityServiceImpl.findDomainBundlesInOrder(BomBundlesAvailabilityServiceImpl.java:74)
at com.vmware.evo.sddc.lcm.services.vvs_availabilty.impl.CurrentTargetVersionBomBundlesAvailabilityImpl.findDomainBomBundlesInOrder(CurrentTargetVersionBomBundlesAvailabilityImpl.java:57)

Environment

SDDC Manager 5.1.x
SDDC Manager 5.2.x
VCF 9.x

Cause

The issue is typically caused by conflicting target version entries or stale entries remaining in the SDDC Manager database, specifically within the lcm service database. These erroneous entries might originate from any of the managed nodes (such as vCenter, ESXi, NSX, etc.).
To validate, check the output of below command;
- psql -h localhost -U postgres lcm -c "select * from domain";Example below;
  target_version | id | target_vxrail_version | creation_time | modification_time | target_sddc_manager_version | target_esx_version | target_nsx_version | target_vc_version
  ----------------+--------------------------------------+-----------------------+-------------------------+-------------------------+------------- ----------------+--------------------+--------------------+-------------------
  9.0.2.0 | 99f#####-####-#####-####-########6c6 | | YYYY-MM-DD HH:MMSS | YYYY-MM-DD HH:MMSS | 9.0.2.0-25150386 | | |
  5.2.1.0 | c79####-####-#####-####-#########b6d | | YYYY-MM-DD HH:MMSS | YYYY-MM-DD HH:MMSS | 5.2.1.0-24307856 | | |
  
  Note: Exact timestamps and specific version numbers will vary by environment. In the documented example, SDDC Manager was upgraded to version 9.0.2.0, but a stale database entry for version 5.2.1.0 persisted.

Resolution

Issue can be resolved by identifying the affected domain ID and clearing the stale upgrade metadata through the SDDC Manager API.

Note: Before proceeding, it is important to take a snapshot of the SDDC Manager virtual machine.

Identify the Stale Domain Entry

Log in to VCF Operation and select Developer Center / API & SDKs .
Select the SDDC Manager API Explorer.
Locate the TargetUpgradeVersion section.
Expand the GET /v1/releases/domains endpoint and click Execute.
Review the DomainReleaseView in the API response.
Verify if the targetSddcManagerVersion field is empty for each domain.
Locate any domain entry where the version is lower than your current source version and record the domainId.

Remove the Stale Entry

1. Stay within the TargetUpgradeVersion section of the API Explorer.
2. Locate the DELETE /v1/releases/domains/{domainId} endpoint.
3. Enter the domainId you retrieved in the previous step into the required field.
4. Click Execute to remove the stale upgrade record.
5. Verify the removal by running the GET request again to ensure the targetSddcManagerVersion is now clear.
6. Restart lcm service using the below command

# systemctl restart lcm