"Getting Upgradable API call failed" and "Retrieving update patches bundles failed" Errors in SDDC Manager Updates tab after upgrade

Products

• VMware SDDC Manager

Issue/Introduction

• After upgrading/updating SDDC Manager, users may encounter errors when attempting to access the updates tab in the SDDC Manager UI. These errors prevent the system from properly retrieving and displaying available updates and patches.
• One or more of the following errors are observed in the SDDC Manager UI under the Updates tab:
  • Getting Upgradable API call failed. Something went wrong. Please retry or contact the service provider and provide the reference token.
  • Retrieving update patches bundles failed. Unable to retrieve aggregated LCM bundles. Error in getDomainBomVersion(): Encountered error requesting http://127.0.0.1/v1/system/upgradables api. Something went wrong. Please retry or contact the service provider and provide the reference token.

• • Getting Upgradable API call failed. Target VCF Version 5.2.1.0 is lower than source VCF Version 9.0.2.0. 
  • Retrieving available bundles failed. Unable to retrieve aggregated domains upgrade status: Encountered fetching http://127.0.0.1/lcm/inventory/upgrades api - Target VCF version 5.2.1.0 is lower than source VCF version 5.2.2.0.

• References from /var/log/vmware/vcf/lcm/lcm.log:

YYYY-MM-DDTHH:MM:SSZ ERROR [vcf_lcm,0000000000000000,0000] o.a.c.c.C.[.[.[.[dispatcherServlet],http-nio-127.0.0.1-7400-exec-6] Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: java.lang.RuntimeException: Target VCF version 5.2.1.0 is lower than source VCF Version 9.0.2.0] with root cause
java.lang.RuntimeException: Target VCF version 5.2.1.0 is lower than source VCF Version 9.0.2.0
YYYY-MM-DDTHH:MM:SSZ ERROR [vcf_lcm,0000000000000000,0000] [c.v.e.s.e.h.VcfFallbackErrorController,http-nio-127.0.0.1-7400-exec-6] [GRKSK5] - Error attributes: {timestamp=<Day-Date-Time>, status=500, error=Internal Server Error, exception=java.lang.RuntimeException, trace=java.lang.RuntimeException: Target VCF version 5.2.1.0 is lower than source VCF Version 9.0.2.0
        at com.vmware.evo.sddc.lcm.services.impl.LcmManifestServiceImpl.getBundleComponentsPair(LcmManifestServiceImpl.java:1534)
        at com.vmware.evo.sddc.lcm.services.impl.LcmManifestServiceImpl.getBundleComponentsPair(LcmManifestServiceImpl.java:1513)
        at com.vmware.evo.sddc.lcm.services.impl.LcmManifestServiceImpl.getVmwBundlesForSkipUpgrade(LcmManifestServiceImpl.java:1082)
        at com.vmware.evo.sddc.lcm.services.vvs_availabilty.impl.BomBundlesAvailabilityServiceImpl.findDomainBundles(BomBundlesAvailabilityServiceImpl.java:197)
        at com.vmware.evo.sddc.lcm.services.vvs_availabilty.impl.BomBundlesAvailabilityServiceImpl.findDomainBundlesInOrder(BomBundlesAvailabilityServiceImpl.java:74)
        at com.vmware.evo.sddc.lcm.services.vvs_availabilty.impl.CurrentTargetVersionBomBundlesAvailabilityImpl.findDomainBomBundlesInOrder(CurrentTargetVersionBomBundlesAvailabilityImpl.java:57)

Environment

• SDDC Manager 5.1.x
• SDDC Manager 5.2.x
• VCF 9.x

Cause

• The issue is typically caused by conflicting target version entries or stale entries remaining in the SDDC Manager database, specifically within the lcm service database. These erroneous entries might originate from any of the managed nodes (such as vCenter, ESXi, NSX, etc.).
• To validate, check the output of below command;
  • psql -h localhost -U postgres lcm -c "select * from domain";
Example below;
    target_version |                  id                  | target_vxrail_version |      creation_time      |    modification_time    | target_sddc_manager_version | target_esx_version | target_nsx_version | target_vc_version
----------------+--------------------------------------+-----------------------+-------------------------+-------------------------+------------- ----------------+--------------------+--------------------+-------------------
9.0.2.0      | 99f#####-####-#####-####-########6c6 |                    | YYYY-MM-DD HH:MMSS | YYYY-MM-DD HH:MMSS | 9.0.2.0-25150386           |                    |                    |
5.2.1.0      | c79####-####-#####-####-#########b6d |                    | YYYY-MM-DD HH:MMSS | YYYY-MM-DD HH:MMSS | 5.2.1.0-24307856           |                    |                    |
    
    Note: Exact timestamps and specific version numbers will vary by environment. In the documented example, SDDC Manager was upgraded to version 9.0.2.0, but a stale database entry for version 5.2.1.0 persisted.

Resolution

• Manual cleanup of entries from SDDC database is required to fix the issue. As editing the database is destructive in nature, Open a case with Broadcom support.