Resolving Corrupted Database Entries in LCM and SDDC Manager During Upgrade Processes. "Retrieving update patches bundles failed. Unable to retrieve aggregated LCM bundles"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Failed configuration updates, fetching subtask info errors related to SAML2 authentication, and depot settings status call failures have been observed during upgrades of the SDDC (Software Defined Data Center) Manager. These issues are accompanied by HTTP parsing issues for vCenter, LCM errors with missing bundle IDs, and incorrect database entries in the lcm/upgrade table.
  
  
• VCF Upgrade Bundle not available or present to initiate and you may observe the following error in the GUI:
  • Retrieving update patches bundles failed. Unable to retrieve aggregated LCM bundles: Encountered error requesting http://127.0.0.1/v1/upgrades api - Encountered error requesting http://127.0.0.1/v1/upgrades api: A problem has occurred on the server. Please retry or contact the service provider and provide the reference token
    
    
• From /var/log/vmware/vcf/lcm/lcm.log
  • YYYY-MM-DDThh:mm:ss.zzz+0000 ERROR [vcf_lcm,<thread_ID#>,6cfa] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7400-exec-5] [Token-Ref####] UPGRADE_GET_FAILED Failed to get upgrade; UpgradeId=#####-#######-######-#####
com.vmware.evo.sddc.lcm.model.error.LcmException: Failed to get upgrade; UpgradeId=#####-#######-######-#####
        at com.vmware.evo.sddc.lcm.services.impl.UpgradeGetterService.getUpgrade(UpgradeGetterService.java:95)
        at com.vmware.evo.sddc.lcm.services.impl.UpgradeServiceImpl.getUpgrade(UpgradeServiceImpl.java:3098)
        at com.vmware.evo.sddc.lcm.services.impl.UpgradeServiceImpl.getUpgradeTask(UpgradeServiceImpl.java:3105)
Caused by: java.lang.IllegalArgumentException: No enum constant com.vmware.evo.sddc.lcm.model.upgrade.Upgrad1eStatus.USER_CANCELLED
    
    OR 
    
    
  • YYYY-MM-DDThh:mm:ss.zzz+0000 WARN  [vcf_lcm,e430e914ee9b44f9,9718] [c.v.e.s.l.d.c.u.UpgradeClientImpl,http-nio-127.0.0.1-7400-exec-1] Skip corrupt upgrade entry UpgradeId=#####-#######-######-#####
java.lang.IllegalArgumentException: No enum constant com.vmware.evo.sddc.lcm.model.upgrade.UpgradeStatus.COMPLETED
YYYY-MM-DDThh:mm:ss.zzz+0000 WARN  [vcf_lcm,e430e914ee9b44f9,9718] [c.v.e.s.l.d.c.u.UpgradeClientImpl,http-nio-127.0.0.1-7400-exec-1] Skip corrupt upgrade entry UpgradeId=#####-#######-######-#####
java.lang.IllegalArgumentException: No enum constant com.vmware.evo.sddc.lcm.model.upgrade.UpgradeStatus.COMPLETED

Environment

• VCF VMware SDDC Manager 5.x

Cause

An incomplete or improperly executed upgrade process leads to corrupted database entries within the Lifecycle Configuration Manager (LCM) and SDDC Manager databases. Specifically, upgrade tasks may be marked as "failed" instead of "completed with failure," preventing the system from correctly identifying failed upgrades and resulting in repeated errors during subsequent upgrade procedures.

Resolution

Please contact Broadcom support for the resolution for this issue.

Additional Information

Error messages observed include "retrieving configuration updates failed" and "Fetching Subtask Info Error."

Depot settings status calls fail due to HTTP parsing issues for vCenter.

Debug logs indicate LCM errors related to missing bundle IDs and incorrect database entries.

The issue is also characterized by internal server errors (vcr_error_internal_server_error) when attempting upgrades.