Unable to download any bundles in bundle management: Retrieving available bundles failed. Unable to retrieve aggregated domains upgrade status: Encountered fetching http://127.0.0.1/lcm/inventory/upgrades api - Request failed with status code 500

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Unable to download any bundles in bundle management in the SDDC Manager. 
• Bundle Management page displays the below errors 
  Retrieving available bundles failed. Unable to retrieve aggregated domains upgrade status: Encountered fetching http://127.0.0.1/lcm/inventory/upgrades api - Request failed with status code 500
Retrieving all applicable bundles failed. Encountered fetching http://127.0.0.1/lcm/inventory/upgrades api - Request failed with status code 500
  
  

• Retrieving update patches bundles failed. Unable to retrieve aggregated LCM bundles. Error in getDomainBomVersion(): Encountered error requesting http://127.0.0.1/v1/system/upgradables api. Something went wrong. Please retry or contact the service provider and provide the reference token.

 

• /var/log/vmware/vcf/lcm/lcm.log has events as shown below: 

DEBUG [vcf_lcm,6849c02663fbce4ed8c740223d304d3a,ad70] [c.v.e.s.l.s.v.i.VvsSddcProductTypeHandler,Scheduled-3] SDDC manager version considered for VVS VMW bundles availability check for domain 29f0369a-#########-b68f-d818142a51b6: {"productType":"SDDC_MANAGER","version":{"version":"5.2.1.2-24690695"}}
DEBUG [vcf_lcm,6849c02663fbce4ed8c740223d304d3a,ad70] [c.v.e.s.l.s.v.i.BomBundlesAvailabilityServiceImpl,Scheduled-3] validating vxrail bundle path [null]
DEBUG [vcf_lcm,6849c02663fbce4ed8c740223d304d3a,ad70] [c.v.e.s.l.d.c.m.LcmManifestClientImpl,Scheduled-3] Found manifest with version 1 and sequence number 101

DEBUG [vcf_lcm,6849c02663fbce4ed8c740223d304d3a,ad70] [c.v.e.s.l.s.i.LcmManifestServiceImpl,Scheduled-3] Current vcfWithVxRailDomainVersions map from inventory: {5.2.1.0=[8.0.320-28921695, 8.0.310-28809519]}
DEBUG [vcf_lcm,6849c02663fbce4ed8c740223d304d3a,ad70] [c.v.e.s.l.s.i.LcmManifestServiceImpl,Scheduled-3] VxRail include list for version: 5.2.1.0 : [8.0.310-28809519, 8.0.310-28809519]
DEBUG [vcf_lcm,6849c02663fbce4ed8c740223d304d3a,ad70] [c.v.e.s.l.s.i.UpgradeSequencerServiceImplPreferred,Scheduled-3] Upgrade order is not defined for target release 5.2.1.0 or BOM is customized:'true'. Using the default upgrade sequence
ERROR [vcf_lcm,6849c02663fbce4ed8c740223d304d3a,e8ef] [c.g.common.eventbus.EventBus.default,Scheduled-3] Exception thrown by subscriber method recieveMessage(com.vmware.evo.sddc.lcm.model.task.AbstractLcmControllableScheduledTask) on subscriber com.vmware.evo.sddc.lcm.core.scheduler.BundleDownloadScheduler@636ee65a when dispatching event: com.vmware.evo.sddc.lcm.core.scheduler.BundleDownloadScheduler@636ee65a
java.lang.NullPointerException: null
 at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)


Environment

VCF 5.2.1 

VCF 5.2.x

Cause

The issue is typically caused by conflicting target version entries or stale entries remaining in the SDDC Manager database, specifically within the lcm service database. 

In VxRail environments, this issue can occur due to missing VxRail version 8.0.320-28921695 in the depot as Flex-BOM support for this VxRail version has been removed due to a critical bug on VxRail.  

Resolution

NOTE: Take a snapshot of the SDDC Manager appliance before making the below changes: 

• Delete old Upgrade plans using the API explorer from Developer Center in SDDC UI by calling the API: DELETE /v1/releases/domains/{domain_id} for each domain.
  
  
• Restart the LCM service: 
  systemctl restart lcm


• Navigate SDDC UI > LifeCycle Management > Bundle Management to validate the errors do not appear again. 
  
  
• Proceed to create upgrade plan from the SDDC manager. 

 

Additional Information

• If the above plan doesn't work (deleting API fails) follow the below steps:

1. ssh to SDDC manager
2. switch to root using command su 
3. psql -h localhost -U postgres lcm -c 'truncate domain'

• Restart the LCM service using systemctl restart lcm
  
  
• Recreate the upgrade plan in SDDC manager.