Unable to download any bundles in bundle management: Retrieving available bundles failed. Unable to retrieve aggregated domains upgrade status: Encountered fetching http://127.0.0.1/lcm/inventory/upgrades api - Request failed with status code 500

Products

• VMware SDDC Manager

Issue/Introduction

• Unable to download any bundles in bundle management in the SDDC Manager. 
• Bundle Management page displays the below errors 
  Retrieving available bundles failed. Unable to retrieve aggregated domains upgrade status: Encountered fetching http://127.0.0.1/lcm/inventory/upgrades api - Request failed with status code 500
Retrieving all applicable bundles failed. Encountered fetching http://127.0.0.1/lcm/inventory/upgrades api - Request failed with status code 500
  
  

 

• /var/log/vmware/vcf/lcm/lcm.log has events as shown below: 

DEBUG [vcf_lcm,6849c02663fbce4ed8c740223d304d3a,ad70] [c.v.e.s.l.s.v.i.VvsSddcProductTypeHandler,Scheduled-3] SDDC manager version considered for VVS VMW bundles availability check for domain 29f0369a-5657-4aa1-b68f-d818142a51b6: {"productType":"SDDC_MANAGER","version":{"version":"5.2.1.2-24690695"}}
DEBUG [vcf_lcm,6849c02663fbce4ed8c740223d304d3a,ad70] [c.v.e.s.l.s.v.i.BomBundlesAvailabilityServiceImpl,Scheduled-3] validating vxrail bundle path [null]
DEBUG [vcf_lcm,6849c02663fbce4ed8c740223d304d3a,ad70] [c.v.e.s.l.d.c.m.LcmManifestClientImpl,Scheduled-3] Found manifest with version 1 and sequence number 101

DEBUG [vcf_lcm,6849c02663fbce4ed8c740223d304d3a,ad70] [c.v.e.s.l.s.i.LcmManifestServiceImpl,Scheduled-3] Current vcfWithVxRailDomainVersions map from inventory: {5.2.1.0=[8.0.320-28921695, 8.0.310-28809519]}
DEBUG [vcf_lcm,6849c02663fbce4ed8c740223d304d3a,ad70] [c.v.e.s.l.s.i.LcmManifestServiceImpl,Scheduled-3] VxRail include list for version: 5.2.1.0 : [8.0.310-28809519, 8.0.310-28809519]
DEBUG [vcf_lcm,6849c02663fbce4ed8c740223d304d3a,ad70] [c.v.e.s.l.s.i.UpgradeSequencerServiceImplPreferred,Scheduled-3] Upgrade order is not defined for target release 5.2.1.0 or BOM is customized:'true'. Using the default upgrade sequence
ERROR [vcf_lcm,6849c02663fbce4ed8c740223d304d3a,e8ef] [c.g.common.eventbus.EventBus.default,Scheduled-3] Exception thrown by subscriber method recieveMessage(com.vmware.evo.sddc.lcm.model.task.AbstractLcmControllableScheduledTask) on subscriber com.vmware.evo.sddc.lcm.core.scheduler.BundleDownloadScheduler@636ee65a when dispatching event: com.vmware.evo.sddc.lcm.core.scheduler.BundleDownloadScheduler@636ee65a
java.lang.NullPointerException: null
 at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)


Environment

VCF 5.2.1 

VCF 5.2.2

Cause

This issue can occur due to missing of VxRail version 8.0.320-28921695 in the depot as Flex-BOM support for this VxRail version has been removed due to a critical bug on VxRail. It can also occur if there are stale Upgrade plans left undeleted. 

Resolution

NOTE: Take a snapshot of the SDDC Manager appliance before making the below changes: 

• Delete old Upgrade plans using the API explorer from Developer Center in SDDC UI by calling the API: DELETE /v1/releases/domains/{domain_id} for each domain.
  
  
• Restart the LCM service: 
  systemctl restart lcm


• Navigate SDDC UI > LifeCycle Management > Bundle Management to validate the errors do not appear again. 
  
  
• Proceed to create upgrade plan from the SDDC manager. 

 

Additional Information

• If the above plan doesn't work (deleting API fails) follow the below steps:

1. ssh to sddc manager
2. su 
3. psql -h localhost -U postgres lcm -c 'truncate domain'

• Restart the LCM service. 
• Recreate the upgrade plan in SDDC manager.