Unable to upgrade VMware Cloud Foundation Operations to version 9.1.1 due to a stuck task- "Refresh Software Depot Metadata."

Products

• VCF Operations

Issue/Introduction

• When attempting to upgrade VCF SDDC Manager, the "Refresh Software Depot Metadata" task remains stuck in an "In Progress" state for an extended duration. Repeated attempts to run the lifecycle metadata sync fail to complete or progress. When navigating to VCF Management > Lifecycle > Tasks, the task continues to reflect an active, non-responsive state.

 

• Under /sos-####/VCFMgmtComponents/vcf-management-services/services-logs/ you may observe entries similar to the following within vcf-fleet-upgrade-service.log:

  org.springframework.web.client.HttpServerErrorException$BadGateway: 502 Bad Gateway: "connection failed: Get "https://#########:443/PROD/metadata/productVersionCatalog/v1/productVersionCatalog.json": unexpected EOF<EOL>"

Environment

VMware Cloud Foundation Operations 9.1.1

Cause

The issue is caused due to network timeouts when downloading the large master catalog manifest (productVersionCatalog.json) from the offline depot. Intermediate network devices terminate the connection after ~9–10 seconds, triggering a 502 Bad Gateway (unexpected EOF) that the vcf-fleet-upgrade-service fails to handle, leaving the parent upgrade task stuck in "In Progress" in the VCF Operations UI.

Resolution

1. Ensure the VCF SDDC Manager has connectivity to the offline depot.
2. Upgrade the Fleet Lifecycle Management component to version 9.1.1.0.25713934 or higher. This update includes updated service logic and restarts the vcf-fleet-upgrade-service microservices, effectively clearing stagnant TCP socket states and resetting task database locks.
3. After the update, navigate to the VCF Management > Lifecycle > Tasks UI.
4. Manually re-trigger the "Refresh Fleet Lifecycle" and "Refresh Software Depot Metadata" tasks.
5. Monitor tasks until completion.