"Deploy VCF Management Components" task cannot be re-initiated after failure

Products

• VCF Operations

Issue/Introduction

When attempting to deploy VCF Management Components during upgrade to VMware Cloud Foundation 9.1, the task fails and the user interface does not allow for a retry. The following can be observed:

• The "Deploy VCF Management Components" task shows as "Failed" in both the VCF Operations and SDDC Manager UIs.
• When attempting to re-deploy VCF Management Components from VCF Operations -->Build-->Lifecycle-->VCF Instance-->select the VCF instance-->SDDC Manager Upgrades, the "Install" button in the "Install Components" step continuously spins, preventing the user from re-initiating the task.
• In /var/log/vmware/vcf/domainmanager/domainmanager.log of SDDC Manager below messages can be found:
  
  Fetched executionContextName VCF_MANAGEMENT_COMPONENTS_DEPLOY_DESCRIPTION for executionId <execution ID>
......
End of Orchestration with FAILURE for Execution ID <execution ID>

Environment

VMware Cloud Foundation 9.1

Resolution

Below steps need to be executed from SDDC Manager to restore the system to a state where VCF Management Components can be re-deployed.

Before proceeding with database modifications, it is recommended to take a snapshot of the SDDC Manager appliance.

1. Login to SDDC Manager via SSH using the vcf user credentials. Switch to the root user with "su -".
2. Execute the following commands to clear out the failed deployment metadata and component references.
   Note: Replace <execution ID> in the final command with the actual ID discovered in domainmanager.log.
   
   psql -U postgres -h localhost -d platform -c "delete from vcf_management_component where component_type = 'FLEET_LCM';"
psql -U postgres -h localhost -d platform -c "delete from vcf_management_component where component_type = 'SDDC_LCM';"
psql -U postgres -h localhost -d platform -c "delete from vcf_management_component where component_type = 'VSP';"
psql -U postgres -h localhost -d platform -c "delete from vsp_cluster;"
psql -U postgres -h localhost -d platform -c "delete from task_metadata where id='<execution ID>';"


3. restart commonsvcs service.
   
   systemctl restart commonsvcs
   
   
4. Wait for 1-2 minutes for the service to fully initiate and refresh the page in VCF Operations, the "Install" button should be available now.
5. Log in to the vCenter Server and check for any orphaned or partially deployed VSP virtual machines. If any exist, power them off and delete them from disk.
6. Return to the VCF Operations UI and re-initiate the Install phase of VCF Management Components.