VMSP deployment fails with error "Failed to stage the VCF Component. Please retry the operation or contact support if the issue persists. [VCFMS-STAGE-COMPONENT-002]"

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• VMSP deployment fails at the 'Deploy Software Depot Component' task with the following error displayed on the UI:
  
  Failed to stage component: "Internal error occurred. Please retry the operation or contact support if the issue persists.", "Internal error occurred. Failed to stage the VCF Component. Please retry the operation or contact support if the issue persists. [VCFMS-STAGE-COMPONENT-002]".


• On the SDDC Manager, in /var/log/vmware/vcf/domainmanager/domainmanager.log:
  
  YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,##########,####] [c.v.v.v.s.f.VspComponentService,dm-exec-###]  VSP task with id ############## (Staging VSP component of type: vcf-fleet-depot and version: 9.1.0.0.25371105) failed with status : FAILED. Task result: null
YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,##########,####] [c.v.v.v.s.f.VspComponentService,dm-exec-###]  Could not stage component with type: vcf-fleet-depot, version: 9.1.0.0.25371105, location: https://<SDDC_FQDN>/vmware/vcf/bundle/#####-####-####-####-#####/#####-####-####-####-#####/vcf-fleet-depot-9.1.0.0.25371105.tgz
YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,##########,####] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-###]  [#####] PUBLIC_STAGE_VSP_COMPONENT_FAILED Failed to stage component: "Internal error occurred. Please retry the operation or contact support if the issue persists.","Internal error occurred. Failed to stage the VCF Component. Please retry the operation or contact support if the issue persists. [VCFMS-STAGE-COMPONENT-002]"


• On VCF Operations and SDDC Manager, attempting to download the https://<SDDC_FQDN>/vmware/vcf/bundle/#####-####-####-####-#####/#####-####-####-####-#####/vcf-fleet-depot-9.1.0.0.25371105.tgz file fails with the following error:
  
  #wget https://<SDDC_FQDN>/vmware/vcf/bundle/#####-####-####-####-#####/#####-####-####-####-#####/vcf-fleet-depot-9.1.0.0.25371105.tgz

ERROR: cannot verify <SDDC_FQDN>'s certificate, issued by ‘CN=<decommissioned_server>,OU=VMware Engineering,O=VMware,L=Palo Alto,ST=California,C=US’: Self-signed certificate encountered.
To connect to <SDDC_FQDN> insecurely, use `--no-check-certificate'.

Environment

VMware Cloud Foundation 9.x

Cause

• The issuer for the SDDC Manager certificate is decommissioned.
• The presence of this signing certificate causes trust verification to fail when VCF operations attempts to securely pull the required bundles for staging.

Resolution

1. Replace the SDDC Manager certificate by renewing the self-signed certificates via SDDC Manager with OpenSSL integration. Refer: Renewing self signed certificates via SDDC manager with Open SSL integration
   
   
2. Import the root certificate of the SDDC Manager to the Trusted store on VCF Operations. Refer: Importing CA Certificates in VCF Operations
   
   
3. Add the SDDC Manager root certificate to the Operating System's Trusted store in VF Operations and SDDC Manager:
   
   
   1. Download the Certificate Chain:
      openssl s_client -showcerts -connect<SDDC_FQDN>:443 </dev/null 2>/dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /etc/ssl/certs/sddc-vmca.pem


   2. Edit the certificate saved as /etc/ssl/certs/sddc-vmca.pem and only keep the signing certificate.
      
      
   3. Rebuild the Trust Store:
      openssl rehash /etc/ssl/certs


4. Import  the SDDC Manager root certificate into the Fleet Depot Service's trust store on the VMSP platform. Refer: How add the TLS certificates into the Fleet Depot Service's trust store