VCF Management Services Upgrade Fails at Deploy Software Depot Component with VCFMS-STAGE-COMPONENT-002

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

When attempting to upgrade VMware Cloud Foundation (VCF) Management Services via the API, the process fails at the subtask "Deploy Software Depot Component".
The following error is displayed in the VCF Operations UI:
Failed to stage component: "Internal error occurred. Please retry the operation or contact support if the issue persists.","Internal error occurred. Failed to stage the VCF Component. Please retry the operation or contact support if the issue persists. [VCFMS-STAGE-COMPONENT-002]"
Following errors are observed on SDDC Manager at location /var/log/vmware/vcf/domainmanager/domainmanager.log
YYYY-MM-DDT1HH:MM:SS.855+0000 ERROR [vcf_dm,6a284cfe9ba5dbe70a21ed2ea6f24f05,0ef8] [c.v.v.v.s.f.VspComponentService,dm-exec-1108] VSP task with id <task_id> (Staging VSP component of type: vcf-fleet-depot and version: 9.1.0.0.25371105) failed with status : FAILED. Task result: null
YYYY-MM-DDT1HH:MM:SS.855+0000 ERROR [vcf_dm,6a284cfe9ba5dbe70a21ed2ea6f24f05,0ef8] [c.v.v.v.s.f.VspComponentService,dm-exec-1108] Could not stage component with type: vcf-fleet-depot, version: 9.1.0.0.25371105, location: https://<sddc_manager_fqdn>/vmware/vcf/bundle/#######-####-####-####-############/vcf-fleet-depot-9.1.0.0.25371105.tgz
YYYY-MM-DDT1HH:MM:SS.869+0000 ERROR [vcf_dm,6a284cfe9ba5dbe70a21ed2ea6f24f05,0ef8] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-1108] [NDC1F5] PUBLIC_STAGE_VSP_COMPONENT_FAILED Failed to stage component: "Internal error occurred. Please retry the operation or contact support if the issue persists.","Internal error occurred. Failed to stage the VCF Component. Please retry the operation or contact support if the issue persists. [VCFMS-STAGE-COMPONENT-002]"

Environment

VMware Cloud Foundation 9.1

Cause

The SDDC Manager's TLS certificate lacks a Subject Alternative Name (SAN) and contains only a legacy Common Name (CN). As a result, the vmsp-operator is unable to pull the fleet-depot bundle over HTTPS. Running the following command on the SDDC Manager yields a blank output:
openssl x509 -in /etc/ssl/certs/vcf_https.crt -text -noout | grep -A 1 "Subject Alternative Name"

Resolution

Renew and replace the SDDC Manager certificate to ensure a valid Subject Alternative Name (SAN) is included in the certificate properties. This can be achieved as follows:

Prerequisites:

Administrative access to the SDDC Manager UI.
SSH access to the SDDC Manager appliance.
SSH/root access to the vCenter appliance that hosts the VMCA (required for signing).
The fully qualified domain name (FQDN) of the SDDC Manager.

Step 1: Generate and download the CSR:

1.1 Log in to the SDDC Manager UI.
1.2 Navigate to Inventory > Workload Domains.1.3 Open the management domain, then select the Certificates tab.
1.4 Note: SDDC Manager is listed in the certificate grid with Resource Type = SDDC Manager.
1.5 Click Generate CSRs and select the SDDC Manager resource.
1.6 Provide the organization details (Country,State,Locality,Organization,Org Unit,Email). The Common Name and the SAN are populated automatically from the SDDC Manager FQDN, Hence, a manual SAN entry is not required.
1.7 After the CSR is generated, click Download CSR(s) to save the CSR file.

Step 2: Sign the CSR with the existing vCenter VMCA:

2.1 Copy the downloaded CSR (vcf_https.csr) to the vCenter Server appliance at /tmp and open an SSH session as root (enable the shell by typing shell if required).
2.2 Create an extensions file at /tmp/v3ext.cnf that carries the SAN into the signed certificate (the FQDN and any extra names must match the CSR):
extendedKeyUsage = serverAuth,clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = <REDACTED_HOSTNAME>2.3 Sign the CSR with the VMCA root key and certificate by executing the following command:
openssl x509 -req \
-in /tmp/vcf_https.csr \
-CA /var/lib/vmware/vmca/root.cer \
-CAkey /var/lib/vmware/vmca/privatekey.pem \
-CAcreateserial \
-sha256 \
-days 730 \
-extfile /tmp/v3ext.cnf \
-out /tmp/vcf_https.crt

Step 3: Install the signed certificate:

3.1 Download the following files from the vCenter and copy to SDDC Manager:
Signed certificate: /tmp/vcf_https.crt
VMCA root certificate: /var/lib/vmware/vmca/root.cer3.2 Return to the management domain's Certificates tab in the SDDC Manager UI.
3.3 Click Install Certificates and select the SDDC Manager resource.
3.4 Provide the downloaded certificate files:
Server / resource certificate: Upload the signed SDDC Manager certificate (vcf_https.crt).
CA certificate: Upload the VMCA root certificate (root.cer).3.5 Click Validate, then click Install.