VCSA update fails with error "Failed to load repository manifest data for the configured upgrade"

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

When setting up the custom repository that has authentication and self-signed certificates which are not trusted by the VCSA, patch the patcher module does not reach the repository and fails to download the files giving the below error: "Failed to load the repository manifest data for the configured upgrade".

Environment

VMware vCenter Server 8.0.0

Resolution

Currently there is no resolution to the issue.

Workaround:

To workaround the issue please follow any one of the solutions provided below as per the following table:

Sl No	SSL Certificate (Trusted/Non-Trusted)	Authentication	Will Work	Workaround
1	Trusted	No	Yes	NA
2	Trusted	Yes	No	Workaround 1
3	Non-Trusted	No	No	Workaround 2
4	Non- Trusted	Yes	No	Workaround1 + Workaround 2

Workaround 1 - Update using Leaf Service patching API

The leaf service update feature is available using the public APIs, but the UI is not available. This API can be used to update the vLCM service. Please use the below APIs to update the vLCM service that should fix the issue:

API to update the list:

curl --location --request GET 'https://<VCENTER_FQDN>:5480/rest/appliance/update/pending?source_type=LOCAL_AND_ONLINE&url=<CUSTOM_REPO_URL>' --header 'Authorization: Basic <Auth>'
(Replace <vCENTER_FQDN> with the impacted VCSA FQDN)
(Replace <CUSTOM_REPO_URL> with the configured repository URL)

API to fetch the leaf service:

curl --location --request GET 'https://<VCENTER_FQDN>:5480/rest/appliance/update/pending/8.0.0.20000/components' --header 'Authorization: Basic <Auth>'
(Replace <vCENTER_FQDN> with the impacted VCSA FQDN)

API to perform the vLCM update:

curl --location --request POST 'https://<VCENTER_FQDN>:5480/rest/appliance/update/pending/8.0.0.20000?action=stage-and-install' --header 'Authorization: Basic <Auth>' --header 'Content-Type: application/json' --data-raw '{"user_data": [ ], "component": "vlcm"}'
(Replace <vCENTER_FQDN> with the impacted VCSA FQDN)

Workaround 2 - Adding certs to trusted root

When the SSL certificates have been untrusted, these need to be added to the VCSA trusted root. This will work by copying the repository certificate to VCSA and trusting this certificate in VCSA and Python's trusted store.

Please follow the below steps to add the certificate to the trusted root:

SSH into the VCSA
Connect to the update repo using command:

# openssl s_client -connect <CustomUpdateRepoFQDN>:<port>

(Replace <CustomUpdateRepoFQDN> with the configured repository FQDN. The port is optional if any custom port is used)

This will list the certificate as the below example:

CONNECTED(00000003)
depth=0 C = IN, ST = KA, L = Bengaluru, O = VMware, OU = VsphereSolutions, CN = <>, emailAddress = <>
verify error:num=18:self signed certificate
verify return:1
depth=0 C = IN, ST = KA, L = Bengaluru, O = VMware, OU = VsphereSolutions, CN = <>, emailAddress = <>
verify return:1
---
Certificate chain
0 s:/C=IN/ST=KA/L=Bengaluru/O=VMware/OU=VsphereSolutions/CN=<>/emailAddress=<>
i:/C=IN/ST=KA/L=Bengaluru/O=VMware/OU=VsphereSolutions/CN=<>/emailAddress=<>
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIC/DCCAmWgAwIBAgIJAO7wGs2akToIMA0GCSqGSIb3DQEBCwUAMIGWMQswCQYD
VQQGEwJJTjELMAkGA1UECAwCS0ExEjAQBgNVBAcMCUJlbmdhbHVydTEPMA0GA1UE
CgwGVk13YXJlMRkwFwYDVQQLDBBWc3BoZXJlU29sdXRpb25zMRcwFQYDVQQDDA4x
MC4xNTguM5R96Hjli
-----END CERTIFICATE-----

Copy the certificate text (from -----BEGIN CERTIFICATE----- till -----END CERTIFICATE-----) and paste it in a text editor (preferably notepad++ not to lose the context of the cert).
Backup the file /usr/lib/python3.7/site-packages/certifi/cacert.pem using the command: # cp /usr/lib/python3.7/site-packages/certifi/cacert.pem /usr/lib/python3.7/site-packages/certifi/cacert.pem.old
Edit the file /usr/lib/python3.7/site-packages/certifi/cacert.pem using vi command (See KB:1020302 to know how to use vi for edit) and paste the text copied in step 4 at the end of the file.