vSAN plugins (com.vmware.vsan.client) fail to download after a major version upgrade of vCenter

Products

VMware vCenter Server

Issue/Introduction

After a major version upgrade of vCenter Server (for example, from 7.x to 8.x), the vSAN plugin fails to download. Users may observe the following:

In the vCenter Tasks pane, the Download plug-in task reports the following error:

Error downloading plug-in. Make sure that the URL is reachable and the registered thumbprint is correct. No issuer certificate for certificate in certification path found.
vSAN configuration and monitor tabs are missing or blank in the vSphere Client.
In vCenter's /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log (The log indicates that vCenter Server is unable to download the vSAN plugin from its own endpoint),

DOWNLOAD_FAILED: Error downloading plugin package com.vwmare.vsan.client:8.0.203.10000 from https://<VCSA_FQDN_OR_IP>:443/vsan/plugins/vsan-ui-repa/plugin.zip. Reason: Download error. Make sure that the URL is reachable and the thumbprint is correct.
SSH to the vCenter and manually download that plug-in by

wget https://<VCSA_FQDN_OR_IP>:443/vsan/plugins/vsan-ui-repa/plugin.zip

is successful.

Environment

After a major version of VMware vCenter Server upgrade (for example, 7.x upgraded to 8.x)

Cause

The certificate thumbprint stored in the vCenter database (VCDB) does not match the current Machine SSL certificate thumbprint of the vCenter Server.

The vSphere Client plugin framework verifies the server thumbprint before downloading plugins. If the thumbprint recorded in the database does not match the actual certificate thumbprint, the plugin download will fail.

This may occur if the vCenter certificate was replaced or regenerated, but the stored thumbprint for the vSAN extensions was not updated.

To verify this, retrieve current thumbprint by logging in to the VCSA via SSH as root and run the following command to get the actual SHA256 thumbprint,

openssl s_client -connect <VCSA_FQDN_OR_IP>:443 </dev/null 2>/dev/null | openssl x509 -fingerprint -sha256 -noout

and retrieve the thumbprints currently stored in the vCenter database,

/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "select ext_server_id, ext_id, server_thumbprint from vpx_ext_server;"

(Locate the entries for com.vmware.vsan.client and com.vmware.vsan.health)

If the thumbprints returned by the two commands are different, the thumbprint stored in the vCenter database must be updated to match the current vCenter Server certificate thumbprint.

Resolution

Update the stored server thumbprint in the vCenter database so that it matches the current vCenter certificate thumbprint.

Take a snapshot of the vCenter Server before making changes.
Stop the vpxd service

service-control --stop vmware-vpxd

Verify the stored extension thumbprints

/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "select ext_server_id, ext_id, server_thumbprint from vpx_ext_server;"

Locate the entries for com.vmware.vsan.client and com.vmware.vsan.health
Obtain the current vCenter certificate thumbprint using the same FQDN or IP address that appears in the plugin download URL in the log file /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log (for example, https://<VCSA_FQDN_OR_IP>/vsan/plugins/vsan-ui-repa/plugin.zip):

openssl s_client -connect <VCSA_FQDN_OR_IP>:443 </dev/null 2>/dev/null | openssl x509 -fingerprint
Update the thumbprint stored in the database if the thumbprint for com.vmware.vsan.client or com.vmware.vsan.health differs from the thumbprint of the current certificate,

/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "update vpx_ext_server set server_thumbprint = '<NEW_SHA256_THUMBPRINT>' where ext_id = 'com.vmware.vsan.client';"

/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "update vpx_ext_server set server_thumbprint = '<NEW_SHA256_THUMBPRINT>' where ext_id = 'com.vmware.vsan.health';"

(Replace <NEW_SHA256_THUMBPRINT> with the thumbprint obtained from the openssl command in Step 5.)
Restart all vCenter services,

service-control --stop --all
service-control --start --all

Additional Information

The plugin package referenced in the log can be accessed directly at,

https://<vcenter_ip_or_fqdn>/vsan/plugins/vsan-ui-repa/plugin.zip

If the package can be downloaded manually (for example using wget), but the vSphere Client still reports a download failure, it may indicate a thumbprint mismatch between the vCenter certificate and the value stored in the VCDB.