Remediating desired state cluster configuration fails with error 'The server certificate does not match the provided'

Products

• VMware vCenter Server

Issue/Introduction

• Checking cluster desired state configuration you observe the warning:

  Host is out of compliance with desired configuration

• Remediating the cluster compliance using either 'Export & Import configuration' or 'Import from host' fails another error:

  com.vmware.vcIntegrty contains a vSphere Configuration Plugin file that failed to Download('The server certificate does not match the provided')

• Reviewing the VUM plugin log, /var/log/vmware/vmware-updatemgr/vum-server/pluginrunner.log, you see error similar to:

  Unable to download com.vmware.vcIntegrity plugins
  ...
  SSLCertVerificationError('The server certificate does not match the provided certificate')

  
  

Environment

• VMware vCenter Server 7.0.x
• VMware vCenter Server 8.0.x

Cause

This issue occurs, when the certificate chain for the vCenter machine SSL certificate stored in /etc/vmware-vpx/ssl/rui.crt is not in the correct order Leaf > Intermediate CAs in correct order > Root CA (from top to bottom).
Output from executing command identifies the certificate chain is in the incorrect order:

# openssl s_client -connect VC_FQDN:443 -showcerts
For example the chain might be in this order:

-----BEGIN CERTIFICATE-----
(leaf certificate)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Intermediate CA Certificate B)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Intermediate CA Certificate A)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Root CA Certificate)
-----END CERTIFICATE-----

When the Certificates should be chained as follows (note the intermediate certificates are in a different order), as is outlined in How to correctly chain custom certificate for vCenter:

-----BEGIN CERTIFICATE-----
(leaf certificate)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Intermediate CA Certificate A)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Intermediate CA Certificate B)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Root CA Certificate)
-----END CERTIFICATE-----

Resolution

To fix this issue, ensure that a fresh backup or snapshot of the vCenter Server Appliance (VCSA) exists. If the affected VCSA is part of an Enhanced Linked Mode (ELM) replication setup, be aware that offline snapshots (in powered off state) of all of the ELM members are required. For further information on this, see VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice. Then take the following steps:

1. Open an SSH connection to the vCenter Server Appliance and login with the root account
2. Edit the /etc/vmware-vpx/ssl/rui.crt file using the VI editor:

   vi /etc/vmware-vpx/ssl/rui.crt

3. Chain the certificates in the correct order, with the leaf (endpoint) certificate at the top, followed by the intermediate CA certificate(s) in the correct order, like so:

   -----BEGIN CERTIFICATE-----
   (leaf certificate)
   -----END CERTIFICATE-----
   -----BEGIN CERTIFICATE-----
   (intermediate CA certificate A)
   -----END CERTIFICATE-----
   -----BEGIN CERTIFICATE-----
   (intermediate CA certificate B)
   -----END CERTIFICATE-----

4. Perform the same correction to the file /etc/vmware-rhttpproxy/ssl/rui.crt if required.
5. Restart the vCenter Services:

   # service-control --stop --all && service-control --start --all

6. Retry preferred import desired state configuration & remediate