Unhealthy state of the vSphere Cluster Services (vCLS) - SSL Thumbprint Mismatch

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

In the vSphere Web Client -> Recent Tasks pane, display failed "Deploy OVF Template". 

DRS Health Check on the cluster will show as "Unhealthy".

In the eam log, you will find logging similar to the following:

YYYY-MM-DDTHH:MM:SS.999Z | INFO | cluster-agent-0 | DeployVMJob.java | 999 | Zero-config deployment attempt failed (EamIOException->EamAppException [Upload request failed. Response:HTTP/1.1 526 Invalid SSL Certificate [content-length: 2086, content-type: text/plain, date: Fri, DD MM
YYYY HH:MM:SS GMT, server: envoy: envoy, connection: close] [Content-Length: 2086, Chunked: false]]), vSAN=false . Trying next.

Environment

Custom CA certificates are in use within the environment.

Cause

This issue occurs when the SSL thumbprint stored in the vCenter database does not match the current SSL thumbprint of the ESXi host.

Access the vCenter database via SSH

Connect to the vCenter Server Appliance (VCSA) using SSH and run the following command to open the vCenter PostgreSQL database:

• /opt/vmware/vpostgres/current/bin/psql -U postgres VCDB
• select * from vpx_host where id = 'xxxx';

Note: replace xxxx with the ESXi Host ID number

Compare the thumbprints:

From the query output, take note of:

• expected_ssl_thumbprint

• host_ssl_thumbprint

Then, run the following command to retrieve the current SSL thumbprint from the host:

• openssl s_client -connect <FQDN>:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin

Note: replace FQDN with the name of one of the Hosts present in the Cluster with the issue.

These values are supposed to match.

Resolution

To resolve this issue, vCenter must update the "Expected SSL Thumbprint" on each ESXi host. This can be achieved by disconnecting and reconnecting the hosts using the vSphere Client (UI).

Steps:

1. Log in to the vSphere Client.

2. Navigate to the cluster where the affected hosts reside.

3. For each ESXi host (perform steps one at a time):

   • Place the host into Maintenance Mode.

   • Right-click the host and select "Disconnect".

   • Once disconnected, right-click the host again and select "Connect".

   • Exit Maintenance Mode.

4. Repeat the above steps for all hosts in the cluster.

There is minimal impact on the environment when running these steps. There is no expected downtime on the Hosts or the VMs.